Philipp Claßen's questions

I'm hardening the security groups for EC2 instances running on a default Ubuntu 20.04 AMI. What system services will break after closing all outgoing ports expect TCP 80 and TCP 443? (I'm assuming all ports required by the application are open, too. My concern is about breaking the implicit services provided by the OS.)

My understanding is that allowing TCP 80/443 is enough for the essential OS services. For example, apt-get updates should work. Or did I miss ports that Ubuntu or AWS will always expect to be open?

Side-note: I wondered how time syncing works. By default, I cannot see any NTP or chrony service preinstalled on the default Ubuntu AMI. Otherwise, UDP port 123 should be open, too. My assumption is that the (guest) hardware clock provided by the VM is already synced by the host (operated by AWS). Apart from that, I cannot think of any port that needs to be allowed from the OS perspective.

Depending on what AWS services the application will use, more ports are required, for example, 6379 for ElastiCache (Redis). Yet I'm concerned about API calls that do not originate from the deployed application. You can assume that the requirements of the application are known. The requirements by the environment (the OS and the EC2 infrastructure) are trickier.

Redis 4 added active memory defragmentation (source: release notes):

Active memory defragmentation. Redis is able to defragment the memory while online if the Jemalloc allocator is used (the default on Linux). Useful for workloads where the allocator cannot keep the fragmentation low enough, so the only possibility is for Redis and the allocator to collaborate in order to defragment the memory.

With Redis 5, the feature (now refered to as version 2) has been improved:

Source 1: tweet from Salvatore Sanfilippo, the Redis main developer

Active defragmentation version 2. Defragmenting the memory of a running server is black magic, but Oran Agra improved his past effort and now it works better than before. Very useful for long running workloads that tend to fragment Jemalloc.

Source 2: AWS announcement of Redis 5

One of the highlights of the previous release was the fact that Redis gained the capability to defragment the memory while online. The way it works is very clever: Redis scans the keyspace and, for each pointer, asks the allocator if moving it to a new address would help to reduce the fragmentation. This release ships with what can be called active defrag 2: It's faster, smarter, and has lower latency. This feature is especially useful for workloads where the allocator cannot keep the fragmentation low enough, so the strategy is for both Redis and the allocator to cooperate. For this to work, the Jemalloc allocator has to be used. Luckily, it's the default allocator on Linux.

Question: Assuming you are already using Jemalloc, is there any reason not to always set activedefrag yes?

Given that the alternative is to restart the instance to deal with fragmentation (which is highly problematic), and given that the overhead of activedefrag seems quite low from what I saw so far, the option seems to be too useful to be disabled by default.

Or are there any situations where it will harm performance?

systemd has the OOMScoreAdjust option, which allows to adjust the oom-killer score of the started process.

To quote from the systemd documentation:

OOMScoreAdjust=

Sets the adjustment level for the Out-Of-Memory killer for executed processes. Takes an integer between -1000 (to disable OOM killing for this process) and 1000 (to make killing of this process under memory pressure very likely). See proc.txt for details.

In my setup, I am deploying a NodeJs server on AWS. Beside the Node server there is not much else running on the EC2 instance (expect monitoring and the essential OS processes). There are ELB health checks in place, which should eventually replace broken EC2 instances.

Still, I wonder if it is considered good practice to increase OOMScoreAdjust to make the kernel prefer kill the Node server process if there are memory issues, as it can be automatically restarted. In systemd, it could look like this:

OOMScoreAdjust=1000
Restart=always

I have to admit that my understanding is limited. My current understanding is that it will most likely not make a real difference, and it is better to leave the defaults in place:

• If the memory draining process is the Node server, it will most likely be killed anyway.
• If the culprit is another process, restarting the Node server will not help and the ELB health checks should eventually take care of replacing the instance.

Still, I am curious if someone with a better understanding has already thought it through. Enabling it would only be one line in the systemd script. And when in doubt, I would rather have the kernel kill the Node process than any random system service.

Amazon recently introduced Target Tracking Policies for EC2 Auto Scaling.

In my production service, I am using two separate auto-scaling groups to support hybrid auto-scaling with a mixture of Spot and On-Demand instances. What I want is that my CPU usage should not exceed 70%, and it should use Spot instances whenever possible but fall back to On-Demand instances if necessary.

First, I set both Auto-Scaling groups (Spot and On-Demand) to use Target Tracking for 70% CPU load and set the minimum size of both groups to one. The traffic on my service is quite predictable (no sudden boost, more traffic during the day, few traffic during the night).

At one point, there were two On-Demand and two Spot instances running. The system had just scaled down because the CPU load of the five servers became very low (around 35%). With the four servers, the CPU load went up and after a few minutes briefly crossed the 70% mark (maybe there was a very minor traffic boost at that time).

The system decided conservatively to scale up again, but as both auto-scaling groups made the decision independently at the same time, two instances were started (one Spot and one On-Demand instance). At this point, there were now six servers running. After a while it scaled down again and finally reached a setup of running four instances.

To avoid that effect, I now changed the setup as follows:

• On-Demand: Target 70% CPU usage, one server minimum
• Spot: Target 65% CPU usage, one server minimum

My assumption is that it should help to prevent that scenario I described. I would expect that the On-Demand group scales down earlier than the Spot group (which is desirable, anyway, as they are more costly). And I expect that the Spot instances scale up sooner, which should protect against unnecessary upscaling from the On-Demand group.

That is my expectation, but I did not find much details in the documentation to confirm it. Can someone shed some light on how the new Target Tracking scaling works in details, and how to apply it to a hybrid setup with Spot and On-Demand instances?

Questions:

• If I set the target to 70% CPU utilization, when will it decide to scale up and when to scale down?
• If I have two Auto-Scaling groups, one with a 70% CPU utilization target and the other with 65%, when will it decide to scale up or down? Will it always prefer to scale down the 70% group? Will it always prefer to scale up the 65% group?
• What happens if the prices in the Spot market suddenly rise to exceed my bid limit. Will the On-Demand auto-scaling group take over?
• Is my understanding correct that manually defining the number of desired instances has only a short-term effect and will be automatically adjusted by the Auto Scaling policy?
• For example, if it scaled down to the minimum during the night and scaled up again next day, does it mean that the initial "number of desired instances" settings from the previous day are now obsolete? In other words, do I need to worry only about setting reasonable value for minimum and maximum, and will AWS will figure out the rest?

My goal is to start some EC2 instances that should be managed by an auto-scaling group. All instances should have public IPs.

In my tests, I noticed that depending on the configuration of the AWS account, this is not always possible. In the test account that I can use, some regions have an default VPC. In that regions, I was able to get public IPs.

In other regions, we have no default VPC configured. In that case, it is my understanding that it is not possible to get public IPs. Is that correct?

Also, the AWS documentation mentions the term ClassicLink. From my understanding that concept applies only to old accounts. It would also allow to have public IPs but only if there are VPCs configured where the "ClassicLink" flag is set. (In my account that is not the case, so I could not test it.)

In summery, I wanted to ask if my understanding is correct. Is it true that you can only have public IPs for EC2 instance in an auto-scaling group if at least one of the conditions holds:

1. In the region, there exists one default VPC
2. You have an old account (supporting EC2-Classic) and in the region there exists at least one VPC where ClassicLink is enabled

(Note that I cannot use an ELB, as the instances are not identical. If an ELB shuffles requests, it would confuse the system, as the sender needs to have control over which instance will handle the request.)

My question is about using Nginx as a proxy behind another proxy. (Somewhat confusing.)

I want to set up Nginx so it acts as a caching proxy server to an npm mirror. Here is the link: http://eng.yammer.com/a-private-npm-cache/

On my local machine, which is not restricted by a firewall, the following configuration works fine:

proxy_cache_path /var/cache/npm/data levels=1:2 keys_zone=npm:20m max_size=1000m
inactive=365d;
proxy_temp_path /var/cache/npm/tmp;

server {
   listen 80;
   server_name classen.abc.lan;
   location / {
      proxy_pass http://registry.npmjs.org/;
      proxy_cache npm;
      proxy_cache_valid 200 302 365d;
      proxy_cache_valid 404 1m;
      sub_filter 'registry.npmjs.org' 'classen.abc.lan';
      sub_filter_once off;
      sub_filter_types application/json;
   }
}

Now I want to apply it to a server that is behind an additional firewall. In the logs, I can confirm that it accesses the correct upstream IP, but the request fails because of the internal firewall.

We have one internal proxy, which I can use to bypass the firewall, for example:

$ curl http://registry.npmjs.org
curl: (7) couldn't connect to host
$ http_proxy=http://proxy.abc.lan:1234/ curl http://registry.npmjs.org
... succeeds ...

This trick does not work with Nginx, as it ignores the http_proxy environment variable. After reading the documentation, I still could not figure out how to modify the configuration, so that it can use the proxy internally.

Is it possible to combine both solutions? It is important that the caching still works, otherwise, you can just use the external mirror registry.npmjs.org directly.

Maybe, Nginx should use the internal proxy (proxy.abc.lan) as proxy_pass, but then how does the internal proxy know that the request should be sent to the external npm mirror (http://registry.npmjs.org)?

Update to Lukas answer

I tried Lukas solution:

rewrite ^(.*)$ "http://registry.npmjs.org$1" break;
proxy_pass http://proxy.abc.lan:1234;

The logs show that the URL is rewritten but it results in a redirect (triggered by curl classen.abc.lan/test-url):

2014/03/24 11:31:16 [notice] 13827#0: *2 rewritten redirect: "http://registry.npmjs.org/test-url", client: 172.18.40.33, server: classen.abc.lan, request: "GET /test-url HTTP/1.1", host: "classen.abc.lan"

The result of the curl call is not the expected JSON string from http://registry.npmjs.org but a html page generated by Nginx:

$ curl classen.abc.lan/test-url
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx/1.4.7</center>
</body>
</html>

I know that on Unix Systems, you can keep an ssh connection open.

Step 1: Create a ssh configuration, for example:

 Host <some.host.name>
 ControlPath ~/.ssh/master-%r@%h:%p
 ControlPersist yes

Step 2: Start ssh with the -M parameter.

Is there something equivalent under Windows to reuse ssh connections? Maybe I looked at the wrong places, but I didn't find any good links on this topic.

(I tried my approach from above in a MingGW environment (using MingGW's ssh client). In real Unix system, it should have created a special socket file in ~/.ssh, but that functionality does not seem to be supported by MingGW.)

I'm using a fresh installed CentOS 6.4 (x86-64) with the EPEL repository.

$ yum install nodejs
...
$ node -v
v0.10.5

So far, so good, but when I try to install npm, I'll get this error message:

$ yum install npm
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: ftp.halifax.rwth-aachen.de
 * epel: mirror.fraunhofer.de
 * extras: centos.psw.net
 * updates: centos.psw.net
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package npm.noarch 0:1.2.17-5.el6 will be installed
--> Processing Dependency: npm(which) < 2 for package: npm-1.2.17-5.el6.noarch
--> Processing Dependency: npm(uid-number) < 1 for package: npm-1.2.17-5.el6.noarch
--> Processing Dependency: npm(tar) < 0.2 for package: npm-1.2.17-5.el6.noarch
--> Processing Dependency: npm(slide) < 2 for package: npm-1.2.17-5.el6.noarch
--> Processing Dependency: npm(semver) < 1.2 for package: npm-1.2.17-5.el6.noarch
...
(skipped)
...
--> Finished Dependency Resolution
Error: Package: nodejs-npmlog-0.0.2-4.el6.noarch (epel)
           Requires: npm(ansi) < 0.2
Error: Package: npm-1.2.17-5.el6.noarch (epel)
           Requires: npm(ansi) < 0.2
Error: Package: nodejs-npmlog-0.0.2-4.el6.noarch (epel)
           Requires: npm(ansi) >= 0.1.2
Error: Package: npm-1.2.17-5.el6.noarch (epel)
           Requires: npm(ansi) >= 0.1.2
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

The complete output is quite big: http://pastebin.com/raw.php?i=Ce3ZJsEB

What should I do? The --skip-broken workaround doesn't seem to be right solution.

Here is the output of yum repolist:

Loaded plugins: fastestmirror, refresh-packagekit, security
Determining fastest mirrors
 * base: centos.copahost.com
 * epel: mirror.fraunhofer.de
 * extras: ftp-stud.fht-esslingen.de
 * updates: centos.copahost.com
 repo id       repo name                                            status
 base          CentOS-6 - Base                                      6,381
 epel          Extra Packages for Enterprise Linux 6 - x86_64       8,909
 extras        CentOS-6 - Extras                                       12
 updates       CentOS-6 - Updates                                     727
 repolist: 16,029