Willem's questions

I get this consistenly (Ubuntu 16.04 LTS):

$ docker pull nginx
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: TLS handshake timeout

However curl TLS works fine (apart from the auth error):

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

And even a small golang program (to mimic docker) works fine:

package main
import (
    "fmt"
    "io/ioutil"
    "net/http"
)
func main() {
    resp, err := http.Get("https://registry-1.docker.io/v2/")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    fmt.Println("Got: ", string(body))
}

The pcap for the docker TLS timeout request:

reading from file docker-timeout.pcap, link-type LINUX_SLL (Linux cooked)
00:38:54.782452 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [S], seq 26945613, win 29200, options [mss 1460,sackOK,TS val 1609360 ecr 0,nop,wscale 7], length 0
00:38:54.878630 IP registry-1.docker.io.https > my-ubuntu.52036: Flags [S.], seq 2700732154, ack 26945614, win 26847, options [mss 1460,sackOK,TS val 947941366 ecr 1609360,nop,wscale 8], length 0
00:38:54.878691 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 1609384 ecr 947941366], length 0
00:38:54.878892 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1609384 ecr 947941366], length 155
00:38:55.175931 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1609459 ecr 947941366], length 155
00:38:55.475954 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1609534 ecr 947941366], length 155
00:38:56.076327 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1609684 ecr 947941366], length 155
00:38:57.280103 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1609985 ecr 947941366], length 155
00:38:59.684095 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1610586 ecr 947941366], length 155
00:39:04.492102 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1611788 ecr 947941366], length 155
00:39:04.879468 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [F.], seq 156, ack 1, win 229, options [nop,nop,TS val 1611884 ecr 947941366], length 0
00:39:04.976015 IP registry-1.docker.io.https > my-ubuntu.52036: Flags [.], ack 1, win 105, options [nop,nop,TS val 947943890 ecr 1609384,nop,nop,sack 1 {156:157}], length 0
00:39:04.976073 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1611909 ecr 947943890], length 155
00:39:05.275922 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1611984 ecr 947943890], length 155
00:39:05.876104 IP my-ubuntu.52036 > registry-1.docker.io.https: Flags [P.], seq 1:156, ack 1, win 229, options [nop,nop,TS val 1612134 ecr 947943890], length 155

What could possibly be going wrong?

Our backup system duplicity does not support Linux hard links. To prevent data loss, I want to enforce that no hard links exist on our systems. AFAIK, there is no ext4 mount option to disable hard link functionality.

I could remove the ln utility but this would not disable the underlying system call (and besides, I want to keep symlinks).

Anyone got a good idea how to solve this? Recompile the kernel with EXT4_LINK_MAX set to 1?

I am looking for a generic way to hard-cap erratically growing logfiles. The existing answers (linked below) suggest using a polling algo with logrotate, but that approach does not enforce a hard limit (as files can get pretty big even within one minute).

Does anyone know of a solution or implementation? If nothing exists, why not?

Related questions:

• How to avoid log files growing too big?
• How to rotate log based on an interval unless log exceeds a certain size?

How do you handle a large volume of cron mails (from a thousand servers) in a smart way? Main use case: a cron error comes in, but its severity does not warrant immediate action. However, I do not want an email about it every other minute. Obviously I could silence it, but then I will forget about the error.

Sample error: a periodic unattended-upgrades failed because there was not enough memory available.

Ideally, I would use an Opbeat or Sentry-like service but for cron output. It would allow me to aggregate (on server and command), assign and mute incidents.

But hopefully somebody else has implemented something clever already.

Thanks for your suggestions!

Update: I found a sentry-cron utility @ https://pypi.python.org/pypi/cron-sentry which seems to do what I want. For Opbeat this is also possible, but nobody has written a wrapper yet.

I have 1M static rewrite rules and using this map configuration. How to determine the optimal values for map_hash_max_size and map_hash_bucket_size? I want to optimize for memory consumption. The documentation is very minimal about this.

Somebody else asked it on the Nginx forum, but without response.

With this Nginx config:

server {

    listen 443 ssl;
    server_name www.x.nl x.nl;

    ssl_certificate /etc/nginx/ssl/x.nl.crtkeyca;
    ssl_certificate_key /etc/nginx/ssl/x.nl.crtkeyca;

    [...]

}

This works:

openssl s_client -servername www.x.nl -connect localhost:443 < /dev/null

This doesn't, as I get the default vhost CN:

openssl s_client -servername x.nl -connect localhost:443 < /dev/null

Am using stock Nginx from Ubuntu 12.04 LTS, unfortunately very old (1.1.19).

Is it supposed to work in newer versions or do I require to duplicate my server {} definitions?

I would like to programmatically update a string in a very simple XML config file, such as this:

<?xml version="1.0"?>
<x>
    <z>should be changed</z>
</x>
<y>
    <z>should NOT be changed</z>
</y>

Does a simple CLI command exist that will update x/z and NOT y/z ? Perhaps with an XPath selector argument, such as:

$ xml-update myfile.xml /x/z "my new string"

I could write something in Perl or Python, but given the simplicity of this problem, I guess there must be a clever Unix utility that does this already.

Update: I found this but it seems a bit overkill. And I would have to learn XSLT first. http://developerblog.redhat.com/2013/12/05/xml-editing-bash-script/

So you boot a new cloud server. Only to discover that its IP is blacklisted by misbehavior of the previous owner. How to make sure your mail gets delivered for a modest price?

Context: I provide my customers each with their own cloud server. They use it to send on average 20 mails per day to their customers, using their own domain name. I have a couple of hundred customers/servers and I plan to have many more ;)

I have looked into Sendgrid, Mandril, Amazon SES, Mailjet, Mailgun but they are all focused on large marketing campaigns, not the occasional mail senders like my customers are. For example, with SendGrid I could create (by API) a "subuser" for each of my customers, but if my customer wants to get rid of the pesky "via sendgrid" header in Gmail/Outlook (and they want that), they are required to pay $20/month/domain. That seems quite expensive for sending < 100 mail. Plus it is a complicated procedure, which would be required for each of hundreds of domains.

So what I want is a basic SMTP relay service, that will somewhat reliably deliver my small volumes of mail for a modest price. It should support segmentation, ie if one customer somehow sends spam, he should be blocked but it shouldn't affect my other customers. Bonus points for not having my customers go through complicated domain verification procedures (ie. not DKIM).

How did you handle this issue?

TIA!

Tun1 is my VPN tunnel. I don't understand why I get "unreachable" here:

# ip route
default via 192.168.1.1 dev wlan1  proto static 
10.1.0.0/16 via 10.9.0.5 dev tun1 
10.9.0.1 via 10.9.0.5 dev tun1 
10.9.0.5 dev tun1  proto kernel  scope link  src 10.9.0.6 
192.168.1.0/24 dev wlan1  proto kernel  scope link  src 192.168.1.66  metric 9 

# ip route get 10.1.1.1
10.1.1.1 via 10.9.0.5 dev tun1  src 10.9.0.6 
    cache 

# ping -c1 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=63 time=40.4 ms

# ip route add 157.166.226.25/32 via 10.1.1.1
RTNETLINK answers: Network is unreachable

# ip route add 157.166.226.25/32 via 10.1.1.1 dev tun1
RTNETLINK answers: Network is unreachable

While debugging a DNS problem (apparently UDP packets get lost on the network), I'm stumbling upon many (but not all) "ethertype Unknown" messages such as follows. Tcpdump is run with "-n -i any port 53"

17:34:43.826196 IP 127.0.0.1.46508 > 127.0.0.1.53: 57861+ A? collector-123.newrelic.com. (44) 17:34:43.826221 IP 127.0.0.1.46508 > 127.0.0.1.53: 20016+ AAAA? collector-123.newrelic.com. (44) 17:34:43.826314 IP 127.0.0.1.53 > 127.0.0.1.46508: 57861 1/0/0 A 50.31.164.223 (60) 17:34:43.826505 IP 82.94.181.250.38280 > 208.78.70.21.53: 4554 AAAA? collector-123.newrelic.com. (44) 17:34:43.826510 Out 00:25:90:4b:94:dc ethertype Unknown (0x0067), length 92: 0x0000: 0000 0800 4500 0048 ef3d 4000 4011 2cab ....E..H.=@.@.,. 0x0010: 525e b5fa d04e 4615 9588 0035 0034 1f02 R^...NF....5.4.. 0x0020: 11ca 0000 0001 0000 0000 0000 0d63 6f6c .............col 0x0030: 6c65 6374 6f72 2d31 3233 086e 6577 7265 lector-123.newre 0x0040: 6c69 6303 636f 6d00 001c 0001 lic.com..... 17:34:43.826512 Out 00:25:90:4b:94:dc ethertype Unknown (0x0067), length 92: 0x0000: 0000 0800 4500 0048 ef3d 4000 4011 2cab ....E..H.=@.@.,. 0x0010: 525e b5fa d04e 4615 9588 0035 0034 1f02 R^...NF....5.4.. 0x0020: 11ca 0000 0001 0000 0000 0000 0d63 6f6c .............col 0x0030: 6c65 6374 6f72 2d31 3233 086e 6577 7265 lector-123.newre 0x0040: 6c69 6303 636f 6d00 001c 0001 lic.com..... 17:34:43.828629 In 00:1e:13:c1:52:00 ethertype Unknown (0x0067), length 159: 0x0000: 0000 0800 4500 008b b1dd 0000 3611 b3c8 ....E.......6... 0x0010: d04e 4615 525e b5fa 0035 9588 0077 8a5a .NF.R^...5...w.Z 0x0020: 11ca 8400 0001 0000 0001 0000 0d63 6f6c .............col 0x0030: 6c65 6374 6f72 2d31 3233 086e 6577 7265 lector-123.newre 0x0040: 6c69 6303 636f 6d00 001c 0001 c01a 0006 lic.com......... 0x0050: 0001 0000 0e10 0037 036e 7331 0370 3231 .......7.ns1.p21 0x0060: 0664 796e 6563 7403 6e65 7400 0c64 6f6d .dynect.net..dom 0x0070: 6169 6e2d 6164 6d69 6ec0 1a00 0002 3400 ain-admin.....4. 0x0080: 000e 1000 0002 5800 093a 8000 000e 10 ......X..:.....

What would cause these? Could it be related to hardware offloading to the nic? Is it harmful?

The more established configuration management (CM) systems like Puppet and Chef use a pull-based approach: clients poll a centralized master periodically for updates. Some of them offer a masterless approach as well (so, push-based), but state that it is 'not for production' (Saltstack) or 'less scalable' (Puppet). The only system that I know of that is push-based from the start is runner-up Ansible.

What is the specific scalability advantage of a pull based system? Why is it supposedly easier to add more pull-masters than push-agents?

For example, agiletesting.blogspot.nl writes:

in a 'pull' system, clients contact the server independently of each other, so the system as a whole is more scalable than a 'push' system

On the other hand, Rackspace demonstrates that they can handle 15K systems with a push-based model.

infastructures.org writes:

We swear by a pull methodology for maintaining infrastructures, using a tool like SUP, CVSup, an rsync server, or cfengine. Rather than push changes out to clients, each individual client machine needs to be responsible for polling the gold server at boot, and periodically afterwards, to maintain its own rev level. Before adopting this viewpoint, we developed extensive push-based scripts based on ssh, rsh, rcp, and rdist. The problem we found with the r-commands (or ssh) was this: When you run an r-command based script to push a change out to your target machines, odds are that if you have more than 30 target hosts one of them will be down at any given time. Maintaining the list of commissioned machines becomes a nightmare. In the course of writing code to correct for this, you will end up with elaborate wrapper code to deal with: timeouts from dead hosts; logging and retrying dead hosts; forking and running parallel jobs to try to hit many hosts in a reasonable amount of time; and finally detecting and preventing the case of using up all available TCP sockets on the source machine with all of the outbound rsh sessions. Then you still have the problem of getting whatever you just did into the install images for all new hosts to be installed in the future, as well as repeating it for any hosts that die and have to be rebuilt tomorrow. After the trouble we went through to implement r-command based replication, we found it's just not worth it. We don't plan on managing an infrastructure with r-commands again, or with any other push mechanism for that matter. They don't scale as well as pull-based methods.

Isn't that an implementation problem instead of an architectural one? Why is it harder to write a threaded push client than a threaded pull server?

I have 10.1.1.1 and 10.2.2.2 bound to eth0.

# ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 04:01:05:ff:42:01 brd ff:ff:ff:ff:ff:ff
    inet 10.2.2.2/24 brd 10.2.2.255 scope global eth0
    inet 10.1.1.1/32 scope global eth0
    inet6 fe80::601:5ff:feff:4201/64 scope link 
       valid_lft forever preferred_lft forever
# ip route get 10.1.1.1
local 10.1.1.1 dev lo  src 10.1.1.1 

Question A: why does it use "dev lo" when the IP is bound to eth0?

I want to force local connections to 10.1.1.1 to have a source address of 10.2.2.2. So I try:

# ip route add 10.1.1.1/32 dev eth0 src 10.2.2.2
# ip route show
default via 10.2.2.1 dev eth0 
10.1.1.1 dev eth0  scope link  src 10.2.2.2 
10.2.2.0/24 dev eth0  proto kernel  scope link  src 10.2.2.2 
# ip route get 10.1.1.1
local 10.1.1.1 dev lo  src 10.1.1.1 

Question B: How do I override this strange "local" route?

Background: the packets are intercepted by the IPVS kernel subsystem. Before it gets there, I need the correct source address, otherwise IPVS doesn't know what to do with it. SNATting with iptables doesn't work, as that happens in the POSTROUTING phase, and IPVS skips that. See also my related question at Linux IPVS in DR mode: VIP unreachable for director

Thanks!

I use direct routing mode. How do I make the loadbalanced service available to the director (DIR) using the virtual service ip (VIP)?

When I try to connect from DIR to the VIP, Linux (3.2.0) sees that the VIP is bound to a local interface and uses the VIP itself as source address.

Tcpdump:

08:47:51.573100 IP VIP.34762 > VIP.ssh: Flags [S]

According to tcpdump, the packet is not forwarded. Ipvsadm does see an extra inactive connection though. Even if it was forwarded, it wouldn't matter, as the realserver (RS) would not be able to send a reply, as it would not know where to send it to (RS has the VIP bound to the lo interface).

IMHO, the solution is to force the DIR to never create packets with the VIP as source address. When connecting to the VIP, it should use the DIR IP as source address.

However, another problem might be, that IPVS doesn't listen on the loopback interface.

# ip route get VIP
local VIP dev lo  src VIP

I could use iptables to rewrite the packets, but that feels quite hackish.

TIA.

Given a VM with 1 CPU. I run two similar CPU intensive tasks in two terminals:

/usr/bin/nice -n -20 perl -e 'while(1){$a=1+1;}'
/usr/bin/nice -n 19 perl -e 'while(1){$a=1+1;}'

I would expect that when I check with top, 1 process would take all the CPU and the other none (because there is no idle time left). But...

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
18175 root       1 -20 20376 1460 1132 R  49.9  0.1   0:18.36 perl
18176 root      39  19 20376 1460 1132 R  49.9  0.1   0:15.16 perl   

Why is that? I want my process to humbly only use idle CPU cycles :)

Tested on Linux 3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

For data redundancy goals, I want to record the binlog of a few hundred mysql servers in a central location. To run a full replicator for each stream would require lots of resources (multiple servers) because the execution of the statements take a lot of cpu. My assumption is that just recording the binlog would take significant fewer resources.

How to do it? I found http://search.cpan.org/~alfie/MySQL-Replication-0.0.2/bin/MySQLReplicationClient.pl but that seems to use non-native replication (it requires an extra database) and seems to be in alpha state.

Perhaps another solution is to install a full mysql server as a replication slave, but somehow disable the execution of replicated statements and just log them.

Thanks!

I was hit by a SYN flood that nearly killed the network stack of my ipvs loadbalancer (80K packets/sec). The "ksoftirqd" process took 100% CPU, the machine was nearly unresponsive from the network. A tcpdump revealed packets such as these:

21:24:40.994920 IP 193.41.136.187.17435 > MY-IP.80: Flags [S], seq 3387555840, win 0, length 0
21:24:40.994940 IP 46.159.109.102.63863 > MY-IP.80: Flags [S], seq 4209573888, win 0, length 0
21:24:40.994961 IP 32.199.104.48.30346 > MY-IP.80: Flags [S], seq 512360448, win 0, length 0

I couldn't find common denominators in the flood, apart from the zero window size, which is possibly a hook for filtering.

1. Is the ksoftirqd indeed the bottleneck?
2. Is there a way to filter on the linux box BEFORE it reaches the ksoftirqd bottelenck?
3. If not, how do I filter on window size == 0 on my Cisco 6509 ?

Thanks!

I run several hundred webservers behind loadbalancers, hosting many different sites with a plethora of applications (of which I have no control). About once every month, one of the sites gets hacked and a flood script is uploaded to attack some bank or political institution. In the past, these were always UDP floods which were effectively resolved by blocking outgoing UDP traffic on the individual webserver. Yesterday they started flooding a large US bank from our servers using many TCP connections to port 80. As these type of connections are perfectly valid for our applications, just blocking them is not an acceptable solution.

I am considering the following alternatives. Which one would you recommend? Have you implemented these, and how?

• Limit on the webserver (iptables) outgoing TCP packets with source port != 80
• Same but with queueing (tc)
• Rate limit outgoing traffic per user per server. Quite an administrative burden, as there are potentially 1000's of different users per application server. Maybe this: how can I limit per user bandwidth?
• Anything else?

Naturally, I'm also looking into ways to minimize the chance of hackers getting into one of our hosted sites, but as that mechanism will never be 100% waterproof, I want to severely limit the impact of an intrusion.

Update: I'm currently testing with these rules, that would have prevented this specific attack. How would you propose to make them more generic? Am I missing a known TCP DoS attack when I only rate limit on SYN packets?

iptables -A OUTPUT -p tcp --syn -m limit --limit 100/min -j ACCEPT
iptables -A OUTPUT -p tcp --syn -m limit --limit 1000/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A OUTPUT -p tcp --syn -j REJECT --reject-with tcp-reset

Cheers!

I need a storage system with the following requirements:

• Dynamic pool of Linux EC2 application servers (1-20) that autoscale daily
• Low latency reads
• Possibly async writes (<5 minutes)
• Most posix features, except locking
• Each application server can work autonomously so needs all the data locally (replication mode)

Because I want to replicate many machines (N-Master setup), I think that I need async writes, which would be acceptable as the application can stand 5 minute delay in write propagation. But am not sure how to do this in GlusterFS and whether it is viable at all. How would you set it up?

Given this setup, let me sneak two more questions in:

1. How would GlusterFS manage conflicting async writes? If I'm not too bothered about data loss, what is the best way to resolve these conflicts?
2. Also, most GlusterFS documentation writes about manually adding/removing bricks. Does anyone succesfully run a setup where the addition/removal of bricks is done automatically, many times a day?
3. Is there perhaps a better alternative to GlusterFS for these particular requirements?

Background: I host 2000 Magento shops, using NFS at the moment. It sucks (SPOF, reliability), so I'm looking for an alternative. Magento software can run on read-only storage, however 98% of these shops use external modules that somehow depend on a shared writable filesystem. Now I could tell the shop developers that they should ditch these modules, but I'm afraid I wouldn't keep many customers ;)

Thanks!

Possible Duplicate:
Is it possible to use an EC2 RDS MySQL instance as a slave to an external master?

My read-intensive application needs to scale horizontally. It will be deployed by various clients using Amazon EC2. My initial thought is that a basic setup will utilize one RDS instance and one EC2 instance. The EC2 (c1.medium) instance will run a webserver and a local MySQL read replica of the RDS server. The application is configured to use the RDS server for writes and localhost for reads. Using an autoscaling group, more c1.medium EC2 instances are booted according to load. Every new EC2 instance will have its own local mysql replicated data for reading.

Now the question: can I hook into RDS to create local read replicas?

The obvious alternative is to boot extra official read replica nodes using the RDS API, but I don't like that because then I would need to scale in steps of 2 machines instead of 1, which is more costly. Also, I prefer to have mysql read queries local because of the latency.

How can I make Apache kill the CGI's that it has spawned, after the client (browser) has closed the connection?

I got a couple of CGI's that perform long-running operations and send output every 15 seconds. As soon as the browser closes the connection, the CGI (and its descendants) should be properly killed instead of lingering around my system and keeping valuable Apache slots occupied.

Preferably Apache should kill them. If not, I would like to know if there is way for the CGI to detect whether the connection with the browser has gone away.

Edit: apparently, the CGI's will have their STDOUT closed. Upon writing, they'll get a SIGPIPE signal and could act accordingly (usually they ignore this signal). In Perl, you could do:

print "fresh output\n" or die "apparently my audience has left the building";

So the question remains: could Apache detect the broken pipe and clean up its garbage?

Running on Debian w/ Apache 2.2.20, and normal suexec mechanism.

Thanks!