smitelli's questions

We have an nginx server that, in some contexts, receives sensitive data in the HTTP username field. More specifically, it's an API key that clients are sending like curl -u "$API_KEY:" ....

The default nginx access_log format includes $remote_user, which writes the entire client API key into the access log and taints the file with sensitive data. I know I can define a different log_format that omits the $remote_user variable entirely, however I can see cases where having at least a hint about who the client was could be enormously helpful for log correlation or incident response. Is there a way to configure nginx to store a severely truncated copy of $remote_user in the access log instead of the full value from the client? (i.e. ABCDEFGH12345678 becomes ABCD* or something along those lines.)

(It also goes without saying I don't want to wreck the actual REMOTE_USER type variables that the WSGI backend relies on for authentication.)

This is nginx 1.10.3, as shipped in the default Debian Stretch repos.

I've got an nginx 1.10.3 server running on Debian Stretch. One of the sites it serves is a WebDAV share that is read and written by a desktop app. The app performs the following steps when a file named myfile is saved to the WebDAV server:

1. DELETE /myfile.tmp
2. PUT /myfile.tmp, body contains new file data
3. DELETE /myfile
4. MOVE /myfile.tmp, Destination: http://webdav.example.com/myfile
5. GET /myfile

The client app compares the response from step 5 to the data sent in step 2, and if the file data does not match an error is raised. These steps happen extremely rapidly on our particular network (the server and client are geographically close, connected to the same Ethernet switch) -- my testing with tcpdump suggests the entire conversation finishes within 45 ms.

The problem is, the data returned in step 5 doesn't immediately match what the client sent in step 2. The data being returned is the previous version of myfile, before the DELETE/MOVE steps replaced it. If I were to go back and repeat step 5 manually a moment later, the file data would be the new version as expected.

I know the client waits for each response to arrive before issuing a subsequent request. My best guess is that different requests are hitting different nginx workers/threads, or maybe there is some kind of cache invalidation or flush that isn't happening fast enough.

How can I fix this behavior without modifying the client app or artificially slowing down the requests?

Full nginx.conf and site config follows:

pid /run/nginx.pid;
user www-data;
worker_processes auto;
worker_rlimit_nofile 20000;

events {
    multi_accept on;
    use epoll;
    worker_connections 4000;
}

http {
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log warn;

    sendfile on;
    server_tokens off;
    tcp_nodelay on;
    tcp_nopush on;
    keepalive_requests 100000;
    keepalive_timeout 65;
    client_body_timeout 10;
    send_timeout 10;
    reset_timedout_connection on;
    types_hash_max_size 2048;

    open_file_cache max=200000 inactive=20s;
    open_file_cache_errors on;
    open_file_cache_min_uses 2;
    open_file_cache_valid 30s;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    gzip on;
    gzip_buffers 16 8k;
    gzip_comp_level 6;
    gzip_disable msie6;
    gzip_http_version 1.1;
    gzip_min_length 10240;
    gzip_proxied any;
    gzip_vary on;
    gzip_types
        application/atom+xml
        application/javascript
        application/json
        application/ld+json
        application/manifest+json
        application/rss+xml
        application/vnd.geo+json
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/x-javascript
        application/x-web-app-manifest+json
        application/xhtml+xml
        application/xml
        application/xml+rss
        font/opentype
        image/bmp
        image/svg+xml
        image/x-icon
        text/cache-manifest
        text/css
        text/javascript
        text/plain
        text/vcard
        text/vnd.rim.location.xloc
        text/vtt
        text/x-component
        text/x-cross-domain-policy
        text/xml;

    server {
        listen 80;
        listen [::]:80;
        server_name webdav.example.com;

        root /var/www/webdav.example.com;

        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
        dav_access user:rw group:r all:r;
        dav_methods DELETE MOVE PUT;
        create_full_put_path on;
    }
}

EDIT: An interesting observation I discovered. If I reload nginx (sudo service nginx reload), the very first attempt to save the file succeeds. But if I try to save it a subsequent time, the same error happens.

Suppose I'm building a microservice that resizes and optimizes image files. It's a long-running daemon which provides an HTTP API responding to standard GET/POST responses. Based on my knowledge of Docker, this is a perfect use case for containerization -- one process (the HTTP server daemon) running indefinitely in a dedicated container.

Where things get murky for me is if we look one level deeper into a possible way this service is built. Let's say that, for every request that comes in, the service spawns an ImageMagick process to resize the image, and then another pngquant or similar process to reduce the file size. Since these programs will be handling potentially untrustworthy user-provided image data, it's important to be able to update each component to the latest available version as soon as these new versions are released. But how do I split these components up in terms of Docker images and containers?

I've come up with a few different approaches, but it still seems like I'm missing something:

1. One big container. When building the HTTP API container, install/compile the ImageMagick/pngquant utilities at the same time. As far as the API daemon knows, it's just like running on any other computer. To update one of the binaries, rebuild the entire container (even if the API daemon itself hasn't changed). If it's necessary to test/develop against ImageMagick independently, it may be awkward because that's not the focus of this container's layout.

2. Containers running containers. The HTTP API is its own container, ImageMagick is its own container, pngquant is its own container, and so on. When the API is handling a request that requires invocation of one of these utilities, the API code starts a container to convert the one image file and that container is destroyed once the conversion is done. As I understand it, the HTTP API code would need some pretty lofty permissions to be able to create a new container; might not be a reasonable approach from a security standpoint.

3. Wrappers and glue. Wrap ImageMagick and pngquant in custom long-running daemon code so these containers never have to exit. Have the HTTP API container communicate with the others over the Docker network as required. Seems like a lot of pointless indirection and complexity for no real benefit.

4. Something about image composition that I'm missing. It doesn't look like there's a clean way to "piecewise" cobble together a container from multiple, independently-replaceable images. It would be interesting if there was a way to combine multiple images, each containing one of ImageMagick, pngquant, and the HTTP API, into a single container. Based on what I've seen, replacing/modifying an image also changes all the images that were built on top of it, making this approach not all that different from #1.

What I'm really looking for, above all else, is to be able to develop/build/test/deploy components of a container's software stack independently without rebuilding or reinstalling the parts that didn't change. If this conflicts with the Docker design philosophy too strongly, I'd be willing to either change my view of the approach or look for different tools.

This may sound like an odd question, but it's generated some spirited discussion with some of my colleagues. Consider a moderately sized RAID array consisting of something like eight or twelve disks. When buying the initial batch of disks, or buying replacements to enlarge the array or refresh the hardware, there are two broad approaches one could take:

1. Buy all the drives in one order from one vendor, and receive one large box containing all the disks.
2. Order one disk apiece from a variety of vendors, and/or spread out (over a period of days or weeks) several orders of one disk apiece.

There's some middle ground, obviously, but these are the main opposing mindsets. I've been genuinely curious which approach is more sensible in terms of reducing the risk of catastrophic failure of the array. (Let's define that as "25% of the disks fail within a time window equal to how long it takes to resilver the array once.") The logic being, if all the disks came from the same place, they might all have the same underlying defects waiting to strike. The same timebomb with the same initial countdown on the clock, if you will.

I've collected a couple of the more common pros and cons for each approach, but some of them feel like conjecture and gut instinct instead of hard evidence-based data.

Buy all at once, pros

• Less time spent in research/ordering phase.
• Minimizes shipping cost if the vendor charges for it.
• Disks are pretty much guaranteed to have the same firmware version and the same "quirks" in their operational characteristics (temperature, vibration, etc.)
• Price increases/stock shortages unlikely stall the project midway.
• Each next disk is on-hand the moment it's required to be installed.
• Serial numbers are all known upfront, disks can be installed in the enclosure in order of increasing serial number. Seems overly fussy, but some folks seem to value that. (I guess their management interface sorts the disks by serial number instead of hardware port order...?)

Buy all at once, cons

• All disks (probably) came from the same factory, made at the same time, of the same materials. They were stored in the same environment, and subject to the same potential abuses during transit. Any defect or damage present in one is likely present in all.
• If the drives are being replaced one-at-a-time into an existing array and each new disk needs to be resilvered individually, it could be potentially weeks before the last disk from the order is installed and discovered to be faulty. The return/replacement window with the vendor may expire during this time.
• Can't take advantage of near-future price decreases that may occur during the project.

Buy individually, pros

• If one disk fails, it shares very little manufacturing/transit history with any of the other disks. If the failure was caused by something in manufacturing or transit, the root cause likely did not occur in any other disk.
• If a disk is dead on arrival or fails during the first hours of use, that will be detected shortly after the shipment arrives and the return process may go more smoothly.

Buy individually, cons

• Takes a significant amount of time to find enough vendors with agreeable prices. Order tracking, delivery failure, damaged item returns, and other issues can be time-consuming to resolve.
• Potentially higher shipping costs.
• A very real possibility exists that a new disk will be required but none will be on-hand, stalling the project.
• Imagined benefit. Regardless of the vendor or date purchased, all the disks came from the same place and are really the same. Manufacturing defects would have been detected by quality control and substandard disks would not have been sold. Shipping damage would have to be so egregious (and plainly visible to the naked eye) that damaged drives would be obvious upon unpacking.

If we're going simply by bullet point count, "buy in bulk" wins pretty clearly. But some of the pros are weak, and some of the cons are strong. Many of the bullet points simply state the logical inverse of some of the others. Some of these things may be absurd superstition. But if superstition does a better job at maintaining array integrity, I guess I'd be willing to go along with it.

Which group is most sensible here?

UPDATE: I have data relevant to this discussion. The last array I personally built (about four years ago) had eight disks. I ordered from one single vendor, but split the purchase into two orders of four disks each, about one month apart. One disk of the array failed within the first hours of running. It was from the first batch, and the return window for that order had closed in the time it took to spin everything up.

Four years later, the seven original disks plus one replacement are still running error-free. (knock on wood.)

I learned a long time ago that if you want an EC2 instance to be able to talk to AWS services like SQS, Kinesis and the like, the instance has to either have a public IP address or there needs to be something within the VPC doing NAT.

When I first heard about VPC Endpoints earlier this year it seemed like a game-changer -- a way to access these services from instances that only had private IPs. But the fact that the only supported service is S3 seems really limiting. Every EC2 instance I currently manage has some other dependency on a non-S3 AWS resource that requires traffic to go out through a public IP.

I'm curious if there is any other benefit to using a VPC Endpoint for S3 anyway, with the understanding that the rest of the stack will still depend on public IPs and will for some time. Is there any measurable improvement in speed/throughput, or any other benefit that can outweigh the management overhead?

I have the following two CloudFormation resources:

"TestELB" { ... },
"TestRecordSetGroup": {
  "Type": "AWS::Route53::RecordSetGroup",
  "Properties": {
    "HostedZoneName": "example.com.",
    "RecordSets": [
      {
        "Name": "subdomain.example.com.",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": {"Fn::GetAtt": ["TestELB", "CanonicalHostedZoneNameID"]},
          "DNSName": {"Fn::GetAtt": ["TestELB", "CanonicalHostedZoneName"]}
        }
      },
      {
        "Name": "subdomain.example.com.",
        "Type": "AAAA",
        "AliasTarget": {
          "HostedZoneId": {"Fn::GetAtt": ["TestELB", "CanonicalHostedZoneNameID"]},
          "DNSName": {"Fn::Join": [".", ["ipv6", {"Fn::GetAtt": ["TestELB", "CanonicalHostedZoneName"]}]]}
        }
      }
    ]
  }
}

After the stack updates, I see both records listed in my zone with the expected alias values. The A record works, as verified with dig:

$ dig A subdomain.example.com
...
;; QUESTION SECTION:
;subdomain.example.com.       IN      A
;; ANSWER SECTION:
subdomain.example.com. 59     IN      A       11.22.33.44
;; Query time: 38 msec
...

But the AAAA record does not work:

$ dig AAAA subdomain.example.com
...
;; QUESTION SECTION:
;subdomain.example.com.       IN      AAAA
;; AUTHORITY SECTION:
example.com.         899     IN      SOA     ns-1234.awsdns-11.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
;; Query time: 54 msec
...

I think it has something to do with Fn::Join being used to add ipv6. to the beginning of the ELB's DNS name. If I change the A record so it uses Fn:Join to prepend dualstack. to the DNS name it also fails in the same way.

Is Fn::Join the correct way to add ipv6. or dualstack. to the beginning of a DNS name?

I have a, let's call it, "legacy" VHost configured on an nginx server. It used to be a full site, but it has since been decommissioned and all the content removed. The server block now looks something like this:

server {
    listen 80;
    server_name defunct-site.com;
    return 410;
}

There are DNS records and applications which are out of my control that still point to this server, and it still gets relatively significant traffic despite my pleads to the third party to update their configuration and stop trying to hit the site. Needless to say, I can't change my IP address without significant pointless effort.

Is there a stronger way to tell these clients that their traffic is no longer welcome here? The source IPs are all over the place, so doing a successful iptables block seems unlikely. And I need to read enough of the HTTP request to get to the Host: header to make sure I'm not clobbering traffic intended for a legitimate VHost.

I have a step in a Puppet module that roughly does the following to set up some application keys:

file { '/root/setup_app_keys.sh':
  ensure => file,
  owner  => 'root',
  group  => 'root',
  mode   => '0700',
  source => 'puppet:///modules/app_module/setup_app_keys.sh',
}

exec { 'setup_app_keys':
  unless  => '/etc/pki/tls/private/app-foo.key',
  command => '/root/setup_app_keys.sh',
  user    => 'root',
  group   => 'root',
}

The setup_app_keys.sh script is a little too long to be made into a (readable) one-liner, so I save it to the machine's filesystem and execute it from there. It creates its files in /etc/pki... and it works well enough.

The annoying thing is, the shell script is basically a one-time use thing. It shouldn't ever run again for the lifetime of the machine, and yet it must stay on the filesystem where Puppet stored it. If it is deleted, Puppet helpfully recreates it again.

I'm thinking there must be a way to rewrite this using exec exclusively, which will allow me to download the script from the puppetmaster when needed, execute it once, then discard the script (or not store it in the first place). But everything I've tried has been of the form:

command => 'puppet:///modules/app_module/setup_app_keys.sh',

or

command => 'curl http://__[various puppetmaster URLs]__ | sh',

and neither approach seems to work. Am I asking too much, or is this approach flawed?

I'm setting up a few logstash-forwarder agents which will send logs to a central Logstash server, and need to choose a port for Logstash to listen on. After doing some searching, there doesn't seem to be a consensus about what this port number should be. The lumberjack protocol doesn't have a well-known port assignment as far as I can see.

The closest I was able to find as a convention was port 5000 (for example, here) but that strikes me as more of a tutorial, get-it-working-fast port.

Are there any established conventions for choosing this port number?

Suppose I have a server with a private interface and a public interface. Public might have things like HTTP(S) servers, private might have MySQL and SSH.

Obviously Nagios is useful to check that the services are running on their respective interfaces. But is it a good idea to build checks that explicitly test that the MySQL and SSH ports are not open on the public interface? The idea is to catch inadvertent misconfigurations that have opened up services that should be private, and alert appropriately.

Part of me has the idea that this wouldn't scale terribly well -- imagine there is an iptables DROP rule, for example, the check would have to wait until the check timeout exceeded before it can complete and move on. But that timeout would have to be sufficiently high to be able to differentiate a blocked service from an open one that's really bogged down.

Is this a practical idea? Is Nagios the right tool? I haven't even looked into the feasibility of negating the result from the TCP check plugins, but I'm sure it's doable...