Mike Conigliaro's questions

Before anyone asks why you'd want to do that, I'm using molecule to test my "infrastructure as code" locally before deploying it to actual machines.

Start by creating two block devices:

# dd if=/dev/zero of=/tmp/test0 bs=1M count=10
# dd if=/dev/zero of=/tmp/test1 bs=1M count=10
# losetup /dev/loop0 /tmp/test0
# losetup /dev/loop1 /tmp/test1

Now create the array:

# mdadm --create /dev/md127 --metadata=0.90 --level=1 --raid-devices=2 /dev/loop0 /dev/loop1
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.76-linuxkit/modules.dep.bin'
modprobe: FATAL: Module md_mod not found in directory /lib/modules/4.19.76-linuxkit
mdadm: Fail create md127 when using /sys/module/md_mod/parameters/new_array
mdadm: unexpected failure opening /dev/md127

I tried installing the linux-generic package to populate the missing /lib/modules directory, but it doesn't look like the correct version of that package exists for the running kernel (this is on Ubuntu 20.04):

# uname -r
4.19.76-linuxkit

# apt show linux-generic | grep Version
Version: 5.4.0.12.15

I swear they changed this in the last year or two, but as of 2017-11-20, the SES developer guide for Postfix recommends setting up TLS via stunnel. This seems odd to me, considering that Postfix can already do TLS if you just add these lines to main.cf:

smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

In fact, I'm pretty sure I originally got the above configuration options from a previous version of the SES docs. Why did they change this?

After cloud-init runs a user data script on the first boot of an EC2 instance, a state file is presumably written so that cloud-init won't run the script again on subsequent reboots. There are cases where I'd like to delete this state file so that the user data script will run again. Where is it?

After running for about 18 hours, this system is using ~10GB of memory, causing the OOM-killer to be triggered when we run our usual tasks:

# free -h
             total       used       free     shared    buffers     cached
Mem:           14G       9.4G       5.3G       400K        27M        59M
-/+ buffers/cache:       9.3G       5.4G
Swap:           0B         0B         0B

# cat /proc/meminfo
MemTotal:       15400928 kB
MemFree:         5567028 kB
Buffers:           28464 kB
Cached:            60816 kB
SwapCached:            0 kB
Active:           321464 kB
Inactive:          59156 kB
Active(anon):     291464 kB
Inactive(anon):      316 kB
Active(file):      30000 kB
Inactive(file):    58840 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                40 kB
Writeback:             0 kB
AnonPages:        291380 kB
Mapped:            14356 kB
Shmem:               400 kB
Slab:             364596 kB
SReclaimable:      18856 kB
SUnreclaim:       345740 kB
KernelStack:        1832 kB
PageTables:         3720 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     7700464 kB
Committed_AS:     313224 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       35976 kB
VmallocChunk:   34359678732 kB
HardwareCorrupted:     0 kB
AnonHugePages:    231424 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     9598976 kB
DirectMap2M:     6260736 kB

However, processes don't seem to be using a significant amount of this memory:

# top -o %MEM -n 1
top - 15:07:00 up 18:28,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 155 total,   1 running, 154 sleeping,   0 stopped,   0 zombie
%Cpu(s): 23.7 us,  4.8 sy,  0.0 ni, 71.4 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:  15400928 total,  9838560 used,  5562368 free,    29764 buffers
KiB Swap:        0 total,        0 used,        0 free.    62760 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1333 root      20   0 5763204 274132   5352 S   0.0  1.8   7:00.19 java
 1466 newrelic  20   0  251484   4884   2056 S   0.0  0.0   0:56.41 nrsysmond
16804 root      20   0  105636   4212   3224 S   0.0  0.0   0:00.00 sshd
16876 root      20   0   21420   3908   1764 S   0.0  0.0   0:00.03 bash
16858 ubuntu    20   0   21456   3828   1684 S   0.0  0.0   0:00.05 bash
  770 root      20   0   10216   2868    576 S   0.0  0.0   0:00.02 dhclient
    1 root      20   0   33700   2216    624 S   0.0  0.0   0:35.50 init
16875 root      20   0   63664   2084   1612 S   0.0  0.0   0:00.00 sudo
16857 ubuntu    20   0  105636   1860    880 S   0.0  0.0   0:00.01 sshd
16920 root      20   0   23688   1528   1064 R   0.0  0.0   0:00.00 top
16803 postfix   20   0   27400   1492   1216 S   0.0  0.0   0:00.00 pickup
  976 root      20   0   43444   1100    748 S   0.0  0.0   0:00.00     systemd-logind
  572 root      20   0   51480   1048    308 S   0.0  0.0   0:00.53     systemd-udevd
 1840 ntp       20   0   31448   1044    448 S   0.0  0.0   0:02.94 ntpd
  990 syslog    20   0  255836    924     76 S   0.0  0.0   0:00.13 rsyslogd
 1167 root      20   0   61372    828    148 S   0.0  0.0   0:00.00 sshd
  945 message+  20   0   39212    788    416 S   0.0  0.0   0:00.12 dbus-daemon
 1323 root      20   0   20692    676      0 S   0.0  0.0   0:40.92 wrapper
 1230 root      20   0   19320    588    244 S   0.0  0.0   0:04.57 irqbalance
 1538 root      20   0   25336    500    188 S   0.0  0.0   0:00.18 master
  567 root      20   0   19604    480     96 S   0.0  0.0   0:00.34     upstart-udev-br
 1175 root      20   0   23648    404    156 S   0.0  0.0   0:00.08 cron
 1005 root      20   0   15272    348     88 S   0.0  0.0   0:00.08     upstart-file-br

Temp and shared memory filesystems are basically empty:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.4G   12K  7.4G   1% /dev
tmpfs           1.5G  384K  1.5G   1% /run
/dev/xvda1      9.8G  6.7G  2.7G  72% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            7.4G     0  7.4G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvda15     104M  4.7M   99M   5% /boot/efi
/dev/xvdb        64G  1.1G   60G   2% /mnt

smem says it's being used by the kernel:

# smem -tw
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory       9525544      92468    9433076
userspace memory             311064      15648     295416
free memory                 5564320    5564320          0
----------------------------------------------------------
                           15400928    5672436    9728492

But slabtop is no help:

# slabtop -o -s c
 Active / Total Objects (% used)    : 2915263 / 2937006 (99.3%)
 Active / Total Slabs (% used)      : 60745 / 60745 (100.0%)
 Active / Total Caches (% used)     : 68 / 103 (66.0%)
 Active / Total Size (% used)       : 356086.71K / 360884.30K (98.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.12K / 14.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
2226784 2226784 100%    0.07K  39764       56    159056K Acpi-ParseExt
273408 272598  99%    0.25K   8544       32     68352K kmalloc-256
  8568   8560  99%    4.00K   1071        8     34272K kmalloc-4096
 52320  52320 100%    0.50K   1635       32     26160K kmalloc-512
  1988   1975  99%    8.00K    497        4     15904K kmalloc-8192
 58044  53370  91%    0.19K   2764       21     11056K kmalloc-192
150016 141356  94%    0.06K   2344       64      9376K kmalloc-64
  5016   3504  69%    0.96K    152       33      4864K ext4_inode_cache
  7280   6834  93%    0.57K    260       28      4160K inode_cache
 20265  20067  99%    0.19K    965       21      3860K dentry
  1760   1721  97%    2.00K    110       16      3520K kmalloc-2048
 19800  19800 100%    0.11K    550       36      2200K sysfs_dir_cache
  2112   1966  93%    1.00K     66       32      2112K kmalloc-1024
   305    260  85%    6.00K     61        5      1952K task_struct
 14616  14242  97%    0.09K    348       42      1392K kmalloc-96
  2125   2092  98%    0.63K     85       25      1360K proc_inode_cache
  2324   2324 100%    0.55K     83       28      1328K radix_tree_node
  9828   9828 100%    0.10K    252       39      1008K buffer_head
  1400   1400 100%    0.62K     56       25       896K sock_inode_cache
    54     39  72%   12.00K     27        2       864K nvidia_stack_cache
   975    975 100%    0.81K     25       39       800K task_xstate
   690    515  74%    1.06K     23       30       736K signal_cache

So far, the only way I've been able to fix this is by rebooting. Where could 10GB of memory be hiding?

Chef-solo’s lack of environment support seems to be pretty controversial. On one hand, one of the features of environments is the ability to pin cookbooks to specific environments, and this makes absolutely no sense with chef-solo. On the other hand, lots of us would like to be able to merge in environment-level attributes and use environment-specific run lists when testing with Vagrant. And I think I may have found a simple way to hack around the problem. Assuming I use the JSON syntax in all my environments and roles, and I stick to the following convention for setting attributes (from lowest to highest precedence):

1. default in cookbook attributes files
2. default in role files
3. override in environment files

It looks like I can just parse these files myself and inject the attributes into chef-solo using the -j option. So for example, I could do something like this in my Vagrantfile:

chef_env_conf = parse_json("./environments/#{ENV['CHEF_ENVIRONMENT']}.json")
chef.json = chef_env_conf["override_attributes"]

Since attributes set via the -j option are applied at the normal priority level, this can be another way to override role defaults, and this might be all you need in a context where the environment isn’t pulled in automatically.

You could do something similar to get per-environment run lists (by parsing the role files):

chef_role_conf = parse_json("./roles/#{role}.json")
chef.run_list = chef_role_conf["env_run_lists"][ENV['CHEF_ENVIRONMENT']]

I realize this isn’t the most elegant hack, but it seems like it could be a workable solution for some people. Does anyone think it’s a really bad idea?

I’m thinking about converting my roles to JSON syntax and storing some extra data in there for an external, non-chef application (specifically Vagrant). My idea is that if I stick to a convention in which each machine only gets a single role, I can have Vagrant iterate through my roles directory and automatically configure a separate VM for each one.

The problem is that different roles may require different Vagrant settings (e.g. CPU, memory, forwarded ports, etc.), so I figured I could store all this stuff under a “vagrant” key in each role file. In my testing, I found that I can create any keys I want in these files, and the Chef server simply strips them off when you upload them. And this is fine, because the only thing that would ever need to look at them is Vagrant anyway (which simply parses the local files).

What does everyone think? Is this a bad idea? I can’t see how this could possibly hurt anything, but since I’ve never heard of anyone else doing anything like this, I figured I should ask around.

I'd like to create a header that contains the full URL of the request. So for example:

reqadd X-Request-URL:\ ${X-Forwarded-Proto}:\/\/${Host}${Path}

Is something like this possible?

I see tons of request errors on one of my frontend proxies (on the order of a few per second), but I can't figure out what's causing them. I've tried using the "show errors" command on the stats socket...

echo "show errors" | socat unix-connect:/var/run/haproxy.stat stdio

But this returns nothing. Debug logging doesn't give me any hints either. Is there some other place I should be looking?

Edit: Just to clarify, there is no "error message" per se (though it would sure help to have one). I'm just looking at the counter labeled "request errors" in the web interface and the socat output, and I'm trying to figure out what's incrementing it.

Given that it's impossible to "aptitude update" your way to a new kernel on an Amazon EC2 instance, I was wondering if it ever makes sense to apply kernel updates. My guess is no, and that I should have apt ignore all kernel updates (using the "aptitude hold" command or similar). Does anyone have any thoughts on this?