strugee's questions

My company has Nginx deployed from upstream's repositories. I would ideally like to include this repository in unattended-upgrades, but I'm concerned that this will break things at some point.

I understand that this carries the risk of bugs, which is a tradeoff I'm willing to make, but what's unclear to me is whether Nginx has any kind of configuration syntax/semantics backwards compatibility guarantee (we don't use third party modules so this is the only kind of backwards compatibility I care about). I've read What’s the difference between the “mainline” and “stable” branches of nginx? and the linked blog post but didn't see anything. Even if I use the stable repository, it seems that when mainline is forked and merged with stable I'm still risking breakage in the absence of a stability promise. Is that interpretation correct? Or am I missing some Nginx policy somewhere or possibly even approaching this problem completely incorrectly?

I'd like to append to the default Postfix value of local_recipient_maps, defined as:

% postconf | grep '^local_recipient_maps'
postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_sender_restrictions
postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_client_restrictions
postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_helo_restrictions
local_recipient_maps = proxy:unix:passwd.byname $alias_maps

Based on the documentation it seemed like I should be able to just reference the default value in the new value, so I did:

# postconf 'local_recipient_maps = $local_recipient_maps hash:/var/lib/mailman3/data/postfix_lmtp'

but when I run postconf again, it gives me a macro error:

% postconf | grep '^local_recipient_maps'
postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_sender_restrictions
postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_client_restrictions
postconf: warning: /etc/postfix/master.cf: undefined parameter: mua_helo_restrictions
postconf: warning: unreasonable macro call nesting: "local_recipient_maps"
postconf: fatal: macro processing error

Obviously this is not a big deal because I can just copy and paste the original default value, but I'd like to know if I'm doing something wrong/if this is (supposed to be) possible because it makes for a nice way to separate different parts of parameters into sections with a comment header explaining what they're for (e.g. this one is going into a # Mailman section, because obviously only the last map in local_recipient_maps is related to Mailman).

If it matters, I am on Postfix 3.4.10-0+deb10u1 (according to apt policy postfix) on Debian 10.

I have a hand-maintained Bugzilla installation that I'm trying to move to being managed with SaltStack. Currently I have the following .sls file to set up the clone:

https://github.com/bugzilla/bugzilla.git:
  git.latest:
    - rev: release-5.0-stable
    - branch: release-5.0-stable
    - target: /srv/http/bugzilla
    - require:
      - pkg: git

My current install has the permissions locked down to basically root:www-data, with owner read-write and group read-only. (IIRC, the Bugzilla install guide said to do this.) I'd like to replicate this configuration with the Salt clone, but can't seem to figure it out. salt.states.git has an option for the user to run the clone as, but nothing about the group (and besides, you can't fiddle with file modes either). It seems like the Right Way™ to do this is to use a salt.states.file state, which requires the git.latest state, but all the file states in the documentation copy stuff from the master, which is not what I want.

How can I set the owner/group/permissions on my new clone?

I was just reading How bad is IPv4 address exhaustion really? and noticed this comment, which seems to imply that carrier grade NAT is actually widely deployed. I was always under the impression that very few ISPs deployed carrier grade NAT and instead just bit the bullet and deployed IPv6 (though now that I think about it I don't know where I got that idea from).

Is that not the case?

Bugzilla upstream ships a .htaccess file that contains, among other things, this snippet of Apache configuration:

# Don't allow people to retrieve non-cgi executable files or our private data
<FilesMatch (\.pm|\.pl|\.tmpl|localconfig.*)$>
  <IfModule mod_version.c>
    <IfVersion < 2.4>
      Deny from all
    </IfVersion>
    <IfVersion >= 2.4>
      Require all denied
    </IfVersion>
  </IfModule>
  <IfModule !mod_version.c>
    Deny from all
  </IfModule>
</FilesMatch>

I'm running Apache 2.4; however I get 500 errors about Deny whenever I visit the Bugzilla homepage. Turns out that Apache is getting Deny from all from inside the <IfModule !mod_version.c> block, and changing this to Require all denied fixes the errors. Other <IfModule> blocks for e.g. mod_rewrite work OK.

Why is Apache behaving this way? I've searched the web pretty extensively with no luck.

Here's some more info about my system:

% sudo apachectl -M | grep version
 version_module (static)

% uname -a # Debian 8
Linux steevie 4.9.0-0.bpo.3-amd64 #1 SMP Debian 4.9.30-2+deb9u2~bpo8+1 (2017-06-27) x86_64 GNU/Linux

% apachectl -v 
Server version: Apache/2.4.10 (Debian)
Server built:   Jul 18 2017 18:32:16

I have a production MongoDB 2.4 that's giving this error in the logs:

SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:37934]

The client's logs report a connection timeout.

mongod --version reports v2.4.10 (I'm on Debian jessie; MongoDB is installed from the jessie repositories) and the net.ipv4.tcp_keepalive_time sysctl is set to 300 as recommended by SocketException in Mongo logs. Also, the error is still present when triggering a database query directly after restarting MongoDB and the application, which to me suggests that this is not SERVER-5632 (mentioned in the above question).

/proc/pid/limits reports unlimited for everything except stack size, core file size, processes, locked memory, pending signals, msgqueue size, nice and realtime priority, and open files. This last one in particular has a soft limit of 1024 and a hard limit of 4096, which isn't what the MongoDB docs recommend but shouldn't be a problem anyway since ls /proc/pid/fd | wc -l reports only 29 open fds.

free -g reports 3 GB of free memory, +/- buffers and cache. Rebooting the entire server seems to help somewhat for a short time, but then it gets just as bad again. Here's a snippet from the log:

Wed Mar  1 20:14:17.466 [conn1]  authenticate db: pumpio { authenticate: 1, user: "pump.io", nonce: "7ac439ffd4c01a6d", key: "abe7862c7eb1f9765ebaa007d7c20ec2" }
Wed Mar  1 20:14:26.445 [conn1] command pumpio.$cmd command: { authenticate: 1, user: "pump.io", nonce: "7ac439ffd4c01a6d", key: "abe7862c7eb1f9765ebaa007d7c20ec2" } ntoreturn:1 keyUpdates:0 locks(micros) W:8978364 r:412 reslen:74 8979ms
Wed Mar  1 20:14:41.809 [initandlisten] connection accepted from 127.0.0.1:58216 #2 (2 connections now open)
Wed Mar  1 20:14:46.302 [initandlisten] connection accepted from 127.0.0.1:58218 #3 (3 connections now open)
Wed Mar  1 20:14:49.212 [initandlisten] connection accepted from 127.0.0.1:58220 #4 (4 connections now open)
Wed Mar  1 20:14:49.212 [initandlisten] connection accepted from 127.0.0.1:58222 #5 (5 connections now open)
Wed Mar  1 20:14:49.212 [initandlisten] connection accepted from 127.0.0.1:58224 #6 (6 connections now open)
Wed Mar  1 20:14:49.213 [initandlisten] connection accepted from 127.0.0.1:58226 #7 (7 connections now open)
Wed Mar  1 20:14:49.213 [initandlisten] connection accepted from 127.0.0.1:58228 #8 (8 connections now open)
Wed Mar  1 20:14:49.213 [initandlisten] connection accepted from 127.0.0.1:58230 #9 (9 connections now open)
Wed Mar  1 20:14:49.213 [initandlisten] connection accepted from 127.0.0.1:58232 #10 (10 connections now open)
Wed Mar  1 20:14:49.213 [initandlisten] connection accepted from 127.0.0.1:58234 #11 (11 connections now open)
Wed Mar  1 20:14:49.213 [initandlisten] connection accepted from 127.0.0.1:58236 #12 (12 connections now open)
Wed Mar  1 20:14:50.685 [conn1] query pumpio.session query: { _id: "9n_7gseqgT8i07tQbnBj7gNgURdQNmS8" } ntoreturn:1 idhack:1 keyUpdates:0 locks(micros) r:13911898 reslen:202 17129ms
Wed Mar  1 20:14:50.695 [conn2]  authenticate db: pumpio { authenticate: 1, user: "pump.io", nonce: "7e918deb40e23f92", key: "0b4b2d78fa404e594dc1d066c9402cc4" }
Wed Mar  1 20:14:52.960 [conn1] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:58214]
Wed Mar  1 20:14:52.960 [conn7] assertion 16550 not authorized for query on reportr.system.indexes ns:reportr.system.indexes query:{ ns: "reportr.alerts" }
Wed Mar  1 20:14:52.960 [conn12] assertion 16550 not authorized for query on reportr.system.indexes ns:reportr.system.indexes query:{ ns: "reportr.alerts" }
Wed Mar  1 20:14:52.960 [conn5] assertion 16550 not authorized for query on reportr.system.indexes ns:reportr.system.indexes query:{ ns: "reportr.events" }
Wed Mar  1 20:14:52.960 [conn10] assertion 16550 not authorized for query on reportr.system.indexes ns:reportr.system.indexes query:{ ns: "reportr.events" }
Wed Mar  1 20:14:58.713 [initandlisten] connection accepted from 127.0.0.1:58238 #13 (12 connections now open)
Wed Mar  1 20:14:58.715 [conn13]  authenticate db: pumpio { authenticate: 1, user: "pump.io", nonce: "fe508e5612e933fa", key: "492c9d11f582c045941da7f7eb16b8bd" }
Wed Mar  1 20:15:07.599 [conn2] update pumpio.session query: { _id: "i_vJ67QJA56LsQvLMF8GuiosMHUj7--P" } update: { cookie: { originalMaxAge: null, expires: null, httpOnly: true, path: "/" }, _id: "i_vJ67QJA56LsQvLMF8GuiosMHUj7--P" } idhack:1 nupdated:1 upsert:1 keyUpdates:0 locks(micros) w:16896964 16897ms
Wed Mar  1 20:15:07.600 [conn2] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:58216]
Wed Mar  1 20:15:07.600 [conn13] command pumpio.$cmd command: { authenticate: 1, user: "pump.io", nonce: "fe508e5612e933fa", key: "492c9d11f582c045941da7f7eb16b8bd" } ntoreturn:1 keyUpdates:0 locks(micros) r:1225 reslen:74 8885ms
Wed Mar  1 20:15:08.050 [conn13] update pumpio.session query: { _id: "deiIhAPc1vaXYZ_JzU72slghiCMOu1Zf" } update: { cookie: { originalMaxAge: null, expires: null, httpOnly: true, path: "/" }, _id: "deiIhAPc1vaXYZ_JzU72slghiCMOu1Zf" } idhack:1 nupdated:1 upsert:1 keyUpdates:0 locks(micros) w:109 447ms
Wed Mar  1 20:15:15.592 [TTLMonitor] query local.system.indexes query: { expireAfterSeconds: { $exists: true } } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:142680 nreturned:0 reslen:20 142ms
Wed Mar  1 20:15:30.773 [conn13] query pumpio.recentdialbackrequests query: { _id: 0 } ntoreturn:1 idhack:1 keyUpdates:0 locks(micros) r:2751358 reslen:95317 2751ms
Wed Mar  1 20:16:14.855 [conn13] query pumpio.comment query: { _uuid: "oudHqMliREq5HSXd-7N62Q" } ntoreturn:1000 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:3247907 nreturned:1 reslen:900 4951ms
Wed Mar  1 20:16:14.855 [conn13] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:58238]

This is right after I rebooted the MongoDB server and the application server. Both of these are on the same host and the application connects to localhost to issue database queries.

Any idea what's going on?

I have a system that's set up with a four-disk RAID1 array based on btrfs. Two disks are 1TB traditional HDDs, and the other two are 128GB SSDs. Most of each disk is filled up with a LUKS container. Inside all four LUKS containers I have a single btrfs filesystem that does the RAID1.

The above configuration should get pretty decent performance, but unfortunately, it's super slow. Here are some benchmarks, run generally around the same time:

% dd if=/dev/zero of=tmpfile bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.18302 s, 908 MB/s
% dd if=/dev/zero of=tmpfile bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.66737 s, 644 MB/s
% dd if=tmpfile of=/dev/null bs=1M count=1024     
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.61369 s, 665 MB/s
% echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
% dd if=tmpfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 8.05449 s, 133 MB/s
% echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
% dd if=/dev/zero of=tmpfile bs=1M count=1024           
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.66874 s, 402 MB/s
% echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
% dd if=tmpfile of=/dev/null bs=1M count=1024           
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.164 s, 208 MB/s

Not really sure what to make of them, but the disk certainly seems slow. And generally when I look at iotop, I see write throughputs of 5-10 MB/s or so. Not so great.

Here's some more info:

% cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       834853 iterations per second
PBKDF2-sha256     548418 iterations per second
PBKDF2-sha512     366122 iterations per second
PBKDF2-ripemd160  508031 iterations per second
PBKDF2-whirlpool  175229 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   379.4 MiB/s  1552.0 MiB/s
 serpent-cbc   128b    49.3 MiB/s   216.5 MiB/s
 twofish-cbc   128b   129.3 MiB/s   258.3 MiB/s
     aes-cbc   256b   325.3 MiB/s  1158.4 MiB/s
 serpent-cbc   256b    65.0 MiB/s   213.1 MiB/s
 twofish-cbc   256b   136.3 MiB/s   259.8 MiB/s
     aes-xts   256b  1326.8 MiB/s  1333.5 MiB/s
 serpent-xts   256b   224.4 MiB/s   216.7 MiB/s
 twofish-xts   256b   255.5 MiB/s   257.2 MiB/s
     aes-xts   512b  1034.8 MiB/s  1009.7 MiB/s
 serpent-xts   512b   225.8 MiB/s   214.1 MiB/s
 twofish-xts   512b   255.1 MiB/s   257.1 MiB/s
% lsmod | grep aes  
aesni_intel           167936  18 
aes_x86_64             20480  1 aesni_intel
lrw                    16384  5 serpent_sse2_x86_64,aesni_intel,serpent_avx_x86_64,twofish_avx_x86_64,twofish_x86_64_3way
glue_helper            16384  5 serpent_sse2_x86_64,aesni_intel,serpent_avx_x86_64,twofish_avx_x86_64,twofish_x86_64_3way
ablk_helper            16384  4 serpent_sse2_x86_64,aesni_intel,serpent_avx_x86_64,twofish_avx_x86_64
cryptd                 20480  9 ghash_clmulni_intel,aesni_intel,ablk_helper
% uname -a
Linux steevie 4.7.0-0.bpo.1-amd64 #1 SMP Debian 4.7.5-1~bpo8+1 (2016-09-30) x86_64 GNU/Linux

As you can see, I'm on Debian 8 running a kernel from jessie-backports.

Also, at the suggestion of https://askubuntu.com/questions/246102/slow-ssd-dm-crypt-with-luks-encryption-in-ubuntu-12-10, I had the initramfs load the cryptd, aes_x86_64, and aesni_intel modules during early boot. However, rebooting and benchmarking again gave almost the exact same speed.

Any idea where this terrible performance could be coming from?

I'm interning at a company which runs in an AWS environment and is starting to look into locking down user privileges, so I'm looking into ways to secure EC2 instances. Specifically, I want to find a way to avoid handing out a keypair that grants unconditional access to ec2-user, which with the default Amazon Linux sudo policy, is essentially root and has the additional downside that there's no audit log (so if someone runs something malicious while SSH'd into an EC2 instance, there's no way to tell who because the Linux user executing commands is always ec2-user).

My first thought was to do this via PAM, similar to how LDAP is integrated into PAM via pam_ldap or somesuch, but I can't find anything that will let me use IAM as an auth backend. I could just manually add users, since there's a very small amount of people that actually need access, but that seems prone to human error as well as being the type of thing that will inevitably become more and more inconsistent over time.

I've searched the web as well with no luck.

What's the best practice here?

I'm running Tomcat 6. I was messing around with my keystore in order to try to fix some certificate chain problems, and I accidentally rm'd the production version instead of the .new file I was working with.

Now, Tomcat is still running and still able to serve HTTPS connections (I tested it in a web browser). Obviously there must be a copy of the private key that it still has access to. My first thought was to check /proc/PID/fd, so I ran sudo lsof | grep tomcat. This properly returns a list of open files:

$ sudo lsof | head -1; sudo lsof | grep tomcat
COMMAND     PID     USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
java       2599     root  cwd       DIR              202,1      4096     413699 /usr/share/apache-tomcat-6.0.44/bin
java       2599     root  mem       REG              202,1     51715     413777 /usr/share/apache-tomcat-6.0.44/lib/tomcat-i18n-fr.jar
java       2599     root  mem       REG              202,1    528303     413771 /usr/share/apache-tomcat-6.0.44/lib/jasper.jar
java       2599     root  mem       REG              202,1   1250454     413767 /usr/share/apache-tomcat-6.0.44/lib/catalina.jar
java       2599     root  mem       REG              202,1    132216     413765 /usr/share/apache-tomcat-6.0.44/lib/catalina-ha.jar
java       2599     root  mem       REG              202,1     54334     413778 /usr/share/apache-tomcat-6.0.44/lib/tomcat-i18n-ja.jar
java       2599     root  mem       REG              202,1   1830791     413768 /usr/share/apache-tomcat-6.0.44/lib/ecj-4.3.1.jar
java       2599     root  mem       REG              202,1     54624     413764 /usr/share/apache-tomcat-6.0.44/lib/catalina-ant.jar
java       2599     root  mem       REG              202,1     70563     413776 /usr/share/apache-tomcat-6.0.44/lib/tomcat-i18n-es.jar
java       2599     root  mem       REG              202,1    112521     413770 /usr/share/apache-tomcat-6.0.44/lib/jasper-el.jar
java       2599     root  mem       REG              202,1     76649     413772 /usr/share/apache-tomcat-6.0.44/lib/jsp-api.jar
java       2599     root  mem       REG              202,1     34622     413769 /usr/share/apache-tomcat-6.0.44/lib/el-api.jar
java       2599     root  mem       REG              202,1    253635     413775 /usr/share/apache-tomcat-6.0.44/lib/tomcat-dbcp.jar
java       2599     root  mem       REG              202,1     15250     413763 /usr/share/apache-tomcat-6.0.44/lib/annotations-api.jar
java       2599     root  mem       REG              202,1    801374     413774 /usr/share/apache-tomcat-6.0.44/lib/tomcat-coyote.jar
java       2599     root  mem       REG              202,1    239165     413766 /usr/share/apache-tomcat-6.0.44/lib/catalina-tribes.jar
java       2599     root  mem       REG              202,1    132240     413773 /usr/share/apache-tomcat-6.0.44/lib/servlet-api.jar
java       2599     root  mem       REG              202,1     32384     413759 /usr/share/apache-tomcat-6.0.44/bin/tomcat-juli.jar
java       2599     root  mem       REG              202,1     24283     413753 /usr/share/apache-tomcat-6.0.44/bin/commons-daemon.jar
java       2599     root  mem       REG              202,1     22807     413749 /usr/share/apache-tomcat-6.0.44/bin/bootstrap.jar
java       2599     root    1w      REG              202,1  17024956     413713 /usr/share/apache-tomcat-6.0.44/logs/catalina.out
java       2599     root    2w      REG              202,1  17024956     413713 /usr/share/apache-tomcat-6.0.44/logs/catalina.out
java       2599     root    4r      REG              202,1     22807     413749 /usr/share/apache-tomcat-6.0.44/bin/bootstrap.jar
java       2599     root    5r      REG              202,1     24283     413753 /usr/share/apache-tomcat-6.0.44/bin/commons-daemon.jar
java       2599     root    6r      REG              202,1     32384     413759 /usr/share/apache-tomcat-6.0.44/bin/tomcat-juli.jar
java       2599     root    7w      REG              202,1      4072     396419 /usr/share/apache-tomcat-6.0.44/logs/catalina.2016-04-21.log
java       2599     root    8w      REG              202,1      1182     396447 /usr/share/apache-tomcat-6.0.44/logs/localhost.2016-04-21.log
java       2599     root    9w      REG              202,1         0     396448 /usr/share/apache-tomcat-6.0.44/logs/manager.2016-04-21.log
java       2599     root   10w      REG              202,1         0     396455 /usr/share/apache-tomcat-6.0.44/logs/host-manager.2016-04-21.log
java       2599     root   11r      REG              202,1    239165     413766 /usr/share/apache-tomcat-6.0.44/lib/catalina-tribes.jar
java       2599     root   12r      REG              202,1    801374     413774 /usr/share/apache-tomcat-6.0.44/lib/tomcat-coyote.jar
java       2599     root   13r      REG              202,1    253635     413775 /usr/share/apache-tomcat-6.0.44/lib/tomcat-dbcp.jar
java       2599     root   14r      REG              202,1    132240     413773 /usr/share/apache-tomcat-6.0.44/lib/servlet-api.jar
java       2599     root   15r      REG              202,1     34622     413769 /usr/share/apache-tomcat-6.0.44/lib/el-api.jar
java       2599     root   16r      REG              202,1     76649     413772 /usr/share/apache-tomcat-6.0.44/lib/jsp-api.jar
java       2599     root   17r      REG              202,1    112521     413770 /usr/share/apache-tomcat-6.0.44/lib/jasper-el.jar
java       2599     root   18r      REG              202,1     70563     413776 /usr/share/apache-tomcat-6.0.44/lib/tomcat-i18n-es.jar
java       2599     root   19r      REG              202,1     54624     413764 /usr/share/apache-tomcat-6.0.44/lib/catalina-ant.jar
java       2599     root   20r      REG              202,1   1830791     413768 /usr/share/apache-tomcat-6.0.44/lib/ecj-4.3.1.jar
java       2599     root   21r      REG              202,1     54334     413778 /usr/share/apache-tomcat-6.0.44/lib/tomcat-i18n-ja.jar
java       2599     root   22r      REG              202,1     15250     413763 /usr/share/apache-tomcat-6.0.44/lib/annotations-api.jar
java       2599     root   23r      REG              202,1    132216     413765 /usr/share/apache-tomcat-6.0.44/lib/catalina-ha.jar
java       2599     root   24r      REG              202,1   1250454     413767 /usr/share/apache-tomcat-6.0.44/lib/catalina.jar
java       2599     root   25r      REG              202,1    528303     413771 /usr/share/apache-tomcat-6.0.44/lib/jasper.jar
java       2599     root   26r      REG              202,1     51715     413777 /usr/share/apache-tomcat-6.0.44/lib/tomcat-i18n-fr.jar
bash      29579 ec2-user  cwd       DIR              202,1      4096     413708 /usr/share/apache-tomcat-6.0.44/conf
sudo      30595     root  cwd       DIR              202,1      4096     413708 /usr/share/apache-tomcat-6.0.44/conf
grep      30596 ec2-user  cwd       DIR              202,1      4096     413708 /usr/share/apache-tomcat-6.0.44/conf
lsof      30597     root  cwd       DIR              202,1      4096     413708 /usr/share/apache-tomcat-6.0.44/conf
lsof      30598     root  cwd       DIR              202,1      4096     413708 /usr/share/apache-tomcat-6.0.44/conf

but unfortunately, my keystore isn't one of them. I guess Tomcat must have loaded it into memory and then closed the file descriptor. (The keystore was in /conf above, but those file descriptors are from my open shell, not from Tomcat itself.)

So: is there any way to recover my private key? I'm on Amazon Linux; uname -a returns Linux ip-10-0-0-105 4.4.5-15.26.amzn1.x86_64 #1 SMP Wed Mar 16 17:15:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux.

I have a really weird problem with my DNS. My domain name (strugee.net) is unresolvable from some networks, and resolvable from others.

For example, on my home network (same network the server's on):

% dig strugee.net

; <<>> DiG 9.10.3-P4 <<>> strugee.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10086
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;strugee.net.           IN  A

;; ANSWER SECTION:
strugee.net.        1800    IN  A   216.160.72.225

;; Query time: 186 msec
;; SERVER: 205.171.3.65#53(205.171.3.65)
;; WHEN: Sat Apr 16 15:42:36 PDT 2016
;; MSG SIZE  rcvd: 56

However, if I log in to a server I have on Digital Ocean, the domain fails to resolve:

% dig strugee.net      

; <<>> DiG 9.9.5-9+deb8u3-Debian <<>> strugee.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 58551
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;strugee.net.           IN  A

;; Query time: 110 msec
;; SERVER: 2001:4860:4860::8844#53(2001:4860:4860::8844)
;; WHEN: Sat Apr 16 18:44:25 EDT 2016
;; MSG SIZE  rcvd: 40

But, going directly to the authoritative nameservers works just fine:

% dig @dns1.registrar-servers.com strugee.net   

; <<>> DiG 9.9.5-9+deb8u3-Debian <<>> @dns1.registrar-servers.com strugee.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30856
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;strugee.net.           IN  A

;; ANSWER SECTION:
strugee.net.        1800    IN  A   216.160.72.225

;; AUTHORITY SECTION:
strugee.net.        1800    IN  NS  dns3.registrar-servers.com.
strugee.net.        1800    IN  NS  dns4.registrar-servers.com.
strugee.net.        1800    IN  NS  dns2.registrar-servers.com.
strugee.net.        1800    IN  NS  dns1.registrar-servers.com.
strugee.net.        1800    IN  NS  dns5.registrar-servers.com.

;; Query time: 3 msec
;; SERVER: 216.87.155.33#53(216.87.155.33)
;; WHEN: Sat Apr 16 18:46:36 EDT 2016
;; MSG SIZE  rcvd: 172

It's pretty clear that there's a problem with some large network somewhere that's failing to resolve my domain, but I can't seem to figure out where. I skimmed the dig manpage for options that might help, but didn't find anything particularly useful.

I'm on Namecheap both as a domain registrar as well as DNS hosting. I have the DNSSEC option turned on. I haven't made any changes to my DNS settings recently.

How can I debug this problem and find the offending nameserver?

I have set up my DNS with afraid.org. I have an A record ssh.strugee.net pointing towards 71.32.87.80. I do not have any AAAA records set up.

When I run ping ssh.strugee.net or traceroute ssh.strugee.net, they complete DNS resolution. However, when I ssh into ssh.strugee.net, DNS resolution times out. ping6 and traceroute6 fail similarly. If I directly put the IP that the DNS record points to, the operation completes.

Because of this I'm pretty sure the problem is that I don't support IPv6 somehow, but I don't know how to continue my diagnostics because I don't know how the network between me and my server is constructed, and I don't know how IPv6 and IPv4 operate in that network. Is there a generic way that I can find this out? How should I continue trying to resolve this issue?

It's worth noting that my ISP is CenturyLink and I have ran the check-ipv6 tests on the same ISP but in a different geographical location. It says that I have no IPv6 address. I use Google Public DNS and OS X. In the OS X DNS priority list, Google DNS IPv6 is first, followed by Google DNS IPv4, followed by the default set of DNS servers (CenturyLink's).

I've manually installed a service called Gate One into /opt. I want to harden its security, so I thought I'd create a system account for it to use, because of least necessary privileges, and all that.

However, when I ran adduser --system gateone, it created a home directory, which I didn't really want. Therefore, I ran adduser --system --home-dir /opt/gateone gateone, because I'd seen some system accounts setting home directories pertaining to them. However, this made the login shell /bin/sh instead of /bin/false, which it had done the first time (when I didn't specify a home directory). I'm a bit confused now on what I should set these fields as.

tl;dr: when creating a system account on a *NIX operating system, what are the best practices for setting the home directory, the login shell, and anything else that's relevant? Why?

_{Note: I'm using Ubuntu 13.04 Raring, if it matters, since IIRC adduser can vary quite a bit distribution-to-distribution.}

After reading What solutions exist to allow the use of revision control for server configuration files?, I decided to install etckeeper on my server.

However, it's a one-server setup, so I only have one physical machine and no load balancer, etc. because there's only one destination to route traffic to (plus I don't have a second machine to do the job).

I assume that it's still worth it to use etckeeper, because it's tracking the local machine.

My real question is whether it's worth it to "upgrade" to something like Puppet or Chef, given that I only have one server. And if so, why?

I'm looking to install Howler on my Ubuntu Server. However, all that page provides is a RPM, not a DEB. Is there a version of Howler available for Debian-based distributions?

I just came across Something is burning in the server room; how can I quickly identify what it is?. In the comments I found the following quote:

you don't let a developer anywhere near your root passwords

As a developer but also someone very interested in sysadmin stuff, I'm wondering if there's anything to this phrase more than just the standard for everyone - that is, don't give away passwords? The commentor states this as though it's fairly common knowledge in the sysadmin community. Is it?