Michael Altfield's questions

How can I write the output of pidstat to a file for analysis later?

In sar, you can monitor usage and write-out to a file for later analysis using the -o argument. From the man page:

       If multiple samples and multiple reports are desired, it is convenient to specify an output file for the sar command.  Run the sar command as  a
       background process. The syntax for this is:

       sar -o datafile interval count >/dev/null 2>&1 &

       All  data are captured in binary form and saved to a file (datafile).  The data can then be selectively displayed with the sar command using the
       -f option. Set the interval and count parameters to select count records at interval second intervals. If the count parameter is  not  set,  all
       the  records saved in the file will be selected.  Collection of data in this manner is useful to characterize system usage over a period of time
       and determine peak usage hours.
...
       -o [ filename ]
              Save the readings in the file in binary form. Each reading is in a separate record. The default value of the filename  parameter  is  the
              current  daily data file, the /var/log/sa/sadd file. The -o option is exclusive of the -f option.  All the data available from the kernel
              are saved in the file (in fact, sar calls its data collector sadc with the option "-S ALL". See sadc(8) manual page).

I need to monitor the resource usage of a specific process by name. The equivalent tool for this is pidstat. Sadly, pidstat has no -o argument

[root@host ~]# pidstat -o pidstat.sadc -C "nginx" 1
Usage: pidstat [ options ] [ <interval> [ <count> ] ]
Options are:
[ -d ] [ -h ] [ -I ] [ -l ] [ -r ] [ -s ] [ -t ] [ -U [ <username> ] ] [ -u ]
[ -V ] [ -w ] [ -C <command> ] [ -p { <pid> [,...] | SELF | ALL } ]
[ -T { TASK | CHILD | ALL } ]
[root@host ~]# 

How can I tell pidstat to output to a file in sadc format so that it can be analyzed later with tools like kSAR?

What is the easiest way to tell varnish:

Hey Varnish processes, please clear your cache. And also please don't cache anything at all right now. I want you to just pass all requests straight to the backend for now. I'll let you know when I want you to go back to normal operation in a bit.

I'm doing some debugging and iterating on a website, but I'm tired of constantly clearing my cache with this command between every change & new request.

varnishadm 'ban req.url ~ "."'

Also, this is a fairly complex varnish config that has many backends for many distinct websites & domains.

I just want the simplest command that will tell varnish to temporarily force all requests to the backend.

# this command does not exist
varnishadm 'caching-pause req.url ~ "."'

And the corresponding simple command to restore varnish back to normal operation.

# this command also does not exist
varnishadm 'caching-resume req.url ~ "."'

Is there some easy way to tell varnish to temporarily stop caching everything?

How do I stop dovecot from telling its clients what flavour of linux I'm running and that it's a dovecot imap server?

If I connect to my smtp server, I see the following snippet

user@host:~$ openssl s_client -crlf -connect mail.example.com:993
CONNECTED(00000003)
...
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot (Debian) ready.
...
user@host:~$ 

Much to my dismay, the above output clearly indicates that I'm running a Debian server and that I'm using dovecot as my IMAP server.

In nginx, you can set server_tokens off to disable such information leakage.

In apache, it's

ServerTokens Prod
ServerSignature Off

What's the equivalent setting to tell dovecot not to leak the OS or imap server I'm running to clients?

Is there a command that I can issue that will tell me if the current set of config files for dovecot are valid or if there's an issue?

I just made a change to my dovecot config files. Before attempting to restart the service, I just want to do a quick sanity check to confirm if I made a fat-finger mistake or not.

I'm looking for the dovecot equivalent of any of the following:

nginx -t
apachectl -t
postfix check

Please let me know if there is some way to check the validity of my dovecot configuration.

How can I setup OpenVPN in conjunction with my firewall in such a way that my VPN users' traffic is DROPed by default to all network resources, and only ACCEPTed through the firewall if that user requires access to the specific resource?

I have an OpenVPN server running on an OpnSense firewall in our network. My understanding is that limiting the routes pushed to the VPN client by the VPN server (eg using route in server.conf) is not sufficient to limit access to the network as-needed, because the user can just add routes on their laptop after they connect.

To actually limit access to the network, we need to configure the firewall to ACCEPT packets going from a given VPN user to a given resource only as-needed, and default to DROP for everything else.

There's a few complexities with this:

1. A firewall doesn't have a notion of what a "VPN user" is. As far as I know, it only can match based on source IP Address

2. In fact, a "VPN user" could have multiple IP Addresses because we want our users to be able to connect to the VPN simultaneously with the same account from many devices (eg laptop, desktop, phone, tablet).

3. Let's say we grow to 1,000+ VPN users, how do we manage all of this in a sane way?

What's the best way to limit access to our private network from our OpenVPN clients on an as-needed basis (following the principal of least privilege) using OpnSense in a way that scales?

How do I change the Ubiquiti Security Gateway's default icmp restrictions from inside the LAN?

It seems that my Ubiquiti Security Gateway's default settings will drop icmp packets if I'm doing more than one traceroute at a time, but I can't find any setting anywhere in the Ubiquiti Controller's wui nor the Security Gateway's firewall rules that look like they're limiting icmp.

For example, when monitoring network issues, I like to kick-off a few simultaneous traceroutes to popular ping farms. Below I kick one off to Google's 8.8.8.8 and CloudFlare's 1.1.1.1 at the same time -- with the ping to Google in a terminal just below the ping to CloudFlare.

Note that the first traceroute has 100% packet loss from my Ubiquiti Security Gateway (ubnt) while the one below it has 0% packet loss. If I stop the traceroute on the bottom, then the other traceroute immediately goes from 100% packet loss to 0% packet loss. So this seems like some kind of overly sensitive icmp flood protection or rate limiting.

Where is this set in the Ubiquity Controller? How can I tune these icmp limits on the LAN to be something more sane?

How do I check what version of Proxmox my server is running?

Wikipedia tells me that Proxmox runs on a modified version of Ubuntu LTS

It is a Debian-based Linux distribution with a modified Ubuntu LTS kernel

But when I check /etc/issue as expected with a Debian-based release, I see only this:

host:~# cat /etc/issue

------------------------------------------------------------------------------

Welcome to the Proxmox Virtual Environment. Please use your web browser to
configure this server - connect to:

  https://192.168.0.5/

------------------------------------------------------------------------------

host:~#

How do I actually determine the OS version of my proxmox server using the command-line?

How do I check what OS is installed on my Synology NAS?

uname -a informed me that my NAS is:

aarch64 GNU/Linux synology_rtd1296_rs819

..but how do I know what version of the Synology OS it's running? Wikipedia tells me that Synology used to use an OS based on BSD, but today it's "Linux-based". I tried to get the version from all the usual places, but none of these files exist:

1. /etc/issue
2. /etc/redhat-release
3. /etc/release

How do I determine what is OS version of Disk Station Manager that my Synology NAS is currently running via the command-line?

How do I configure journald so that it never, ever, ever deletes my logs.

Yes, what I want to do is tell journald to keep writing until the disk (read: /var partition's filesystem) is full.

No, I don't want it to keep storing to memory until the RAM is full.

According to man 5 journald.conf,

SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=,
RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles=
    Enforce size limits on the journal files stored. The options
    prefixed with "System" apply to the journal files when stored on a
    persistent file system
...
    The first pair defaults to 10% and the second to 15% of the size of
    the respective file system, but each value is capped to 4G.

So the default is some percent of the filesystem with a cap at 4G. Unfortunately, I don't see any mention of how to set it to unlimited in the above man page.

How do I make journald never delete my logs? Bonus points for additional options that increase rotation & compression, etc.

I'd like to receive an immediate alert every time something reads the contents of my private key on Debian Linux. How can I do that?

I have an RSA private key stored on my server's disk at the following location:

/etc/ssl/private/super-secret.key

It's only readable by root, but I still want to have a log of every time this secret key has been read by a person or a process, and store the context of that read event for alerting and auditing purposes

How can I setup immediate alerting when a very sensitive file has been read from disk?

Is there some site that aggregates status updates from the companies that manage the transoceanic fiber cables so that I can get a world view of global internet outages that break connectivity between continents for millions of users in semi-real-time?

Several times per year, I end up on a network that's insufferably unusable due to packet loss, and traceroute shows that the issue isn't:

1. between my laptop and my WAN modem nor
2. between my WAN modem and the next few hops through my ISP's network

But the issue appears to be on some other pipe downstream from my ISP whoose DNS suggests it's operated by a tier 1 provider, such as Level3.net (Lumen Technologies).

Moreover, the packet loss is only detectable when using 10%-30%-ish of the internet, and it roughly appears to be regional--perhaps a transatlantic route.

That's about as much debugging as I can do with traceroute. The next thing I usually do is try to find a "status" page for the company whoose line is flapping, but all I can usually find on their website is content marketing trash.

How do I know if the transoceanic pipes are clogged or running smooth? Where is that magical dashboard that shows the world's Internet backbone status, per provider--or as close as we can hope for?

More generally, where can I go to find out if the largest operators of the world's fiber networks (Lumen, Verizon, Liberty Global, Tata, AT&T, etc) are experiencing issues that may be causing issues that affect millions or billions of people using the Internet right now?

How can I create a .zip compressed archive in Windows from the command-line that preserves symlinks?

I'm trying to create .zip file in PowerShell that includes a relative symlink from one file to another file in that archive, but I either encounter an error or the shortcut turns into a non-symlink file after extraction.

mkdir test
cd test
echo 'foo' > bar
cmd /C mklink bar_link bar
cd ..
Compress-Archive -DestinationPath test.zip -Path test

What are the commands I should use to create a .zip file from a directory that contains a relative symlink to a file in the directory to be archived?

How can I list all of the Docker Content Trust root keys on my system?

I am setting up a CI process that will use the debian:stable-latest docker image to build my application's releases in ephemeral cloud instances. I want to make sure that every time my fresh build system does a docker pull debian:stable-latest, it doesn't just blindly TOFU the root public key used to sign debian's docker images--thus defeating the entire security model of DCT.

Before downloading a given docker image, how can I check to see if the system already has the image's root public key or not?

I'm trying to curl a website and grep it for a specific line in PowerShell. How can I do that?

Here's the equivalent of what I'm trying to do, but in BASH

user@host:~$ curl -s www.isxkcdshittytoday.com | grep YES
title="YES">YES</a>
user@host:~$ 

But when I do what I'd expect to be the equivalent in PowerShell, I get the entire response--not just what matches my sls (grep) expression

PS C:\Users\user> curl -UseBasicParsing isxkcdshittytoday.com | sls -ca YES

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Is xkcd shitty today?</title>
<link rel="alternate" title="Is xkcd shitty today?" href="rss.xml" type="application/rss+xml" />

<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-26584170-2', 'isxkcdshittytoday.com');
  ga('send', 'pageview');
</script>

</head>

<body style="text-align: center; padding-top: 200px;">

<a href="rss.xml" style="font-weight: bold; font-size: 120pt;
font-family: Arial, sans-serif; text-decoration: none; color: black;"
title="YES">YES</a>

</body>
</html>



PS C:\Users\user>

Why is the above command on PowerShell not working like it should? How can I pipe something from curl into sls as an alternative to grep in PowerShell?

How can I tell nginx to output all incoming traffic for a given server{} to a specific/additional/redundant location after decrypting it (like port mirroring on routers) so that it can be analyzed for troubleshooting purposes?

I need to debug the traffic coming-in to an https-only website. The server is headless, and I'd rather

1. Not install a GUI just for wireshark
2. Not move my private keys off the server for later analysis on my local GUI with wireshark

So wireshark (the most commonly-cited solution for decrypting an https stream) is not a solution. I want something that's CLI-only and requires minimal tools done live on the server. I checked tcpdump's https support; it's absent but you can combine a ton of other tools and get it functional, but it's not exactly ideal.

My web server is nginx. I'm wondering if there's some way to setup nginx such that--in addition to the way it normally handles its traffic for a given server{} block--send the decrypted traffic to some mirror (just for debugging) that I can then analyze directly with tcpdump, ideally in real-time.

Is it possible to add a proxy_pass mirror to an nginx server{} block such that nginx handles the traffic as it usually does plus sending the decrypted traffic to some "debug" location (which could just be an nc -l 8080 or something)?

How can I get a space-separated list of the packages with pending updates on CentOS7 in the format that is required for passing as an argument to the yum update-to command?

I want to update all the packages on my production server running CentOS7. Before I update them on production, I want to update them on staging and verify that the changes don't break our server's applications. After validation, we'll schedule a change ticket for production and run that same/vetted yum update-to command (as opposed to the potentially non-deterministic & non-idempotent yum update command) on production at a later date.

So I'm trying to [a] determine all the packages that have an update available for them on my server and [b] prepare a command using yum update-to that will list exactly the packages and their versions to be updated.

Here's an example of the yum update-to command's required syntax:

yum update-to dbus-1.10.24-13.el7_6 dbus-libs-1.10.24-13.el7_6 java-1.8.0-openjdk-1.8.0.242.b08-1.el7 java-1.8.0-openjdk-headless-1.8.0.242.b08-1.el7 tzdata-java-2019c-1.el7 unzip-6.0-21.el7

The problem is that yum update --assumeno and yum list updates both output the list of packages in a totally distinct format than what yum update-to expects.

Here's an example of the above commands' output related to the packages in the example above:

[root@cent7 ~]# yum update --assumeno | grep -E 'dbus|java|unzip'
---> Package dbus.x86_64 1:1.6.12-17.el7 will be updated
---> Package dbus.x86_64 1:1.10.24-13.el7_6 will be an update
---> Package dbus-libs.x86_64 1:1.6.12-17.el7 will be updated
---> Package dbus-libs.x86_64 1:1.10.24-13.el7_6 will be an update
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.181-3.b13.el7_5 will be updated
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.242.b08-1.el7 will be an update
--> Processing Dependency: gtk2(x86-64) for package: 1:java-1.8.0-openjdk-1.8.0.242.b08-1.el7.x86_64
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.181-3.b13.el7_5 will be updated
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.242.b08-1.el7 will be an update
--> Processing Dependency: pcsc-lite-libs(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.242.b08-1.el7.x86_64
--> Processing Dependency: cups-libs(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.242.b08-1.el7.x86_64
---> Package tzdata-java.noarch 0:2017b-1.el7 will be updated
---> Package tzdata-java.noarch 0:2019c-1.el7 will be an update
---> Package unzip.x86_64 0:6.0-16.el7 will be updated
---> Package unzip.x86_64 0:6.0-21.el7 will be an update
 dbus                     x86_64 1:1.10.24-13.el7_6              base     245 k
 dbus-libs                x86_64 1:1.10.24-13.el7_6              base     169 k
 java-1.8.0-openjdk       x86_64 1:1.8.0.242.b08-1.el7           base     293 k
 java-1.8.0-openjdk-headless
 tzdata-java              noarch 2019c-1.el7                     base     187 k
 unzip                    x86_64 6.0-21.el7                      base     171 k
[root@cent7 ~]# 
[root@cent7 ~]# yum list updates | grep -E 'dbus|java|unzip'
dbus.x86_64                           1:1.10.24-13.el7_6                base    
dbus-libs.x86_64                      1:1.10.24-13.el7_6                base    
java-1.8.0-openjdk.x86_64             1:1.8.0.242.b08-1.el7             base    
java-1.8.0-openjdk-headless.x86_64    1:1.8.0.242.b08-1.el7             base    
tzdata-java.noarch                    2019c-1.el7                       base    
unzip.x86_64                          6.0-21.el7                        base    
[root@cent7 ~]# 

How do I get translate the output of the packages and their versions from yum update --assumeno or yum list updates into a format that is accepted by yum update-to?

Why does a fresh container on a small-ish docker image say that its disk is full of 10G when it's not?

I'm running this in a Debian 10 AppVM in QubesOS. In Debian 10, I do:

sudo apt-get -y install docker.io
sudo docker pull node:13-buster-slim

At the time of writing, this gives me docker v18.09.1 using the 'overlay2' storage driver by default.

root@coviz:~# sudo docker --version
Docker version 18.09.1, build 4c52b90
root@coviz:~# docker info | grep Storage
Storage Driver: overlay2
root@coviz:~# 

My docker host now has only this 181M docker image and no containers. The docker host is using only 0.5G out of 20G available. Plenty of free space.

root@coviz:~# sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@coviz:~# sudo docker image ls -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
node                13-buster-slim      e4217af9b7c7        9 days ago          181MB
root@coviz:~# sudo docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              1                   0                   180.7MB             180.7MB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B
root@coviz:~# 

I'm working on creating a Dockerfile for my project, so I execute the following command to spin up a new container from the above base image and drop me in a shell on that temporary container

root@coviz:~# docker run --rm -it --entrypoint /bin/bash e4217af9b7c7
root@97a318c599ab:/#

It isn't long before I encounter issues when testing out commands to install dependencies with apt-get. I think the issue is that apt needs to store cache data to /var/lib/apt/lists/. The actual error is at least one invalid signature was encountered, but it actually appears to be a disk fill issue (the apt key verification fails because it can't store the signature to disk). Running an apt-clean doesn't help; it's already empty. This is a fresh container based on a fresh image.

Checking the disk with df in this fresh container immediately shows that there's only 17M of disk space available, but I can only account for ~200M with du. Again, this is a fresh container, so I highly doubt this is an issue with a file stuck in a 'deleting' state still opened by a process.

root@97a318c599ab:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         9.6G  9.1G   17M 100% /
tmpfs            64M     0   64M   0% /dev
tmpfs           255M     0  255M   0% /sys/fs/cgroup
/dev/xvda3      9.6G  9.1G   17M 100% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           285M     0  285M   0% /proc/asound
tmpfs           285M     0  285M   0% /proc/acpi
tmpfs           285M     0  285M   0% /proc/scsi
tmpfs           285M     0  285M   0% /sys/firmware
root@97a318c599ab:/# du -sh /*
4.8M    /bin
4.0K    /boot
0   /dev
612K    /etc
20K /home
12M /lib
4.0K    /lib64
4.0K    /media
4.0K    /mnt
5.2M    /opt
du: cannot access '/proc/11/task/11/fd/4': No such file or directory
du: cannot access '/proc/11/task/11/fdinfo/4': No such file or directory
du: cannot access '/proc/11/fd/4': No such file or directory
du: cannot access '/proc/11/fdinfo/4': No such file or directory
0   /proc
136K    /root
8.0K    /run
4.1M    /sbin
4.0K    /srv
0   /sys
2.2M    /tmp
160M    /usr
5.9M    /var
root@97a318c599ab:/# 

Moreover, docker ps -s shows that the size of the "writable layer" on my container is empty (0B):

root@coviz:~# docker ps -a -s
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES               SIZE
320af1498086        e4217af9b7c7        "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes                           epic_leakey         0B (virtual 181MB)
root@coviz:~# 

So why is this fresh docker container's disk (based on a ~200M image) full? What's taking-up those ~9G of unaccounted space?

Does varnish support unix socket files?

I'd like to accelerate my website by adding a varnish reverse proxy (HTTP accelerator/cache) before my web server. My web server's virtual host is currently configured to listen for connections on a "UNIX-domain socket". For more info on this config, see the "path" argument for the "listen" directive in NGINX:

• https://nginx.org/en/docs/http/ngx_http_core_module.html#listen

Is it possible for varnish to communicate to my existing nginx server's config over a unix domain socket file as its backend?

And is it possible for varnish to also itself listen on a domain socket?

how do I delete a table in iptables (as opposed to a chain)?

I have some empty tables that are getting output by iptables-save even though I'm only using the 'filter' table.

For example, I'd like iptables-save to not produce any output regarding the 'mangle' table. Today I was playing around with iptables, and I used the mangle table. My output of iptables-save used to look like this:

# Generated by iptables-save v1.6.0 on Thr Jun 21 00:00:00 2018
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -j DROP
COMMIT
# Completed on Thr Jun 21 00:00:00 2018

But now it looks like this:

# Generated by iptables-save v1.6.0 on Sat Jun 23 00:00:00 2018
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Sat Jun 23 00:00:00 2018
# Generated by iptables-save v1.6.0 on Sat Jun 23 00:00:00 2018
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -j DROP
COMMIT
# Completed on Sat Jun 23 00:00:00 2018

How do I delete this unused 'mangle' table to cleanup my iptables-save output?

How can I validate that I've properly setup my Apache 2.4.6 server with a custom 2048-bit (or 4096-bit) dhparams config?

Following the weakdh.org sysadmin guide, I created my own dhparams.pem file with openssl dhparam -out dhparams.pem 2048. The guide says to add this to the apache mod_ssl config with SSLOpenSSLConfCmd DHParameters "{path to dhparams.pem}", but this is only valid for Apache >= v2.4.7. I'm using CentOS 7, which uses Apache v2.4.6.

According to this server fault question, the solution in Apache v2.4.6 is to append it to the certificate file. So I did a cat /etc/pki/dhparam/dhparam.pem >> /etc/letsencrypt/live/openbuildinginstitute.org/cert.pem (and also cat /etc/pki/dhparam/dhparam.pem >> /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem` && restarted apache.

But how do I verify from the client-side (my browser) that this config is in effect?

This is especially an issue as we use Let's Encrypt, so we want to make sure our 90-day cert renewals include this step, and I want to be able to verify it from the browser.

I already tried downloading the certificate with firefox's "View Certificate" -> "Details" -> "Export...", and I confirmed that the "-----BEGIN DH PARAMETERS-----" was absent from the resulting file.