gertvdijk's questions

I have Postfix running with a number of smtpd processes configured in master.cf like this:

# Internet facing one
1.2.3.4:25      inet  n       -       y       -       -       smtpd
  -o ...   # internet-only overrides

# Internal facing one
10.0.0.1:10026      inet  n       -       y       -       -       smtpd
  -o ...   # internal-only overrides

Now, I'd like to add a single header, with static name and value, to incoming mail depending on which smtpd it was received on.

Example:

X-Gert-Postfix-Received-From: the evil internet

My options considered:

1. Add the header_checks option and use the PREPEND action in the file.

   Nearly there, but:

   • It requires to match an existing header and will then add one more on subsequent matches.
   • I don't always have a certain header present already, perhaps even a From is missing, for example.
   • In case you have existing header_checks, there's no easy way to stack two header_check files, I think.

2. Build a custom app that uses the Milter protocol and hook that up to Postfix with smtpd_milters.

   Of course, this will work. I can inspect the mail in my own app, then inject the header there. Seems over-engineering for a simple task like adding a header. Additionally, it requires extra maintenance with the need to run another daemon app, quite some boilerplate code, etc.

3. As suggested in a comment, use check_recipient_access (related Q).

   Same downsides as header_checks (see 1).

I feel like I'm missing something simple. Anyone got a better idea?

I have:

• A commodity Linux server, in this I have installed:
• 1x HBA storage adapter Broadcom (LSI) 9500-8e having 2x mini-SAS-HD connector, connected to:
• 1x Storage enclosure with dual input mini-SAS-HD connector, SAS-3 capable, using:
• 2x mini-SAS-HD cables running between the controller and the enclosure.
• A bunch of SAS 3.0 hard disks in the enclosure.

The enclosure and disks work just fine with a single cable attached, but they can optionally be connected with two. Using two cables is recommended by the manual of the enclosure:

^{(The other two mini-SAS-HD connectors listed in the specs of the enclosure are for daisy-chaining more enclosures, but I have just one.)}

I'd like to know:

• Main question: How do I know the controller and enclosure actually have 8x 12Gbit/s lanes (as provided by 2x four lanes of mini-SAS-HD) bandwidth available? (a.k.a. wide port configuration enabled)
• How to tell which cables are connected? For reasons of monitoring, I'd like to know about cable failure, similar to an ethernet link status. It would be great to be able to alert when the second link goes down.

Please note that my question here is unrelated to a high-availability scenario; the enclosure I have does not support high-availability (multi-path). My question is about port configuration / bandwidth.

What I do know:

The Broadcom StorCLI reference manual suggests looking at the "load balance" mode status/configuration.

set loadbalancemode =[on|off] Enables (on) or disables (off) automatic load balancing between SAS phys or ports in a wide port configuration.

However, StorCLI reports neither on or off, it reports "None" in the "Policies Table":

Load Balance Mode = None

Listing the drives in the enclosure lists two connectors, though, so that sounds okay to me:

Drive /c0/e18/s0 Device attributes :
==================================
Manufacturer Id = HGST    
[...]
Device Speed = 12.0Gb/s
Link Speed = 12.0Gb/s
[...]
Connector Name = C0   & C1   

However, all of the disks including the enclosure are listed in kernel messages with just C0 connector, e.g.: scsi 5:0:39:0: enclosure level(0x0001), connector name( C0 ). ?

Similar to my previous Q&A in which I successfully set up an authenticating Dovecot IMAP proxy with Kerberos/GSSAPI, I want to do the same with TLS client certificates;

• My upstream (backend) IMAP server allows to authenticate without a password (trusts this Dovecot proxy to authenticate users properly).
• This Dovecot proxy is set up to validate a TLS client certificates and take the username from the Common Name field of the certificate.

The Dovecot configuration file:

# Default configuration will have it listen on IMAP tcp/143 with StartTLS required and IMAPS tcp/993 with TLS required.
protocols = imap

hostname = myhostname.domain.tld

passdb {
  driver = static
  # Backend IMAP server that accepts any/none password for a given user.
  args = proxy=y host=10.1.2.3 port=9999 pass=masterpass nopassword=y
}

# Deliberately omitted userdb, because this is a proxy.

# local username only
auth_username_format = %n

# Logging to foreground with some verbose logging for authentication.
log_path = /dev/stderr
auth_verbose = yes
verbose_ssl = yes
auth_debug = yes

ssl = required
ssl_cert = </etc/dovecot-ssl/cert.crt
ssl_key = </etc/dovecot-ssl/key.pem
ssl_prefer_server_ciphers = yes
ssl_min_protocol = TLSv1.2
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:[... omitted for brevity ...]

# SSL client certificate authentication required (no password required by client).
# https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/#client-certificate-verification-authentication
ssl_ca = </etc/dovecot-ssl/client-ca.crt
ssl_verify_client_cert = yes
auth_ssl_require_client_cert = yes
# I don't have CRL set up at this point (will do later) and Dovecot requires to disable CRL check or else it fails.
ssl_require_crl = no

# Take the username from the client certificate (CN)
auth_ssl_username_from_cert = yes
ssl_cert_username_field = commonName

This works really well, except for one thing: the clients are still required to provide a bogus username/password. ?

I would have expected to be able to omit username/password in IMAP clients (authentication type = client cert), e.g. K-9 mail.

Server side error in logs shows a no auth attempts info level logged error message:

[...]
Apr 27 18:37:57 imap-login: Info: Received valid SSL certificate: [...]
[...]
Apr 27 18:37:57 imap-login: Debug: SSL: where=0x2002, ret=1: SSL negotiation finished successfully
Apr 27 18:37:57 imap-login: Debug: SSL alert: close notify
Apr 27 18:37:57 imap-login: Info: Disconnected (no auth attempts in 0 secs): user=<>, rip=10.9.9.9, lip=10.1.2.4, TLS, session=<Yw6fj/jA1JsKMgB0>
Apr 27 18:37:57 imap-login: Debug: SSL alert: close notify

When setting a bogus username+password, the login seems to work, though:

[...]
Apr 27 18:33:36 imap-login: Debug: SSL: where=0x2002, ret=1: SSL negotiation finished successfully
Apr 27 18:33:36 imap-login: Info: proxy(gert): started proxying to 10.1.2.3:9999: user=<gert>, method=PLAIN, rip=10.9.9.9, lip=10.1.2.4, TLS, session=<bSMSgPjA0psKMgB0>

So, bottom line my question is: how do I configure Dovecot to log in with TLS client certificate only and to avoid having to tell my users to pick a random password?

I've tried:

• setting nologin=y as passdb arg option, but that does not seem to have any effect.
• set auth_mechanisms = anonymous, but that does not seem to have any effect.
• set auth_mechanisms = (empty), but that fails any connection with auth: Fatal: No authentication mechanisms configured

My question is regarding the Delivery Status Notification (DSN) handling of my inbound Postfix mail server instances. The solution to this may be completely outside of scope of DSNs however.

Let's say this is the flow for mail arriving at my domain. D, E and F are in my control, and A, B and C are the sender's.

 A
  +----------+
  |          |
  |   MUA    |
  | (sender) |
  |          |
  +----+-----+
       |
       |
B      |            C                   D                  E
 +-----v-------+     +-----------+       +------------+     +------------+
 |             |     |  Outbound |INTERNET Inbound    | LAN | Local mail |
 | Submission  +----->  gateway  +-------> gateway    +---->+ storage    |
 | (MTA)       |     |  (MTA)    |       | (MTA)      |     | delivery   |
 |             |     |  sender   |       | recipient  |     | (MTA)      |
 +-------------+     +-----------+       +------------+     +-----+------+
                                                                  | LMTP
                                                             LAN  | (final
                                                                  | delivery)
                                                           F      |
                                                            +-----v------+
                                                            |            |
                                                            | Mailbox    |
                                                            | devlivery  |
                                                            | (MDA)      |
                                                            +------------+

D (edge, internet facing, highly available) and E (internal, budget SLA) are separate machines, as I'd like to separate things for scalability, security and availability. This way mail can be queued for days at D in case E in a cheap network/cluster is down amid a disaster scenario.

The sender can request a DSN in his mail client application (E.g. in Thunderbird, enable DSN in the compose view menu) and with Postfix at its default, it will inform the sender of successful delivery from E as soon as it has delivered it to F. In this DSN message, the sender gets to know a lot about my internal network and that's what I'd like to avoid.

From the perspective of D, itself is not a final destination of the mail (as E is configured as relay for the domain). Moreover, when it's relaying to E, it's still not a final destination upon relaying the mail. This makes it only at E to inform the sender.

I found that Gmail has a very elegant way. It somehow triggers the DSN already upon the submission of C to D (where D is then Gmail's edge MTA). I'd like to know how to configure my Postfix instances like Gmail's internet facing MTAs. That would make the sender's outbound MTA responsible of informing the delivery status to the remote MTA.

Since I have no conditional delivery set up, bounces turned off along this path, all mail accepted at D will be delivered, one way or another to a mailbox. So, I'd like to tell remote MTAs already at that point that delivery is successful, and I want to disable any further DSN handling on my own servers. This should let C inform the sender of successful delivery.

I've tried to set smtpd_discard_ehlo_keywords = silent-discard, dsn at D already as per Postfix's DSN readme, but as expected, senders do not get informed at all anymore and that's too much to turn off.

In short; how can I have Postfix tell (lie to) remote MTA C that D is final delivery for the message?
... in an attempt to mimic Gmail's handling.

I've deployed a pod with Apache httpd (official image, tag 2.4.41), serving HTTP/2.0 plaintext on port 8082, and it's working "fine", but I see restarts every few hours (kubectl get pod/mypod, showing 60 restarts in 5 days).

The logs always show "caught SIGWINCH, shutting down gracefully" (-p, for previous pod):

$ kubectl logs -c httpdcontainer -p pod/mypod
  [...]
  127.0.0.1 - - [15/Jan/2020:11:12:27 +0000] "GET / HTTP/2.0" 200 9578
  [Wed Jan 15 11:12:40.433582 2020] [mpm_event:notice] [pid 1:tid 139963289400448] AH00492: caught SIGWINCH, shutting down gracefully
  127.0.0.1 - - [15/Jan/2020:11:12:37 +0000] "GET / HTTP/2.0" 200 9578

The SIGWINCH signal is probably coming from a stop signal via Docker (as per official Dockerfile), e.g. Kubernetes liveness probe? The GET requests are the configured liveness probes on /, but as you can see Apache is returning 200 OK just fine.

Kubernetes kubelet seems to disagree with the 200 OK from Apache and is restarting the pod on purpose:

$ kubectl describe pod/mypod
[...]
Type     Reason     Age                       From                                    Message
----     ------     ----                      ----                                    -------
Warning  Unhealthy  38m (x208 over 5d)        kubelet, node01.kube.mydomain.tld  Readiness probe failed: Get http://192.168.87.178:8082/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Normal   Killing    38m (x60 over 4d23h)      kubelet, node01.kube.mydomain.tld  Container docs failed liveness probe, will be restarted
Warning  Unhealthy  38m (x221 over 5d)        kubelet, node01.kube.mydomain.tld  Liveness probe failed: Get http://192.168.87.178:8082/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Normal   Pulled     38m (x60 over 4d23h)      kubelet, node01.kube.mydomain.tld  Container image "myregistry.mydomain.tld/foo/bar/myimage@sha256:<checksum>" already present on machine
Normal   Created    38m (x61 over 5d19h)      kubelet, node01.kube.mydomain.tld  Created container docs
Normal   Started    38m (x61 over 5d19h)      kubelet, node01.kube.mydomain.tld  Started container docs
Warning  Unhealthy  13m (x1644 over 5d19h)    kubelet, node01.kube.mydomain.tld  Readiness probe failed: HTTP probe failed with statuscode: 503
Warning  Unhealthy  3m16s (x1717 over 5d19h)  kubelet, node01.kube.mydomain.tld  Liveness probe failed: HTTP probe failed with statuscode: 503

I have no clue how to tell what is to blame here. I'm using Kubernetes 1.16.4, deployed with kubeadm and Calico CNI.

In my Ansible playbook I have this:

- name: "A: Check to see if we need to run task B"
  [... implementation omitted, not relevant ...]
  register: task_a_result
  check_mode: no  # even run in check mode
  changed_when: no  # this only reads/checks stuff

- name: "B: Write x to file"
  shell: "echo {{ my_var|quote }} > /path/to/file"
  when: task_a_result.stdout_lines[0].startswith('ABCDEF')
  changed_when: yes  # when run, it always changes the state

I do not want to run B in check mode (hence, no check_mode: no), but I want to report it as changed if it would have been run in non-check mode. I want this behaviour, because I do not want surprises when running in non-check mode. However, despite having changed_when: yes and having set a conditional myself, Ansible keeps showing the task as skipped and thus unchanged to me:

skipping: [myhost] => changed=false 
  msg: skipped, running in check mode

(The above in check mode, and it reports 'changed' in regular non-check mode.)

I found this bug report that appears misinterpreted and closed inappropriately: 14950, but I cannot comment on that any longer.

Am I overlooking something basic? Other modules usually report a "would have changed" status perfectly fine in Ansible, but is this also possible for shell/command?

Using Ansible 2.7.12 and 2.8.2 yields the same results.

I am hoping to avoid nasty hacks in the command itself like:

- name: "B: Write x to file"
  shell: "echo {{ my_var|quote }} {{ '>' if task_a_result.stdout_lines[0].startswith('ABCDEF') else '' }} /path/to/file"
  when: task_a_result.stdout_lines[0].startswith('ABCDEF')
  changed_when: yes

^{And yes I know I can write to a file with the copy/template modules, but I can't seem to overwrite files with it, e.g. echo 1234 > /sys/module/zfs/parameters/zfs_arc_max, because copy/template would attempt to replace the file it seems and setting a kernel parameter wouldn't work like that. And no, this kernel module parameter is not exposed through sysctl on Linux.}

I have a Job that I'd like to retry a few times in Kubernetes. A Job can have a backoffLimit and a activeDeadlineSeconds set. Cool so far.

However, the latter setting takes precedence over the first, unfortunately. My Job blocks for a long time until it fails, so I'd like to have a way to retry the task in a new pod with each retry having a deadline.

E.g. try five times with a deadline of 30 seconds for each attempt.

Is that possible in Kubernetes?

Let's say I manage file myfile.conf with Ansible, e.g.:

- template:
    src: "myfile.conf.j2"
    dest: "/etc/myfile.conf"

I can then see the diff in a dry-run, like this:

$ ansible-playbook --check --diff myplaybook.yaml

Cool. But it seems to only display the diff when the file exists on the remote host. If the file is absent, I don't see the output other than task OK.

How do I tell Ansible to show the diff if the file is absent currently?

Perhaps similar to how Git would show a diff in a commit of a newly created file in style like with /dev/null as base in the diff:

--- /dev/null
+++ b/myfile.conf.j2

For integration testing purposes I'm creating a very simple single-node Kafka deployment:

• 1x Zookeeper
• 1x Kafka
• 1x Kafka client (e.g. AdminClient, creating topics)

(All cleanly deployed in fresh Docker containers.)

I'm seeing intermittent failures to connect from the client to Kafka, but whenever I try to connect again it will just connect fine. I've enabled debug logging on the Kafka client and this is what I'm seeing on the first connection:

1. Client connects to its configured bootstrap server just fine:

   [main] DEBUG o.a.k.c.a.i.AdminMetadataManager - 
          [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata
          Cluster(id = null, nodes = [kafka:9092 (id: -1 rack: null)], 
          partitions = [], controller = null).
   [...]
   [kafka-admin-client-thread | adminclient-1] DEBUG 
          o.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1]
          Completed connection to node -1. Fetching API versions.
   

2. Hundreds of lines showing that requesting the nodes in the cluster yields no entries:

   [kafka-admin-client-thread | adminclient-1] DEBUG
          o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-1]
          Updating cluster metadata to Cluster(id = q7XgghZqQUW_o5W2-Nn5Qw,
          nodes = [], partitions = [], controller = null)
   

   Note the nodes = [] part here in particular.

   This goes on for at least a few seconds, sometimes even 30 seconds! I can't seem to understand why the Kafka server can't list itself as a node.

   In an unlucky case, the timeout is reached and I see the infamous exception thrown:

    Exception in thread "main" java.util.concurrent.ExecutionException:
        org.apache.kafka.common.errors.TimeoutException: Timed out waiting
        for a node assignment.
   

3. In a lucky case, after a while, it lists itself and it can connect just fine:

   [kafka-admin-client-thread | adminclient-1] DEBUG 
          o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-1]
          Updating cluster metadata to Cluster(id = q7XgghZqQUW_o5W2-Nn5Qw,
          nodes = [kafka:9092 (id: 0 rack: null)], partitions = [],
          controller = kafka:9092 (id: 0 rack: null))                                     
   [kafka-admin-client-thread | adminclient-1]
          DEBUG o.apache.kafka.clients.NetworkClient -
          [AdminClient clientId=adminclient-1] Initiating connection to node
          kafka:9092 (id: 0 rack: null) using address kafka/xxxxx
   

   Note the presence of nodes = [kafka:9092 (id: 0 rack: null)] here.

My problem is the huge variance in time between 1 and 2. I fail to understand why a client is not given a node rightaway (itself in this case!). It not only slows down my automated testing, it also hurts scrolling through logs full of failures with timeouts, as it regularly does not connect within the default timeout window. Any successive attempt after waiting long enough will immediately connect just fine without all of the oddness, though...

Kafka client + server versions: 2.1.0, running on Kubernetes. Kafka server logs don't indicate anything special, all silent already seconds before the attempt by the client is made, until after the connection is established (if at all).

What am I missing here? Am I looking at random pressure back-offs or do I have to tell Zookeeper/Kafka that it's fine to run on it's own, without having to wait for other nodes?

I have read this infamous blog article about listener configurations, but that's not relevant in my case: my Kafka does not advertise the node(s) for a while, until it does, despite having a list of advertised hosts. And after that, it connects just fine - those having issues with advertised listeners cannot connect at all.

In my TLS enabled OpenVPN configuration I would like to use the additional security offered by using tls-auth. The good news is, is that it works as expected. However, I have a question about the optional key-direction parameter (either as a second parameter to the tls-auth option or as key-direction option).

The configuration parameter is described in the manpage as follows (version 2.3.x, relevant snippets):

--tls-auth file [direction]
      Add  an  additional layer of HMAC authentication on top of the TLS control
      channel to protect against DoS attacks.

      In a nutshell, --tls-auth enables a kind of "HMAC firewall"  on  OpenVPN's
      TCP/UDP  port, where TLS control channel packets bearing an incorrect HMAC
      signature can be dropped immediately without response.

      file (required) is a key file which can be in one of two formats:

      (1) An OpenVPN static key file generated by --genkey (required  if  direc‐
      tion parameter is used).

      (2) A freeform passphrase file.  In this case the HMAC key will be derived
      by taking a secure  hash  of  this  file,  similar  to  the  md5sum(1)  or
      sha1sum(1) commands.

      OpenVPN  will  first  try  format (1), and if the file fails to parse as a
      static key file, format (2) will be used.

      See the --secret option for more information  on  the  optional  direction
      parameter.

      ... (snip)

--secret file [direction]
      Enable Static Key encryption mode (non-TLS).  Use pre-shared  secret  file
      which was generated with --genkey.

      The optional direction parameter enables the use of 4 distinct keys (HMAC-
      send, cipher-encrypt, HMAC-receive, cipher-decrypt),  so  that  each  data
      flow  direction  has  a different set of HMAC and cipher keys.  This has a
      number of desirable  security  properties  including  eliminating  certain
      kinds of DoS and message replay attacks.

      When  the direction parameter is omitted, 2 keys are used bidirectionally,
      one for HMAC and the other for encryption/decryption.

      The direction parameter should always be complementary on either  side  of
      the connection, i.e. one side should use "0" and the other should use "1",
      or both sides should omit it altogether.

      The direction parameter requires that file contains a 2048 bit key.  While
      pre-1.5  versions  of  OpenVPN generate 1024 bit key files, any version of
      OpenVPN which supports the direction parameter, will also support 2048 bit
      key file generation using the --genkey option.

      ... (snip)

--key-direction
      Alternative way of specifying the optional  direction  parameter  for  the
      --tls-auth  and --secret options. Useful when using inline files (See sec‐
      tion on inline files).

Note that I am not using static key encryption mode, but only showing the --secret manpage snippet because it is suggested by the --tls-auth description.

I fail to understand the explanations here. Is security improved when specifying a direction (0 for server, 1 for client) or not if omitted?

On my internet-facing Postfix SMTP server running Debian Wheezy, I'd like to set up a secure connection with some known servers. Some are my own, running my own PKI/CA, some are public SMTP servers like Google's Gmail servers. To check the certificates on the servers I'm running myself, I've specified

smtp_tls_CAfile = /usr/local/share/ca-certificates/Gert_van_Dijk_Root_CA_2014.crt

and to make sure the public servers can also be verified against the built-in system certificate store, I've explicitly specified

tls_append_default_CA = yes

In order to make sure Postfix verifies the certificate properly I've set the global TLS security level to 'secure'. I will change this back to 'may' with some domains to secure in smtp_tls_policy_maps once I've finished debugging this issue described here.

smtp_tls_security_level = secure

It verifies my own servers just fine! Those are all running with certificates signed by the Gert_van_Dijk_Root_CA_2014.crt. However, it appears the SMTP client does not actually append the certificates by the tls_append_default_CA setting when it tries to deliver mails to Google's SMTP servers. I would expect it to append those from /etc/ssl/certs on Debian.

postfix/smtp[32271]: effective TLS level: secure
[...]
postfix/smtp[32271]: < alt1.gmail-smtp-in.l.google.com[64.233.164.26]:25: 220 2.0.0 Ready to start TLS
[...]
postfix/smtp[32271]: DE6D0403EB: Server certificate not verified

Here's what I've tried:

• Disabling chroot in master.cf for the smtp client like this:

  smtp      unix  -       -       n      -       -       smtp
  

  Didn't help.

• Add verbose logging. I've added -v to the line above. Had to disable rsyslog rate limiting, but that didn't improve anything. The very one line it mentions about TLS verification is the DE6D0403EB: Server certificate not verified line.

• Trying to strace the process to see if it bumps into any issue accessing the CA certificates. It appears the SMTP client is spawned as a new process for every delivery attempt. This makes the general strace -p <PID> impossible.

• Search for similar problems. I only seem to find this very similar issue, but that's about not specifying the smtp_tls_CAfile setting, which I do. With a smtp_tls_CAfile set, it should really append the system-wide CA certs.

• Reverting all back to Debian defaults, with the exception to smtp_tls_security_level = secure. It fails too in delivering to Google with the very same error. Bug in Debian's Postfix then?

• Same as above, then disabling chroot for SMTP client and then set smtp_tls_CApath = /etc/ssl/certs explicitly. Still fails on certificate verification.

• Adding TLS debugging level. Shows:

  postfix/tlsmgr[17486]: write smtp TLS cache entry smtp:64.233.164.26:25:mx.google.com&p=1&c=aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH:!aNULL&l=268439647: time=1421622891 [data 1975 bytes]
  postfix/smtp[17485]: alt1.gmail-smtp-in.l.google.com[64.233.164.26]:25: subject_CN=mx.google.com, issuer_CN=Google Internet Authority G2, fingerprint 88:C0:85:C8:CB:96:29:8F:4E:15:11:80:C5:9A:89:0D, pkey_fingerprint=E0:32:29:21:69:38:EA:F9:B6:0C:F6:BD:86:12:16:B9
  postfix/smtp[17485]: Untrusted TLS connection established to alt1.gmail-smtp-in.l.google.com[64.233.164.26]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
  postfix/smtp[17485]: DE6D0403EB: Server certificate not trusted
  

• OpenSSL manual verification on the same system shows everything really should be fine.

  $ openssl s_client -connect 64.233.164.26:25 -starttls smtp
  CONNECTED(00000003)
  depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
  verify error:num=20:unable to get local issuer certificate
  verify return:0
  

• With also disabling chroot for tlsmgr I've FINALLY got a Trusted TLS connection established, but then postfix still thinks the connection is insecure (Server certificate not verified):

  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: certificate verification depth=3 verify=1 subject=/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: certificate verification depth=2 verify=1 subject=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: certificate verification depth=1 verify=1 subject=/C=US/O=Google Inc/CN=Google Internet Authority G2
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: certificate verification depth=0 verify=1 subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mx.google.com
  postfix/smtp[27198]: SSL_connect:SSLv3 read server certificate A
  postfix/smtp[27198]: SSL_connect:SSLv3 read server key exchange A
  postfix/smtp[27198]: SSL_connect:SSLv3 read server done A
  postfix/smtp[27198]: SSL_connect:SSLv3 write client key exchange A
  postfix/smtp[27198]: SSL_connect:SSLv3 write change cipher spec A
  postfix/smtp[27198]: SSL_connect:SSLv3 write finished A
  postfix/smtp[27198]: SSL_connect:SSLv3 flush data
  postfix/smtp[27198]: SSL_connect:SSLv3 read server session ticket A
  postfix/smtp[27198]: SSL_connect:SSLv3 read finished A
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: aspmx.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt1.aspmx.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt2.aspmx.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt3.aspmx.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt4.aspmx.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: gmail-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt1.gmail-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt2.gmail-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt3.gmail-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt4.gmail-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: gmr-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt1.gmr-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt2.gmr-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt3.gmr-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: alt4.gmr-smtp-in.l.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: mx.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: aspmx2.googlemail.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: aspmx3.googlemail.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: aspmx4.googlemail.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subjectAltName: aspmx5.googlemail.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25 CommonName mx.google.com
  postfix/smtp[27198]: gmail-smtp-in.l.google.com[173.194.67.26]:25: subject_CN=aspmx.l.google.com, issuer_CN=Google Internet Authority G2, fingerprint 88:C0:85:C8:CB:96:29:8F:4E:15:11:80:C5:9A:89:0D, pkey_fingerprint=E0:32:29:21:69:38:EA:F9:B6:0C:F6:BD:86:12:16:B9
  postfix/smtp[27198]: Trusted TLS connection established to gmail-smtp-in.l.google.com[173.194.67.26]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
  postfix/smtp[27198]: DE6D0403EB: Server certificate not verified
  

Is there a way to debug the SMTP client in Postfix running in the foreground? All debugging stuff appears to be regarding the smtpd process. :-(

Relevant further configuration:

# postconf -n | grep -E "^(smtp_|tls_)"
smtp_tls_CAfile = /usr/local/share/ca-certificates/Gert_van_Dijk_Root_CA_2014.crt
smtp_tls_security_level = secure
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
tls_append_default_CA = yes

# /etc/postfix/master.cf:
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
smtp      unix  -       -       n       -       -       smtp -v

My goal is to ensure proper security for clients connecting to my nginx. I'm following Mozilla's guide to configure TLS properly on my nginx installation, but I don't have an overview of the actual protocols/ciphersuites being used in practice.

What I have now:

server {
    listen 443;
    ssl on;
    ssl_certificate /path/to/signed_cert_plus_intermediates;
    ssl_certificate_key /path/to/private_key;
    ssl_dhparam /path/to/dhparam.pem;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'the_long_ciphersuite_listed_there';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:50m;
}

With this, I'd like to log which SSL protocol was used for a connection and what ciphersuite was chosen after the client/server negotiated. E.g.:

10.1.2.3 - - [13/Aug/2014:12:34:56 +0200] "GET / HTTP/1.1" 200 1234 "-" "User agent bla"

to

10.1.2.3 - - [13/Aug/2014:12:34:56 +0200] ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 "GET / HTTP/1.1" 200 1234 "-" "User agent bla"

This way I can quickly identify clients which are using outdated browsers or automated machines which do not support PFS or other relevant security enabling technologies.

How do I configure nginx to log this information?

I'm running ZoL 0.6.2 from their PPA on Ubuntu 12.04. It's on a host with 16GB of memory intended to run some VMs using KVM/Libvirt. After some time ZoL is using an insane amount of memory, reaching 98% of RAM usage with some VMs running. This results new processes refusing to start "unable to allocate memory". I can't even start all my VMs anymore which before using ZFS were using about 40-50% of RAM.

As far as I understand, without tweaking, ZoL should release memory as soon as the system is short on memory. Well, it doesn't. So I decided to set the arc_max setting to 1GB.

# echo 1073741824 >> /sys/module/zfs/parameters/zfs_arc_max

Still, it does not release any memory.

As you can see from the below ARC statistics, it's using more memory than it's configured to (compare c=7572030912 with c_max=1073741824).

What am I doing wrong here?

# cat /proc/spl/kstat/zfs/arcstats
4 1 0x01 84 4032 43757119584 392054268420115
name                            type data
hits                            4    28057644
misses                          4    13975282
demand_data_hits                4    19632274
demand_data_misses              4    571809
demand_metadata_hits            4    6333604
demand_metadata_misses          4    289110
prefetch_data_hits              4    1903379
prefetch_data_misses            4    12884520
prefetch_metadata_hits          4    188387
prefetch_metadata_misses        4    229843
mru_hits                        4    15390332
mru_ghost_hits                  4    1088944
mfu_hits                        4    10586761
mfu_ghost_hits                  4    169152
deleted                         4    35432344
recycle_miss                    4    701686
mutex_miss                      4    35304
evict_skip                      4    60416647
evict_l2_cached                 4    0
evict_l2_eligible               4    3022396862976
evict_l2_ineligible             4    1602907651584
hash_elements                   4    212777
hash_elements_max               4    256438
hash_collisions                 4    17163377
hash_chains                     4    51485
hash_chain_max                  4    10
p                               4    1527347963
c                               4    7572030912
c_min                           4    1038188800
c_max                           4    1073741824
size                            4    7572198224
hdr_size                        4    66873056
data_size                       4    7496095744
other_size                      4    9229424
anon_size                       4    169150464
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    1358216192
mru_evict_data                  4    1352400896
mru_evict_metadata              4    508928
mru_ghost_size                  4    6305992192
mru_ghost_evict_data            4    4919159808
mru_ghost_evict_metadata        4    1386832384
mfu_size                        4    5968729088
mfu_evict_data                  4    5627991552
mfu_evict_metadata              4    336846336
mfu_ghost_size                  4    1330455552
mfu_ghost_evict_data            4    1287782400
mfu_ghost_evict_metadata        4    42673152
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_hdr_miss              4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    66583
memory_indirect_count           4    7657293
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    427048272
arc_meta_limit                  4    2076377600
arc_meta_max                    4    498721632

# free -m
             total       used       free     shared    buffers     cached
Mem:         15841      15385        456          0         75         74
-/+ buffers/cache:      15235        606
Swap:            0          0          0

I'm setting up Icinga (Nagios fork) and I have some machines with multiple interfaces. Some services are only listening on one of them and to check them correctly, I like to know if it's possible to have multiple IP addresses configured for a single host in Icinga.

Here's a minimal example:

Remote Server:

• eth0: 1.2.3.4 (public IP)
• eth1: 10.1.2.3 (private IP, secure tunnel)
• Apache listening on 1.2.3.4:80. (public only)
• OpenSSH listening on 10.1.2.3:22. (internal network only)
• Postfix SMTP listening on 0.0.0.0:25 (all interfaces)

Icinga Server:

• eth0: 10.2.3.4 (private IP, internet access)

Now if I define a host:

define host {
        use                     generic-host
        host_name               server1
        alias                   server1.gertvandijk.net
        address                 10.1.2.3
        }

This will not check the HTTP status correctly. And defining an additional host:

define host {
        use                     generic-host
        host_name               server1-public
        alias                   server1.gertvandijk.net
        address                 1.2.3.4
        }

will check everything, but shows up as two independent hosts. Now I want to 'aggregate' these two hosts to show up as a single host, yet providing an easy configuration to check the services on their proper address.

What is the most elegant number-of-configuration-lines-saving solution to this? I read about several plugins available to workaround this, but I can't figure out what is the current way to address it. Solutions go back to 2003, but I'm running Icinga 1.7.1, already capable of the address6 option, yet that triggers IPv6-only resolving on the hostname...

Ideally, I wish to configure Icinga to be intelligent enough to know that the Postfix instance running on 10.1.2.3:25 is the same as 1.2.3.4:25 and thus not triggering two alarms.

I guess this must have been tackled before and sysadmins have it set up now. Please share your solution to this. Thanks! :)

For OpenVZ, an implementation of container-based virtualization, it seems that host and all guests are sharing the filesystem cache. This sounds paradoxical when talking about virtualization, but this is actually a feature of OpenVZ.

It makes sense too. Because only one kernel is running, it's possible to benefit from sharing the same pages of filesystem cache in memory. And while it sounds beneficial, I think a set up here actually suffers in performance from it. Here's why: my machines aren't actually sharing any files on disk so I can't benefit from it.

Several OpenVZ machines are running MySQL with MyISAM tables. MyISAM relies on the system's filesystem cache for caching of data files, unlike InnoDB's buffer pool. Also some virtual machines are known to do heavy and large I/O operations on the same filesystem in the host. For testing purposes, I ran cat *.MYD > /dev/null on some large database in one machine, I saw the filesystem cache lowering in another, monitored by htop. This essentially flushes all the useful filesystem cache in guests (FIFO) and so it flushes the MySQL caches in the guests. Am I right in my analysis until here?

Now users are complaining that MySQL is very slow. And it is. Some simple SELECT queries take several seconds on times disk I/O is heavily used by other machines.

So, simply put:

Is there a way to avoid filesystem cache being wiped out by other virtual machines in container-based virtualization?

Rather than migrate to KVM or InnoDB (see below), there might be a confuration option I'm missing out on.

Some thoughts:

• Choosing algorithm for flushing filesystem cache in the kernel. (possible? how?)
• Reserving a certain amount of pages for a single VM. (seems no option for filesystem cache type of pages that reading man vzctl)
• Will running MySQL on another filesystem get me anywhere?
• Upgrading the RAM in the machine (as suggested by @michael-hampton). While being a cheap option, it does not resolve it probably. Upgrading +8GB is not in proportional to the amount of disk I/O operations overnight, say some tens of Gigabytes.

If not, I think my alternatives are:

• Use KVM for MySQL-MyISAM running VMs. KVM actually assigns memory to the VM and does not allow swapping out caches unless using a balloon driver.
• Move to InnoDB and tune the buffer pools, dirty pages, etc. This is now considered to be 'nice to have' on the long-term as not everyone responsible for administration of the system understands InnoDB.
• more suggestions welcome.

System software: Proxmox (now 1.9, could be upgraded to 2.x). One big LV assigned for the VMs.