QuantumMechanic's questions

I am running Ubuntu 14.04LTS. I have grabbed certbot-auto from the EFF website since certbot isn't in 14.04.

I've tried running it as

certbot-auto --apache -d my.domain

and it consistently times out during the TLS-SNI-01 challenge with the following message:

Failed authorization procedure. my.domain (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to ww.xx.yy.zz:443 for TLS-SNI-01 challenge

I have verified that inbound TLS is not being blocked by running openssl's s_server command and successfully connecting to the s_server "server" from external locations.

Looking at /var/log/apache2/error.log I do see a suspicious warning about (I assume) the docroot that certbot-auto is temporarily trying to set up not being found:

[Tue Dec 06 00:34:02.306032 2016] [mpm_prefork:notice] [pid 19521] AH00171: Graceful restart requested, doing restart
[Tue Dec 06 00:34:03.001014 2016] [mpm_prefork:notice] [pid 19521] AH00163: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Tue Dec 06 00:34:03.001094 2016] [core:notice] [pid 19521] AH00094: Command line: '/usr/sbin/apache2'
[Tue Dec 06 00:34:14.596461 2016] [mpm_prefork:notice] [pid 19521] AH00171: Graceful restart requested, doing restart
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
[Tue Dec 06 00:34:14.661372 2016] [ssl:warn] [pid 19521] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Dec 06 00:34:15.000674 2016] [mpm_prefork:notice] [pid 19521] AH00163: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f configured -- resuming normal operations
[Tue Dec 06 00:34:15.001293 2016] [core:notice] [pid 19521] AH00094: Command line: '/usr/sbin/apache2'

In /var/log/letsencrypt/letsencrypt.log I see the following, which looks totally reasonable:

2016-12-06 05:34:13,247:INFO:certbot.auth_handler:Performing the following challenges:
2016-12-06 05:34:13,268:INFO:certbot.auth_handler:tls-sni-01 challenge for mydomain.com
2016-12-06 05:34:13,359:INFO:certbot_apache.configurator:Enabled Apache socache_shmcb module
2016-12-06 05:34:13,520:INFO:certbot_apache.configurator:Enabled Apache ssl module
2016-12-06 05:34:14,345:DEBUG:certbot_apache.tls_sni_01:Adding Include /etc/apache2/le_tls_sni_01_cert_challenge.conf to /files/etc/apache2/apache2.conf
2016-12-06 05:34:14,351:DEBUG:certbot_apache.tls_sni_01:writing a config file with text:
 <IfModule mod_ssl.c>
<VirtualHost *>
    ServerName b103a17811594b35d3ade8eb763c8c42.74b423d383109b22eaecca3ea14cd1f7.acme.invalid
    UseCanonicalName on
    SSLStrictSNIVHostCheck on

    LimitRequestBody 1048576

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /var/lib/letsencrypt/mRH5i2aIrWMqjsZn3jMXvdNnYV5Ejd0GBtcDoIJqQ4U.crt
    SSLCertificateKeyFile /var/lib/letsencrypt/mRH5i2aIrWMqjsZn3jMXvdNnYV5Ejd0GBtcDoIJqQ4U.pem

    DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
</VirtualHost>

</IfModule>

If I watch /var/lib/letsencrypt while certbot-auto runs I don't ever see a tls_sni_01_page being created. Here's the sequence I see -- a temp directory and some temp certs being made and then deleted when the challenge fails, but never tls_sni_01_page

/var/lib/letsencrypt$ ls
backups
/var/lib/letsencrypt$ ls
backups
/var/lib/letsencrypt$ ls
backups
/var/lib/letsencrypt$ ls
backups  temp_checkpoint
/var/lib/letsencrypt$ ls
backups  temp_checkpoint
/var/lib/letsencrypt$ ls
backups  e1pvy9TgUvOmBxC4dlkwG5Ly2mTY0DU63qhUFpBik2Y.crt  e1pvy9TgUvOmBxC4dlkwG5Ly2mTY0DU63qhUFpBik2Y.pem  temp_checkpoint

I've also changed /etc/letsencrypt/options-ssl-apache.conf to set the logging level to DEBUG and to have that temporary virtual host log to its own error and access logs. I see output but no problems in the error log, but don't see anything at all in the access log.

I've tried running tcpdump -n port 443 and I do see some sort of connection attempt from the remote machine:

09:14:44.388328 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [S], seq 1627589669, win 29200, options [mss 1460,sackOK,TS val 2834688174 ecr 0,nop,wscale 7], length 0
09:14:44.388435 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [S.], seq 773250860, ack 1627589670, win 28960, options [mss 1460,sackOK,TS val 424229197 ecr 2834688174,nop,wscale 7], length 0
09:14:44.451190 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 2834688237 ecr 424229197], length 0
09:14:44.451546 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [P.], seq 1:220, ack 1, win 229, options [nop,nop,TS val 2834688237 ecr 424229197], length 219
09:14:44.451619 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [.], ack 220, win 235, options [nop,nop,TS val 424229216 ecr 2834688237], length 0
09:14:44.454461 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [P.], seq 1:393, ack 220, win 235, options [nop,nop,TS val 424229217 ecr 2834688237], length 392
09:14:44.454626 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [F.], seq 393, ack 220, win 235, options [nop,nop,TS val 424229217 ecr 2834688237], length 0
09:14:44.517433 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [.], ack 393, win 237, options [nop,nop,TS val 2834688303 ecr 424229217], length 0
09:14:44.517481 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [P.], seq 220:227, ack 394, win 237, options [nop,nop,TS val 2834688303 ecr 424229217], length 7
09:14:44.517514 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [F.], seq 227, ack 394, win 237, options [nop,nop,TS val 2834688303 ecr 424229217], length 0
09:14:44.517544 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [.], ack 228, win 235, options [nop,nop,TS val 424229235 ecr 2834688303], length 0

Any idea what the heck is going on? Anyone ever get this working with Ubuntu 14.04LTS and apache?

I have a linode that is currently on one of Linode's Xen servers. I see that I am eligible to upgrade to a KVM linode.

One thing that concerns me is that Linode's upgrade guide says block devices will change from the /dev/xvda naming scheme to the /dev/sda naming scheme.

My /etc/fstab is:

proc        /proc        proc    defaults                       0 0
/dev/xvda   /            ext3    noatime,errors=remount-ro      0 1
/dev/xvdb   none         swap    sw                             0 0

so I would think I am going to have to modify it in some way in conjunction with the upgrade.

Should I just edit the /dev/xvda and /dev/xvdb entries to /dev/sda and /dev/sdb before shutting down and upgrading?

Should I switch to the UUID= style of declaring devices? (Would the device UUID even stay the same, now that I think about it?)

I did open a support ticket with Linode and all they said was "turn on Distro Helper in your config profile before you upgrade and that should take care of it". Which leaves open the chance that it won't, so I'd prefer to do a known good thing proactively.

Relatedly, I assume the "root device" option in the configuration profile in the management UI (which is a dropdown that gives the /dev/xvdN options) will correctly map /dev/xvda to /dev/sda, etc. when you upgrade?

All in all, I'd love to hear from people who has put their linode through the Xen->KVM transition to see how this issue (and others I don't know about) shake out.

This is apache 2.2.14 on SLES9.

Out of nowhere (i.e. it had been working fine for ages) I am seeing apache2 suddenly start using 100% of the CPU at startup, and never completing startup. Nothing is getting written to /var/log/error_log (when it did back when things were OK). ps only shows the main httpd process and not any of the spawned threads. When things were OK, it would show the spawned threads.

So it appears httpd is going into some sort of infinite loop right at startup and isn't even completing startup. It's not an issue of being overloaded by connections -- this happens even when nothing is trying to contact it.

The config files haven't changed (or at least they haven't changed in a way that changed their last-modified time).

I've tried added -e debug -E /var/log/apache2/startup_info to the command line, but nothing is put in the file.

Any ideas what could be happening?

How do you deal with the case of:

• wanting to whitelist a domain so that emails from it won't get eaten, but
• not having emails forged to appear to be from that domain get bogusly whitelisted

whitelist_from_recvd looks promising, but then you have to know at least the TLD of every host that could send you mail from that domain. Often RandomBigCompany.com will outsource email to one or more sending companies (like Constant Contact and the like) in addition to using servers that reverse-resolve to something in its own domain. But it looks like whitelist_from_recvd can only map to one sending server pattern so that would be problematic.

Is there a way to say something like "if email is from domain X, subtract N points from the spam score"?

The idea would be that if the mail is legit, that -N will all but guarantee it isn't considered spam. But if it is spam, hopefully all the other failed tests will render it spam even with the -N being included.

I have an Ubuntu 8.04LTS system running Postfix 2.5.1. On that system SMTP AUTH runs fine. The contents of /etc/postfix/sasl/smtpd.conf are:

pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN

The SASL-related properties are:

smtpd_sasl_type = cyrus
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = smtpd
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_sasl_local_domain = $myhostname

When I do sudo sasldblistusers2 I get:

[email protected]: userPassword

Like I said, that all works fine on the 8.04LTS system.

However, I am trying to migrate this over to an Ubuntu 12.04LTS system running Postfix 2.9.3 and I just cannot get it to work. I'm doing everything the same, but postfix gives authentication failures every time.

It's not the /etc/sasldb2 file. I've tried bringing over the file from the old system and that doesn't work. And I've created a new file using:

saslpasswd2 -c -u mail.mydomain.com authusername

and that doesn't work, though it WILL work on the old system if I copy it to the old system, which is how I know there's nothing wrong with the file.

Similarly, I know postfix is seeing the smtpd.conf file. If I add more mechanisms to the mech_list line of the file, I see those extra mechanisms being advertised when I connect to the smtpd daemon. And when I remove them they go away again. So /etc/postfix/sasl/smtpd.conf is clearly getting used.

I am testing both by using an actual mail client and by manually talking to the server after generating a token with this:

perl -MMIME::Base64 -e 'print encode_base64("\000authusername\000thePassword");'

then:

openssl s_client -quiet -starttls smtp -connect the.newsystem.com:587

The resulting conversation is:

250 DSN
EHLO example.com
250-the.newsystem.com
250-PIPELINING
250-SIZE 20971520
250-ETRN
250-AUTH PLAIN
250-AUTH=PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH PLAIN theBase64EncodedToken
535 5.7.8 Error: authentication failed: authentication failure

But if I instead connect to the.oldsystem.com:587 and do the same thing, I get:

235 2.7.0 Authentication successful

The output of saslfinger on the new machine is:

# sudoh saslfinger -s
saslfinger - postfix Cyrus sasl configuration Sat Jul 21 00:24:24 EDT 2012
version: 1.0.4
mode: server-side SMTP AUTH

-- basics --
Postfix: 2.9.3
System: Ubuntu 12.04 LTS \n \l

-- smtpd is linked to --
        libsasl2.so.2 => /usr/lib/i386-linux-gnu/libsasl2.so.2 (0xb76c5000)


-- active SMTP AUTH and TLS parameters for smtpd --
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_path = smtpd
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = cyrus
smtpd_tls_CAfile = /etc/ssl/certs/MyCA.pem
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/postfix/ssl/server.crt
smtpd_tls_key_file = /etc/postfix/ssl/server.key
smtpd_tls_loglevel = 1
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_session_cache_timeout = 3600s


-- listing of /usr/lib/sasl2 --
total 16
drwxr-xr-x  2 root root 4096 Jul 20 23:00 .
drwxr-xr-x 67 root root 8192 Jul 20 21:25 ..
-rw-r--r--  1 root root    1 May  4 00:17 berkeley_db.txt

-- listing of /etc/postfix/sasl --
total 20
drwxr-xr-x 2 root root 4096 Jul 20 21:29 .
drwxr-xr-x 5 root root 4096 Jul 20 23:58 ..
-rw-r--r-- 1 root root   64 Jul 20 21:29 smtpd.conf



-- content of /etc/postfix/sasl/smtpd.conf --
pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN

-- content of /etc/postfix/sasl/smtpd.conf --
pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN


-- active services in /etc/postfix/master.cf --
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
smtp      inet  n       -       -       -       -       smtpd
submission inet n       -       -       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

[snipping the rest of the services]

-- mechanisms on localhost --

-- end of saslfinger output --

What could I be missing/doing wrong? As far as I've been able to tell, all the config is the same, yet it will not work on the new system.

This is Tomcat 6.0.18, Java 1.7.0_03 (32-bit), and SLES11 SP2 (64-bit). As for kernel information:

$ uname -a
Linux server-1 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 x86_64 x86_64 GNU/Linux

We were doing a load and longevity test on three servers. On all three separate machines we had Tomcat exit within one second of 2^32 milliseconds (49+ days) of when each Tomcat started up. On each machine two threads produced stack traces before the JVM exits (Tomcat itself calls System.exit(1) when it gets the SocketTimeoutException which is why the JVM exits).

One thread is the one that (by default) listens on port 8005 for the shutdown command (verified that by looking at Tomcat source):

Jun 22, 2012 9:10:15 AM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: accept: 
java.net.SocketTimeoutException: Accept timed out
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
      at java.net.ServerSocket.implAccept(Unknown Source)
      at java.net.ServerSocket.accept(Unknown Source)
      at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
      at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
      at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
      at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

The other thread is (we believe, though we didn't check Tomcat source to verify) the one that handles incoming port 8080 connections:

Jun 22, 2012 9:10:15 AM org.apache.jk.common.ChannelSocket acceptConnections
WARNING: Exception executing accept
java.net.SocketTimeoutException: Accept timed out
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
      at java.net.ServerSocket.implAccept(Unknown Source)
      at java.net.ServerSocket.accept(Unknown Source)
      at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:307)
      at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:661)
      at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:872)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
      at java.lang.Thread.run(Unknown Source)

Tomcat isn't doing anything wild. In the first case it's just a while (true) loop that gets a Socket by calling ServerSocket.accept() and the accept() call bombs out.

Any ideas why this is happening and what I could try to look at/for to figure out how to prevent it in the future?

Note that while Tomcat was running for 2^32 milliseconds, the system had already been up when Tomcat was started. Of course that doesn't rule out some process variable that was created when Tomcat started being involved.

(Note: I am using dovecot 1.0.10 on Ubuntu 8.04.4 LTS. Yes, I know I need to upgrade before next year :)

(Note: The SMTP/IMAP server in question only servers a few users. Certainly what I propose below, even it it works, would be a logistical nightmare with any significant number of users).

I have noticed (and have confirmed via google) that the iOS mail app is terrible in its handling of IMAP subscriptions, namespaces, etc. For example, my iPhone and iPad will see EVERYTHING (all mailboxes, folders, etc.), whereas clients like Thunderbird, alpine, etc. only see what I tell them to see. This makes it an incredible pain to move mail between mailboxes because I have to scroll through a gazillion things. The mail_location in dovecot.conf is:

mail_location = mbox:%h/Mail/:INBOX=/var/mail/%u

To get around this, I've been considering doing the following for user foo:

• Create a dovecot userdb with a foo-ios virtual user in it, whose UID is identical to that of the real (in /etc/passwd) foo user and with a homedir of /home/foo-ios.
• ln -s /var/mail/foo /var/mail/foo-ios
• mkdir -p /home/foo-ios/Mail
• cd /home/foo-ios/Mail
• ln -s /home/foo/Mail/mailbox-i-want-visible mailbox-i-want-visible
• Make symlinks for the rest of limited set of mailboxes/folders I want visible to the iOS mail app.
• chown -R foo:foo /home/foo-ios
• Change iOS mail app settings to log in as user foo-ios instead of user foo.

Will this work or will there be some index/file corruption hell because there will be two sets of indexes (one set living in /home/foo/Mail/.imap and other set living in /home/foo-ios/Mail/.imap) indexing the same underlying mbox files?

And I'd be more than happy to hear of a better way to do this with dovecot! (Or to hear that dovecot 2.x works better with iOS devices).

When certain time-related programs (like ntpd) are running on a Linux system, the kernel will switch into so-called "eleven minute mode" (see the hwclock man page) whereby it will automatically update the hardware clock from the system clock every eleven minutes.

On SLES11 I have empirically determined that if I set the hardware clock to be something like 10 hours behind the system clock, 11-minute mode seems incapable of making the hardware clock match the system clock. But if I set the hardware clock 5 minutes behind the system clock, 11-minute mode makes a perfect match.

So apparently there's some maximum update that 11-minute mode can handle and I'm wondering what it is.

Update:

This is weird...

More experimentation shows that when I have the HW clock around 20 minutes behind the system clock the 11-minute mode will set the HW clock to be exactly 30 minutes behind the system clock (!):

# date
Tue Dec  6 10:16:52 EST 2011
# hwclock --set --date "12/6/11 09:56"
#
# date
Tue Dec  6 10:17:16 EST 2011
# hwclock --show
Tue Dec  6 09:56:06 2011  -0.156551 seconds
#
# date
Tue Dec  6 10:23:09 EST 2011
# hwclock --show
Tue Dec  6 10:01:58 2011  -0.535772 seconds
#
# date
Tue Dec  6 10:34:28 EST 2011
# hwclock --show
Tue Dec  6 10:04:27 2011  -0.192025 seconds

Update:

I ran across this: https://bugs.archlinux.org/task/27408 which does imply that for good or bad the kernel will not update the hardware clock when the hardware clock time is too far off from the system clock time.