Matija Nalis's questions

Google's gmail has recently started marking all mail from domain example.com (real domain name is hidden) as spam, with grey-box explanation of:

"Why is this message in spam? Lots of messages from example.com were identified as spam in the past."

also, messages to domains whose email is hosted on Google (AKA "G Suite"?) are generation bounces from [email protected] in the form of:

** Message not delivered **

There was a problem delivering your message to [email protected]. See the technical details below.

Learn more here: https://support.google.com/a/answer/168383

The response was:

Your email to group [email protected] was rejected due to spam classification.

The owner of the group can choose to enable message moderation instead of bouncing these emails.

More information can be found here: https://support.google.com/a/answer/168383.

Good facts are:

• domain example.com does publish SPF, DKIM and DMARC records, and has valid FcRDNS

• domain example.com gets score 10/10 on https://www.mail-tester.com/ and PASS on DKIM, SPF, DMARC on https://www.port25.com/authentication-checker/ too

• bounce messages from [email protected] also indicate SPF, DKIM and DMARC test are passed OK:

• all of the email for domain example.com always originate from same SMTP server on IP X.X.X.X (which is correctly indicated in SPF record)

Received-SPF: pass (google.com: domain of [email protected] designates X.X.X.X as permitted sender) client-ip=X.X.X.X;

Authentication-Results: mx.google.com; dkim=pass header.i=@example.com header.s=default header.b=eeWEtVgZ; spf=pass (google.com: domain of [email protected] designates X.X.X.X as permitted sender) smtp.mailfrom=[email protected]; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=example.com

• outgoing SMTP traffic from X.X.X.X to google SMTP servers is TLS-encrypted (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256)

• domain example.com and its SMTP server IPv4 address X.X.X.X are NOT on any of http://multirbl.valli.org RBL lists

• E-mail [email protected] promptly deals with any problem, is on several FBL notification lists, and has not received any spam complaints in months.

• domain example.com does not send any mass-mailings, and number of it's originated e-mails are quite low (few dozens emails per day at the most)

• https://postmaster.google.com for domain example.com indicate User-reported spam rate of 0, no IP reputation, Authenticated traffic of 100% for SPF and DKIM success rate (but jumping from 0-100% for DMARC - for unknown reasons as that is static DNS record as are SPF/DKIM ones), 100% Inbound TLS Encryption rate, 0 delivery errors

• other domains which share same SMTP server with IP X.X.X.X (and thus have same SPF, but different DKIM records) are unaffected by the issue, and can send mail normally to gmail.com users, without it being identified as a spam.

• non-google based emails do not seem to have a problem with example.com emails

BAD facts are:

• https://postmaster.google.com for domain example.com unfortunately also indicate "Domain reputation: Bad" (which is the worst on shown scale Bad, Low, Medium, High)

• what seems to have caused a problem is that some of the users on example.com have set up their e-mail forwards from example.com mailboxes to their gmail.com private addresses. And as example.com domain does indeed receive a fair amount of spam (in the order of few hundred per day), all that has been forwarded for months to gmail (with envelope-from of example.com), which is probably what nuked the domain reputation to "bad" level.

• the "forwarding spam to gmail" issue has been dealt with several days ago; and we have already asked several of gmail.com users to find e-mails in Spam folder and click on "Not spam" button and add senders to addressbook, but while that fixes the specific mail for specific gmail user, all mail from example.com to all other gmail users still continues to end up in spam folder with explanation that "Lots of messages from example.com were identified as spam in the past"

Question: what else can be done to improve e-mail domain reputation on google, so email will stop ending up in Spam folder because "lots of messages were identified as spam in the past"? Will just waiting cure the problem (and if so, any estimates how long before it gets better - especially if you've been in similar situation)? Any other hints what to try?

(related to this question)

When using sshd server (Win32-OpenSSH) on windows (with key-based auth for automation), it doesn't work unless it has STDIN (or PTY). For example, running this (from Debian Jessie openssh client) works OK (returning windows username):

ssh -4 -T -o Batchmode=yes winserver whoami

However, those two do not (they simply terminated without executing command and without returning any output):

ssh -4 -T -o Batchmode=yes winserver whoami < /dev/null
ssh -4 -n -T -o Batchmode=yes winserver whoami

This presents problem as it is impossible to run non-interactive ssh commands from programs which do not have open STDIN (like cron(8) or atd(8)).

When using Debian openssh server, it of course works without any problems. Problem only happens with windows ssh servers (Fails with Win32-OpenSSH and FreeSSHD. Bitvise SSHD however seems to work OK, but we're looking at gratis solutions for windows ssh servers; open source, simple and maintained are bonuses)

This has been reported, but does anybody have solution or workaround in the meantime?

I'm experiencing sporadic problems when trying to ssh to windows servers (running either freesshd or Win32-OpenSSH, same results). When trying to run non-interactive commands (from Debian Jessie with OpenSSH client), sometimes (but not always, and at different intervals) remote command will stop and require pressing ENTER to continue.

For example:

ssh -4 -T somehost "mysqldump --master-data --all-databases" > backup.sql &

on some windows machines it finishes ok (if the databases are small enough) but on bigger ones it will usually stop (either immediately, with backup.sql remaining with size 0, or after some time, with backup.sql having from few megabytes to few hundred megabytes) and then shell will inform me that ssh client process is waiting for input:

[1]+  Stopped                 ssh -4 -T somehost "cat mysqldump --master-data --all-databases" > backup.sql

if I bring it to fg and press ENTER, it will continue working, even after I CTRL+Z and bg it. Sometimes it will finish then, other times it will stop again after some time.

strace confirms the process is waiting for TTY input (on file descriptor 4, which is RW opened /dev/pts/2 or similar PTY of shell spawning the ssh client) and that it is why process has been stopped.

If I run ssh from script without associated TTY it will die (as there is nobody to press ENTER). If I provide -n option to redirect stdin from /dev/null, ssh will die immediately upon connection. That poses additional problem, as ssh connections which are not run from terminal (but for example from cron(8)/atd(8)) will fail immediately.

Only close question I've found is this one, but the provided workaround (forcing PTY use) is not really applicable to my case (programs in question modify their behaviour when PTY is detected, adding ANSI sequencing and word wrapping etc).

Needless to say, the same commands that connect to hundreds of Debian (and other GNU/Linux) systems running OpenSSHD servers experience no problems whatsoever (weather using -n or not). Problems only ever arise when sshd server is running on microsoft windows.

Does anyone have ideas how to fix or work around the problem? I've tried naive approach of:

yes '' | ssh -4 -T somehost "mysqldump --master-data --all-databases" > backup.sql &

but it does not work well (while it fixes the ssh waiting for keyboard, it also randomly manages to insert newlines into output, which is unacceptable). Also related OpenSSH server on Windows refusing to work without STDIN even in noninteractive mode

We have an postfix SMTP server, used to relay mail for customers web scripts. Postfix is using recipient address verification with reject_unverified_recipient in smtpd_recipient_restrictions (it is a must, due to high number of unwanted bounces otherwise):

smtpd_recipient_restrictions = 
    reject_unknown_recipient_domain
    reject_unverified_recipient
    permit_mynetworks
    reject_unauth_destination
    check_recipient_access hash:/etc/postfix/validrcptto

Problem is that first time some e-mail address is used, it will result in 450 4.1.1 <[email protected]>: Recipient address rejected: unverified address: Address verification in progress temporary errors, which client is supposed to retry later (as defined in section 4.5.4.1 of RFC 5321)

While the 4xx error is gone in few seconds (in vast majority of the cases, or replaced with 5xx permanent error), the customers web scripts (of course) try only once.

While it is relatively trivial to implement in web scripts (replace SEND_MAIL with while (count++ < 10) { SEND_MAIL; sleep(3) } pseudo-code), the number of different customers, third-party codebases with its auto upgrades etc) just explaining the problem over and over again is prohibitively resource-draining...

What I would like is something like reject_unverified_recipient_after_autoretry=30s which would not reply immediately to RCPT TO with 450 Address verification in progress but would instead block, waiting a few seconds in hope verify(8) will finish, and try again, and only return 450 if after (say) 30 seconds the code is still 450.

Note: accepting all mails from web scripts is not acceptable (even if sender is set to <> or some unattended/blackholed mailbox) - we really want address verification, and we want web apps to get 5xx errors if we can know the mail won't be able to reach recipient.

Several bacula-fd clients are backing up to one bacula SD (using 2G disk files on massive RAID, not tape), usually 2-3 of them concurrently. It works fine, except when 2 bigger clients (about 400-900GB each) need to run full backups, which get terribly slow (usually about 200-2500 Kbyte/s), so full backup does not finish in several days, which is a problem for us (so we cancel it and go with incremental).

The server and clients are in different VLANs/subnets, so are routed over another debian wheezy machine with VLANs and few switches. NICs are 1Gbps on all machines (one having active-passive network bonding - failing it over does not help with speed), as are the switches. Machines are quad and 8-core, having 8-64GB RAM, not going into swap, and having load between 0.2-2, so I guess it is not CPU, I/O nor memory starvation. Bacula-sd is also on hardware RAID which does not seem to be under load. Network is also mostly idle at the time, so it should not be bandwidth congestion. Bacula version is standard wheezy 5.2.6+dfsg-9, and Linux kernel is also standard wheezy 3.2.60-1+deb7u3.

It seems the transfer starts just fine (at about 20+ Mbytes/sec, which is to be expected with lots of small files), than at one moment Send-Q goes up and does not come down for dozens of seconds (or minutes), and netstat shows socket in "unkn-4" timer, restarting with exponentially rising timeouts:

# netstat -tpno   | grep bacula
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name Timer
tcp        0 967688 10.66.3.135:49668   10.66.2.11:9103         ESTABLISHED 2964/bacula-fd   unkn-4 (1.86/0/0)
tcp        0      0 10.66.3.135:9102    10.66.2.11:54499        ESTABLISHED 2964/bacula-fd   keepalive (5882.64/0/0)

then after some time, the packets seems to start again:

# netstat -tpno    | grep bacula
tcp        0  38054 10.66.3.135:49668   10.66.2.11:9103         ESTABLISHED 2964/bacula-fd   on (0.21/0/0)
tcp        0      0 10.66.3.135:9102    10.66.2.11:54499        ESTABLISHED 2964/bacula-fd   keepalive (385.49/0/0)

and backup continues (confirmed with status client=blowgun-fd on bconsole). For example:

* status client=axe-fd
newaxe-fd Version: 5.2.6 (21 February 2012)  x86_64-pc-linux-gnu debian 7.0
Daemon started 24-Oct-14 17:27. Jobs: run=0 running=0.
 Heap: heap=683,600 smbytes=761,617 max_bytes=807,280 bufs=396 max_bufs=426
 Sizeof: boffset_t=8 size_t=8 debug=200 trace=1 
Running Jobs:
JobId 12640 Job axe.2014-10-24_23.05.01_40 is running.
    Full Backup Job started: 24-Oct-14 23:05
    Files=2,529,050 Bytes=253,018,715,824 Bytes/sec=1,479,901 Errors=6
    Files Examined=2,529,050
    Processing file: /home/users/novoselo/public_html/~2511/templates/js_corp/images/bg.jpg
    SDReadSeqNo=5 fd=5
Director connected at: 26-Oct-14 21:34

bg.jpg is 1.2MB in size, and it was staying on it for minutes...

Hearbeat interval is set to 120 in director, SD and file deamon configurations, and seems to be working fine. Enabling debug trace file with setdebug level=200 trace=1 all, I see:

newaxe-fd: backup.c:371-0 FT_REG saving: /home/users/novoselo/public_html/~2511/templates/js_corp/images/bg.jpg
newaxe-fd: backup.c:469-0 bfiled: sending /home/users/novoselo/public_html/~2511/templates/js_corp/images/bg.jpg to stored
newaxe-fd: crypto.c:607-0 crypto_digest_new jcr=2f01748
newaxe-fd: backup.c:1467-0 No strip for /home/users/novoselo/public_html/~2511/templates/js_corp/images/bg.jpg
newaxe-fd: backup.c:609-0 type=3 do_read=1
newaxe-fd: bfile.c:963-0 open file /home/users/novoselo/public_html/~2511/templates/js_corp/images/bg.jpg
newaxe-fd: backup.c:1194-0 Send data to SD len=65135
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: backup.c:1194-0 Send data to SD len=65562
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0
newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0

strace seems to confirm that:

# strace -tt -ff -s999 -p 3907
Process 3907 attached with 4 threads - interrupt to quit
[pid 27650] 22:25:15.705796 write(5, "[....]"..., 55110 <unfinished ...>
[pid 27661] 22:25:15.706103 select(6, [5], NULL, NULL, {2, 804806} <unfinished ...>
[pid  3912] 22:25:15.706147 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid  3907] 22:25:15.706168 select(4, [3], NULL, NULL, NULL <unfinished ...>
[pid  3912] 22:25:16.619938 <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid  3912] 22:25:16.620008 futex(0x397d82d0240, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  3912] 22:25:16.620092 futex(0x397d82d0284, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 13229, {1414358746, 620076000}, ffffffff <unfinished ...>
[pid 27661] 22:25:18.513819 <... select resumed> ) = 0 (Timeout)
[pid 27661] 22:25:18.513858 write(15, "newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0\n", 47) = 47
[pid 27661] 22:25:18.513928 select(6, [5], NULL, NULL, {5, 0}) = 0 (Timeout)
[pid 27661] 22:25:23.519025 write(15, "newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0\n", 47) = 47
[pid 27661] 22:25:23.519139 select(6, [5], NULL, NULL, {5, 0}) = 0 (Timeout)
[pid 27661] 22:25:28.524240 write(15, "newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0\n", 47) = 47
[pid 27661] 22:25:28.524317 select(6, [5], NULL, NULL, {5, 0}) = 0 (Timeout)
[pid 27661] 22:25:33.529409 write(15, "newaxe-fd: heartbeat.c:96-0 wait_intr=0 stop=0\n", 47) = 47
[pid 27661] 22:25:33.529508 select(6, [5], NULL, NULL, {5, 0}^C <unfinished ...>

fd 5 is network connection to bacula-sd, and the process is blocking when writing to that. Researching unkn-4 seems to indicate it is actually "zero window probe timer is pending".

So, it looks to me like it is either bacula-sd doing the throttling for some reason (bug?) or (more likely IMHO) some sort of networking issue.

it doesn't seem like there are errors on active ethernet adapters. I've tried using ethtool to disable offloading and other features, it didn't help. ping -f does not lose packets even when the problem in TCP is manifesting itself:

axe# ping -s1400 -f -c1000 10.66.2.11
--- slingshot.tomsoft.lan ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 607ms
rtt min/avg/max/mdev = 0.391/0.582/0.672/0.039 ms, ipg/ewma 0.608/0.585 ms

I'm looking for ideas how to proceed to troubleshoot (and finally fix, of course) this?

UPDATE1: even after tuning up TCP buffers situation is not any better - just the queue gets bigger, but still blocks, and backup is still slow. After looking some more at wireshark dumps, it seems to be bacula-sd software problem, and that TCP ZeroWindow is normal kernel way of throttling TCP in that case. So machine seems to receive data OK, but bacula-sd does not seem to be able to process data fast enough, although the machine is not under load; this is on bacula-sd side:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name Timer
tcp   3952353      0 10.66.2.11:9103         10.66.3.132:45226   ESTABLISHED 15839/bacula-sd  keepalive (4863.09/0/0)
# uptime
 05:23:13 up 2 days, 14:40,  2 users,  load average: 0.42, 0.32, 0.27