simendsjo's questions

This looks like the same issue as Windows Server 2022 Time Service Jumping into the future. I've also added a support ticket at Microsoft (Feedback Hub) for the issue: https://aka.ms/AAkwnpl

As the system clock is essential for correctly working software and probably the most central shared mutable state, this issue is wreaking havoc on both our systems and everyone we communicate with, causing ripples all the way to critical infrastructure.

We noticed this for the first time in august 2022 on a 2019 server. The clock was set to January 2023, but corrected itself. Unfortunately, this was found some time after logs had been purged, so we were unable to debug it further.

But last month, we experienced it again, this time on a 2016 server. The clock was set to 55 days to the future.

15 seconds later, Time-Service noticed the clock was different than our domain controller, and that it must change the clock back -4454176 seconds. It backs off as it things it's larger than 4294967295.

15 minutes after the first change, the clock is set again, this time backwards to 12h26m43s in the future.

15 seconds after the second change, Time-Service notice the clock is off and this time corrects it as it's within a reasonable window.

And then the same thing happened again three weeks later on the same server, only differing in details. In the mean time, the server had both been rebooted and updated with a new monthly update.

We're using VMWare, configured with two physical hardware clocks. We have two domain controllers configured to use pool.ntp.org -- should probably be moved to our own stratum 0 hardware, although it's probably not related to our issues.

With the help from a few external experts, we have pretty much excluded erroneous configuration, manual intervention (by mistake, security breach or disloyal employee) and hardware issues, and we're left with "strange Windows bug".

Unfortunately, 2016 doesn't include much details related to these events, so it's difficult to debug further. 2019+ includes more information.

@chris1out in Windows Server 2022 Time Service Jumping into the future had the same issue for servers not enrolled in a domain, so we can probably rule out the domain controller. It was also using the standard time server and not pool.ntp.org. This means we can probably rule out those two too. This pretty much leaves a bug in Time-Service as the probable cause. This serverfault question is the only documented event of this we're able to find.

Some semi-recent upgrades must have broken my postfix+dovecot configuration, but I'm having problems finding out what the cause is.

My domain is simendsjo.me with the MX record mail.simendsjo.me.

I can send mail to both local and external recipients, and it delivers mail from internal mailboxes.

The problem is that mail from external senders isn't delivered, and nothing is logged at all. The external sender also doesn't receive any errors.

I have no idea where to ever start looking as nothing is logged at all when external mail is sent to my server.

So the first issue would be: How can I turn on some debug messages for postfix?

I've tried:

debug_peer_level = 2
debug_peer_list = simendsjo.me

.. And _level = 999 and _list = gmail.com where I'm trying to send emails from. but nothing is logged.

When sending mails from a local mailbox (but from an outside computer, not localhost), a lot is logged.

I don't have any rules in iptables either.

Any ideas how I can get some debug messages for postfix?

I've set up postfix to pipe emails through rdm-mailhandler.rb (redmine), but I'd also like the emails to be delivered to the mailbox.

transport:

[email protected] redmine

master.cf:

redmine   unix  -       n       n       -       -       pipe user=mongrel argv=/opt/redmine-1.2.2/processemail.sh

How can I set up so the emails are delivered to the mailbox too and not just eaten by redmine?

Possible Duplicate:
How to configure postfix to pipe all incoming email to a script?

I'm having problems setting up simple piping for Redmine (or '|cat > /tmp/temp')

/etc/postfix/virtual:

[email protected] support

/etc/postfix/aliases:

support: "|/opt/redmine-1.2.2/processemail.sh"

main.cf

virtual_alias_maps = hash:/etc/postfix/virtual, mysql:/etc/postfix/mysql_virtual_forwards.cf
virtual_mailbox_domains = myhost.com, mysql:/etc/postfix/mysql_virtual_domains.cf
virtual_mailbox_maps = hash:/etc/postfix/virtual_alias, mysql:/etc/postfix/mysql_virtual_mailboxes.cf

alias_maps = hash:/etc/postfix/aliases
alias_database = $alias_maps

processemail.sh:

#!/bin/sh
/opt/redmine-1.2.2/extra/mail_handler/rdm-mailhandler.rb --url http://myhost.com --key f5obfctmGfqZWnOWNtR8 --project test --tracker support --allow-override tracker,project --verbose --unknown-user accept --no-permission-check

I've run newaliases; postmap virtual; postalias aliases; postalias virtual_alias; newalias - Yeah, probably overdoes it, but still misses the pouint :(

I've run newaliases and restarted postfix. Mail is delivered as usual, but nothing is added to /tmp

This is the only log entry saying anything about virtual.

Dec 10 04:08:41 master postfix/virtual[3112]:
A3F46AD1BF: to=<[email protected]>, relay=virtual, delay=0.4,
delays=0.38/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to maildir)

If I run cat someemail | processemail.sh in the shell, everything works. Processemail isn't even run by postfix.

Anyone knows what might be wrong? Or how I can turn on more logging?

Everything works nicely if I have worker_processes=1, or I run one request at a time. If I open a browser on a fresh nginx install and holds F5 down a couple of seconds, the sockets gets lingering as CLOSE_WAIT, and nginx wont do anything (other than stuffing more sockets in CLOSE_WAIT..)

The more workers I use, the sooner it gets stuck. Just one of the workers has listed any sockets though..

Any idea what might be causing this?