Tilman Schmidt's questions

I am managing a couple of web proxies running Squid 4.10 on Ubuntu 20.04LTS in several locations distributed worldwide. One of them has developed a nasty habit of occasionally failing to access a web page. The user receives instead an error page saying:

Hmmm... can't reach this page
It looks like the webpage at <URL> might be having issues,
or it may have moved permanently to a new web address.
ERR_TUNNEL_CONNECTION_FAILED

After adding %err_code/%err_detail to the end of the relevant logformat as recommended on this mailing list post, Squid access.log entries for the failing accesses look like this:

1635169354.239    171 10.72.1.103 NONE/503 0 CONNECT ad.360yield.com:443 - HIER_
NONE/- - ERR_DNS_FAIL/-

Squid status is NONE/503, and the error code and detail always ERR_DNS_FAIL/-. The timestamp, client IP address and requested URL vary of course.

Each occurrence of the problem affects a single FQDN or very small number of FQDNs, often all from the same organisation (eg. lm.licenses.adobe.com and cc-api-data.adobe.io, both from Adobe.) All other accesses continue to work normally. An occurrence lasts typically between five and ten minutes. During that time all clients trying to access that FQDN are affected. Before and after that, the same FQDN works without a problem. There is no discernible regularity in the affected FQDNs.

Some of the occurrences are accompanied by a message like:

2021/10/25 15:42:34 kid1| ipcacheParse No Address records in response to 'ad.360yield.com'

in /var/log/squid/cache.log but in the majority of cases nothing is logged there.

How can I find out what goes wrong there?

When investigating a problem on a Windows client, I regularly find myself looking for relevant event log entries in a pretty well defined timeframe, but with no idea which of the myriad of folders in Windows Event Viewer on the involved client I should look at. Inspecting every one of these folders in turn typically shows that the majority of them is completely empty or at least does not contain any events in the time interval I am interested in.

Is there a quick way to find out which of the Windows event logs actually contain any entries in a given time interval?

I'm investigating an issue with a Ubuntu server acting as the Internet gateway for a branch office stopped working after being switched to a new Internet connection.

The IP address on the external interface is acquired from the ISP router via DHCP. When the interface was disconnected from the old line and connected to the new one, there was no Internet connectivity. Checking the log, I found that the Ethernet link on the interface had come up but was still trying to use the IP address it had been using on the old line. There weren't any dhclient log entries after the Link is Up message.

The way I understand DHCP, if the link on an Ethernet interface is going down and up again, the machine should assume that it might have been moved to a different network segment, and consequently try to reacquire its IP address before using it again.

Am I wrong?

I am running a (Linux based) rsync server for software distribution. A (Windows based) source repository server which is outside my control pushes software packages to it via rsync, and about a hundred satellite servers worldwide pull from it, also via rsync.

The source repository contains many big duplicate files. I want to reduce disk space and bandwidth consumption on the satellite servers by replacing those duplicates by hardlinks. The administrator of the source repository is unwilling or unable to do so at the source, so I'm trying to do it after the fact on the distribution server. I have created a simple bash script based on the fdupes command which finds groups of duplicates and replaces them with hardlinks to a single file. The rsync transfers to the satellite servers preserve these hardlinks as desired thanks to the -H option. The transfer from the source repository however produces inconsistent results. Sometimes the deduplication is preserved. Sometimes the source server retransmits all of the files of a deduplicated group and the deduplication is broken even though the source files did not change.

Hence my question: What is the official behaviour of rsync in case it is asked to sync two identical but separate files and the files do already exist in the destination with the correct content, but as hardlinks to the same file? What is the exact criteria for retransmitting a file? Is there a way to ensure that the hardlink in the destination is preserved in that situation even though the hardlink does not exist in the source?

Within three weeks, on two of my Ubuntu 20.04LTS servers systemd has suddenly become unresponsive. Symptoms:

• All systemctl commands for controlling services or accessing logs fail with error messages:

Failed to retrieve unit state: Connection timed out
Failed to get properties: Connection timed out

• systemd does not heed the signal from logrotate for reopening its log, continuing to write to the renamed log file /var/log/syslog.1 while the newly created /var/log/syslog remains empty.
• Lots of zombie processes accumulating from cronjobs and system management tasks, ie. PID 1 systemd neglects its duty of reaping orphaned processes.
• Running services continue to run normally but starting or stopping services is no longer possible as even the legacy scripts in /etc/init.d redirect to the non-functional systemctl.
• Nothing unusual in the logs except the Connection timed out messages from attempted interactions with systemd.

The commonly proposed corrective measures:

• systemctl daemon-reexec
• kill -TERM 1
• removing /run/systemd/system/session-*.scope.d

do not fix the problem. The only remedy is to reboot the entire system, which is of course both disruptive and problematic for a server on the other side of the globe.

The same problem occurred with Ubuntu 16.04LTS about once per month in a population of about 100 servers. It is much less frequent since the upgrade to 20.04LTS, but not completely gone. Of the two servers that have been hit since 20.04LTS, one had already been hit when it was still running 16.04LTS.

Questions:

• What are possible causes for that sort of systemd malfunction?
• How can I diagnose this further?
• Is there a less disruptive way to recover from an unresponsive systemd than to reboot?

I am trying to build an Apache reverse proxy to make a set of servers accessible through a single point of access. The servers all offer a web admin interface on port 3000, and I intend to present all of them as directories on the reverse proxy. The interfaces contain local links which must of course be rewritten to go to the correct server's subdirectory on the proxy.

I can achieve the required behaviour with configuration snippets like this for each server individually:

<Location /testadmin-warsaw/>
        ProxyPass http://warsaw.example.com:3000/
        ProxyPassReverse /
        ProxyHTMLEnable  On
        ProxyHTMLURLMap  / /testadmin-warsaw/ L
        RequestHeader    unset Accept-Encoding
</Location>

As this gets rather tedious and error-prone as servers come and go, I am aiming for a dynamic configuration. According to the Apache documentation, the following should work:

<LocationMatch "^/testadmin-(?<OFFICENAME>\w+)/(.*)$">
        ProxyPassMatch   http://$1.example.com:3000/$2
        ProxyPassReverse /
        ProxyHTMLEnable  On
        ProxyHTMLInterp  On
        ProxyHTMLURLMap  / /testadmin-${env:MATCH_OFFICENAME|unknown}/ VL
        RequestHeader    unset Accept-Encoding
</LocationMatch>

<LocationMatch> would set the environment variable MATCH_OFFICENAME to the office name part of the directory, and ProxyHTMLURLMap would insert this name at the appropriate place in the rewritten links.

But when I test that configuration, a link originally pointing to /other/page.html is rewritten to /testadmin-unknown/other/page.html instead of /testadmin-warsaw/other/page.html as intended. In other words, ProxyHTMLURLMap acts as if the environment variable MATCH_OFFICENAME was unset.

If I omit the env: part and put just /testadmin-${MATCH_OFFICENAME}/ as the to-pattern, Apache logs a warning: "AH00111: Config variable ${MATCH_OFFICENAME} is not defined".

Where's my mistake?

I have installed Bacula 7.4.4 on an openSUSE Leap 42.3 System from the repository https://build.opensuse.org/package/show/home%3AXimi1970%3AopenSUSE%3AExtra/bacula recommended on https://software.opensuse.org/package/bacula. Those packages use openSUSE's alternatives mechanism to configure the DMBS to use for the catalog - in my case, MySQL. Unfortunately, the package is a bit buggy. After installation of the bacula-director and bacula-mysql packages, the symlinks for the libbaccats library in /usr/lib64 look like this:

libbaccats.so -> /etc/alternatives/libbaccats.so
libbaccats-mysql.so -> libbaccats-mysql-7.4.4.so
libbaccats-stub.so -> libbaccats-7.4.4.so
libbaccats-7.4.4.so -> libbaccats-stub-7.4.4.so

The last two are obviously nonsense and cause any attempts to run the director or the dbcheck utility to fail with the error message:

Fatal error: Please replace this null libbaccats library with a proper one.

This is of course easily fixed by issuing the commands:

ln -sf libbaccats-stub-7.4.4.so libbaccats-stub.so
ln -sf /etc/alternatives/libbaccats-7.4.4.so libbaccats-7.4.4.so

to produce the desired result:

libbaccats.so -> /etc/alternatives/libbaccats.so
libbaccats-7.4.4.so -> /etc/alternatives/libbaccats-7.4.4.so
libbaccats-mysql.so -> libbaccats-mysql-7.4.4.so
libbaccats-stub.so -> libbaccats-stub-7.4.4.so

which allows the symlinks in /etc/alternatives:

libbaccats.so -> /usr/lib64/libbaccats-mysql.so
libbaccats-7.4.4.so -> /usr/lib64/libbaccats-mysql-7.4.4.so

to correctly direct libbaccats references to the MySQL variant.

However, each time the ldconfig(8) command is run it resets the /usr/lib64/libbaccats-7.4.4.so symlink to point to libbaccats-stub-7.4.4.so again, breaking Bacula. Since ldconfig is run automatically by the system on various occasions this is pretty annoying.

The same problem exists with Bacula 9.0.6 from the repository https://download.opensuse.org/repositories/home:/cristyde/openSUSE_Leap_42.3/.

How can I fix ldconfig's idea where that symlink should point?

On one of our servers, yum history reports:

[tschmidt@sl-was01p ~]$ sudo yum history
Loaded plugins: product-id, search-disabled-repos, security, subscription-
              : manager
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    30 | <dlewandowski>           | 2016-10-07 11:18 | E, I, U        |   38 EE
    29 | <dlewandowski>           | 2016-09-16 16:13 | Erase          |    3   
[...]

But the reported login user swears he wasn't near the machine (physically or logically) anywhere near that time, and last seems to support this:

[tschmidt@sl-was01p ~]$ last|grep dle
dlewando pts/0        al-dlewandowski. Tue Oct 11 09:01 - 09:23  (00:22)    
dlewando pts/0        al-dlewandowski. Tue Oct 11 08:37 - 08:40  (00:02)    
dlewando pts/1        al-dlewandowski. Tue Oct  4 11:04 - 11:09  (00:04)    
dlewando pts/0        al-dlewandowski. Tue Oct  4 10:50 - 11:11  (00:21)    

Syslog doesn't report any sudo activity from or to that user around the yum activity in question, either.

I'd like to find out why yum history reports that apparently incorrect user. Where does it pull that information from? The username does not appear anywhere in /var/log/yum.log.

Given a Microsoft Active Directory domain configured as a subdomain of the company's public DNS domain, say:

• public domain contoso.com
• Active Directory domain inside.contoso.com

There are five Domain Controllers (DCs) in four sites. All of them are also DNS servers configured with contoso.com and inside.contoso.com as separate AD integrated zones. Three of the DCs are running Server 2008R2, the other two, Server 2012.

Now irritatingly, a folder named "inside" keeps appearing in zone contoso.com which contains A records for some or all of the DCs. If I delete all of these records the folder disappears, only to reappear a few minutes later with a single DC's A record, and A records for the other DCs added gradually. It looks as if, when the DCs register themselves in DNS, the entries get added to zone contoso.com instead of inside.contoso.com where they belong.

Simple question: why?

Note 1: All the DCs are also correctly listed as A records in zone intern.contoso.com. I do not know if these entries have been added manually, though.

Note 2: None of the other host entries in zone inside.contoso.com are ever duplicated to the inside folder in zone contoso.com.

A fully IPv6 dual stack enabled network is monitored by Nagios 3.5.1 installed from EPEL on a CentOS 6.7 server. Monitored hosts run NRPE agents, all of which are configured to run as daemons (not via xinetd) with the configuration line

allowed_hosts=bombur.example.com

where bombur.example.com is the Nagios server's FQDN which resolves in DNS to both the IPv4 and IPv6 addresses:

% host bombur
bombur.example.com has address 192.0.2.28
bombur.example.com has IPv6 address 2001:db8:f00:ba8::28

This works fine for hosts running NRPE releases before 2.15 which lack IPv6 support and ony accept IPv4 connections. But on hosts with NRPE 2.15, which does support IPv6, connections from the Nagios server are rejected with the log message:

nrpe[21665]: Host 2001:db8:f00:ba8::28 is not allowed to talk to us!

Apparently the allowed_hosts=<hostname> directive allows only the IPv4 address of the given host, not its IPv6 address.

Is that deduction correct? Is there a way around this behaviour, preferably without hardcoding the numeric IPv6 address in all the NRPE agent configurations?