Tim Stoop's questions

I have a very weird issue on Debian Buster. I've enabled unattended-upgrades on the server as this is a very bare bones server and it should just update automatically. However, it seems that the apt timers for this never start.

When I check all timers, I get the following:

# systemctl list-timers
NEXT                         LEFT          LAST                         PASSED               UNIT                         ACTIVATES
Fri 2021-01-15 20:00:00 UTC  58min left    Fri 2021-01-15 19:00:00 UTC  1min 13s ago         logrotate.timer              logrotate.service
Sat 2021-01-16 00:00:00 UTC  4h 58min left Fri 2021-01-15 00:00:00 UTC  19h ago              man-db.timer                 man-db.service
Sat 2021-01-16 18:03:26 UTC  23h left      Fri 2021-01-15 18:03:26 UTC  57min ago            systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a                          n/a           Thu 2020-11-26 06:15:45 UTC  1 months 20 days ago apt-daily-upgrade.timer      apt-daily-upgrade.service
n/a                          n/a           Wed 2020-11-25 20:32:27 UTC  1 months 20 days ago apt-daily.timer              apt-daily.service

When I check the definition of the timer, it seems fine:

# systemctl cat apt-daily.timer
# /lib/systemd/system/apt-daily.timer
[Unit]
Description=Daily apt download activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target

And analyze seems to agree that the syntax is fine:

# systemd-analyze calendar "*-*-* 6,18:00"
  Original form: *-*-* 6,18:00
Normalized form: *-*-* 06,18:00:00
    Next elapse: Sat 2021-01-16 06:00:00 UTC
       From now: 10h left

But they do not fire. I tried running systemctl start apt-daily.timer, systemctl enable --now apt-daily.timer and systemctl restart timers.target, no effect. The command returns without an error, but nothing changes.

I'm at a loss on how to debug this further, any hints would be much appreciated.

EDIT As requested, the content of the apt-daily.service unit:

# /lib/systemd/system/apt-daily.service
[Unit]
Description=Daily apt download activities
Documentation=man:apt(8)
ConditionACPower=true
After=network.target network-online.target systemd-networkd.service NetworkManager.service connman.service

[Service]
Type=oneshot
ExecStartPre=-/usr/lib/apt/apt-helper wait-online
ExecStart=/usr/lib/apt/apt.systemd.daily update

EDIT The service itself is not disabled, either, but it's dead:

# systemctl status apt-daily.service
● apt-daily.service - Daily apt download activities
   Loaded: loaded (/lib/systemd/system/apt-daily.service; static; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:apt(8)

This might actually not be possible, but trying anyway: I want to create a container image for Apache without the need to run as root. So I set up the VirtualHost to listen to port 8080 instead of anything <1024. There's a proxy before Apache which directs the traffic for port 80 to port 8080 on the container. However, when I make Apache redirect, it always adds the port to the Location header. So the redirect send back is "Location: http://mysite:8080/index", which will fail as there's nothing listening on port 8080 publicly.

Is there any way of telling Apache to never add the port number to these redirects? I tried fiddling with UseCanonicalPhysicalPort, but that does not seem to have the desired effect. Is there any way to prevent it from ever adding that port number to anything?

EDIT

Ok, looks like this behaviour is hardcoded, when I look at the source code of mod_rewrite I notice it calls "ap_is_default_port" to determine whether it should add the port in the answer or not. And the only way to override it would be by overriding this line in the source, I think. If anyone knows of a better answer, I'm all ears.

We're running across three zones in GCE with a TCP Proxy loadbalancer in front. The backend for the loadbalancer is on a single node within one of the zones. Using the health check, the loadbalancer can determine where to send the traffic. This works as expected when traffic originates outside of Google. For outgoing traffic we use a NAT gateway and a route with a lower priority than the default Google route (800 instead of 1000) to actually route the traffic for 0.0.0.0/0 there.

However, when one of the backend servers that's not in the same zone as the (active) backend for the TCP loadbalancer tries to access the external address, it will get a connection time out. The route does not seem to get used. If the request originates from the same zone as the active backend, it works. The nat gateway route does not seem to be used.

One solution would of course be to make sure that there's an active backend in all zones, but we prefer not to do that. One thing we note is that when we get traffic from within the cluster through this external IP address, the client IP address in our logs seems to be the loadbalancer address itself. Any idea how to solve this otherwise? Is this expected behaviour or did we possibly misconfigure some route or firewall?

I'm trying to enable os login for an instance. I've already set the Metadata os-login: TRUE and added my public key to my account with the gcloud compute os-login ssh-keys add command. Now, I'm trying to add my role to the instance, as described here: https://cloud.google.com/compute/docs/instances/managing-instance-access#configure_users

I'm running into an issue with the command however and the error message doesn't make it clear to me what's going wrong. Here's my command and the output:

$ gcloud beta compute instances add-iam-policy-binding tim-test --project='my-project' --zone='europe-west4-a' --member='user:[email protected]' --role=roles/compute.osAdminLogin
ERROR: (gcloud.beta.compute.instances.add-iam-policy-binding) HTTPError 400: Invalid value 'projects/packer-automation/zones/europe-west4-a/instances/tim-test'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?|[1-9][0-9]{0,19}'

What am I doing wrong?

I'm trying to create a DaemonSet with a specific affinity, I want it only to create Pods on nodes with type=prod. I use the following test code:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: test0
  namespace: kube-system
spec:
  template:
    metadata:
      labels:
        app: test0
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In
                values:
                - prod
      containers:
      - name: test0
        image: gcr.io/google_containers/pause:2.0

However, the kubectl exits with the following error:

error: error validating "test.yaml": error validating data: found invalid field affinity for v1.PodSpec; if you choose to ignore these errors, turn validation off with --validate=false

I'm at a loss as to waht I'm doing wrong here. I tried placing the affinity block under the template block as well, same error.

I should note that the cluster is still Kubernetes 1.4.8, though. Affinity is part of Kubernetes since 1.2, so I don't think that would be the problem?

Is it possible to add a variable via the boot prompt to the Debian installer, so that variable can be used in a preseed file?

In particular, I'm trying to solve the following problem:

We have a pretty extensive post-install script that is generally downloaded from a server. But now I want to create Packer images and keep the post-install script in version control together with the other Packer files. For accessing the preseed, I can do "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg" in the boot command. But now I want the installer to download the post-install script from that same location.

Currently, the post-install hook looks like this:

d-i preseed/late_command string wget -q -O /tmp/postinstall.sh http://our.public.server/postinstall.jessie.sh ; sh /tmp/postinstall.sh

Ideally, I'd like to do something like:

d-i preseed/late_command string wget -q -O /tmp/postinstall.sh http://{{ .HTTPIP }}:{{ .HTTPPort }}/postinstall.jessie.sh ; sh /tmp/postinstall.sh

But of course the Debian installer will not replace those with the required values. So I was thinking that it might be possible to pass environment variable-like variables to the installer that we can use in the preseed file.

Any hints or tips are appreciated!

EDIT: Tried adding the late_command to the boot command, but that didn't get picked up.

EDIT: Tried preseed/run, but it runs in a different environment which does not allow the in-target command.

EDIT: This can be a workaround: How do I pipe commands together in a debian preseed file? But I'd prefer to have the script in a separate file. If it's not possible, it's not possible, though.

We've recently upgraded a MySQL 5.0 master-master setup to Percona 5.6. Slaving went b0rken due to some failures on our side, but we thought we could simply fix it by using xtrabackup to create a backup from the running server and importing it to the slave. I've been trying to do that this entire weekend (partly because it's a huge database with an insane amount of databases and tables), but to no avail. Can someone shed a light on what I might be doing wrong here?

First, I run the following on the current in production master:

ulimit -n 409600 innobackupex --defaults-file=/etc/mysql/debian.cnf /mnt

When that is done, I copy the resulting directory to the other server and run:

innobackupex --use-memory=4G --apply-log /srv/restore

It exits with an OK message eventually. Now I restore it to the database with:

innobackupex --move-back /srv/restore

All goes well and I can start MySQL again (after I chown the /srv/mysql directory, which is our datadir). Data is in there and the database is running fine. Now I start slaving on this database:

/usr/bin/mysql --defaults-file=/etc/mysql/debian.cnf -e "CHANGE MASTER TO MASTER_HOST='10.x.x.x', MASTER_USER='replication', MASTER_PASSWORD='verysecret', MASTER_AUTO_POSITION=1; START SLAVE"

Slaving starts but immediately stops due to a 1062 error. After investigation, I find out that the entry it's trying to apply was added on the master db right after I started the backup. I can fix that, but I immediately get a new error.

To me, it seems like the backup did not contain all the latest GTIDs, only the ones that were available at the start of the backup? I thought this was exactly what XtraBackup was supposed to fix? I see no alternative now to making sure no writes are done on the database during a backup. Am I doing something wrong here? Is this supposed to happen?

Running on Debian Wheezy with all the latest patches.

Server version: 5.6.25-73.1-log Percona Server (GPL), Release 73.1, Revision 07b797f $ innobackupex --version InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. $ xtrabackup --version xtrabackup version 2.2.11 based on MySQL server 5.6.24 Linux (x86_64) (revision id: )

I've got the following pinnings set:

Package: linux-image-amd64
Pin: release a=wheezy-backports
Pin-Priority: 1001

Package: *
Pin: release a=wheezy-backports
Pin-Priority: 499

I get the following error:

$ sudo apt-get install linux-image-amd64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-image-amd64 : Depends: linux-image-3.16.0-0.bpo.4-amd64 but it is not going to   be installed
E: Unable to correct problems, you have held broken packages.

The policies seem to be correct:

$ apt-cache policy linux-image-amd64
linux-image-amd64:
  Installed: 3.2+46
  Candidate: 3.16+63~bpo70+1
  Package pin: 3.16+63~bpo70+1
  Version table:
     3.16+63~bpo70+1 1001
        499 http://ftp.nl.debian.org/debian/ wheezy-backports/main amd64 Packages
 *** 3.2+46 1001
        500 http://ftp.nl.debian.org/debian/ wheezy/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy linux-image-3.16.0-0.bpo.4-amd64
linux-image-3.16.0-0.bpo.4-amd64:
  Installed: (none)
  Candidate: 3.16.7-ckt4-3~bpo70+1
  Version table:
     3.16.7-ckt4-3~bpo70+1 0
        499 http://ftp.nl.debian.org/debian/ wheezy-backports/main amd64 Packages

So I don't get the error. What am I doing wrong?

I'm trying to use puppet to upgrade all Wheezy machines to the backports kernel (a reboot streak is planned), which is why I want to solve it correctly. I could manually login to all servers to run:

sudo apt-get install -t wheezy-backports linux-image-amd64

Which works, but is not as convenient.

UPDATE

As requested:

/etc/apt/preferences.d$ cat linux-image 
Package: linux-image-*
Pin: release a=wheezy-backports
Pin-Priority: 1001

Package: *
Pin: release a=wheezy-backports
Pin-Priority: 499

/etc/apt/preferences.d$ sudo apt-get update
Hit http://debian.kumina.nl wheezy-kumina Release.gpg
.
.
.
Reading package lists... Done
/etc/apt/preferences.d$ sudo apt-get install linux-image-amd64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-image-amd64 : Depends: linux-image-3.16.0-0.bpo.4-amd64 but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
/etc/apt/preferences.d$ aptitude why-not linux-image-3.16.0-0.bpo.4-amd64
Unable to find a reason to remove linux-image-3.16.0-0.bpo.4-amd64.

We're having a really strange problem when trying to boot from an mdraid on Debian Wheezy. All references online that I can find tell me that grub 1.99 (which is part of Debian Wheezy) should have no trouble booting from an mdraid volume. We're keep getting a grub rescue prompt, however.

We verified that grub can work with the disks, since booting from a non-raid ext4 formatted partition works without problem. As soon as we put /boot on a RAID array we created with mdadm, grub no longer recognises it.

Although we started out with a RAID5 array with LVM on top, while testing we've moved back to a simply /boot on a 4-disk RAID1 array. These are 4TB disks, so we're using GPT. We installed grub on all disks with the following command:

grub-install --no-floppy --modules="raid mdraid09 mdraid1x" /dev/sda

And for sdb, sdc and sdd, of course.

Grub keeps throwing us to grub rescue. An ls at this time only shows disks and gpt partitions, no md partitions. We've tried recreating the RAID1 with --metadata=0.9, but that didn't change the behaviour at all.

The hardware is a Dell PowerEdge R520 with the PERC 710i RAID controller. We've create RAID0s in the RAID controller for each disk and this seems to work as expected.

No obvious errors are thrown at installation time, either the OS or grub complain.

A reinstall of the grub-pc package doesn't solve the problem either.

We have no further idea what to try and are hoping for some input!

EDIT

We have indeed installed grub to every disk. We are getting a grub prompt, it just cannot read the mdraid. If we add a 'normal' ext4 partition to the machine to boot from, it works.