How to wait for snapd auto-refresh to complete?

I have been trying to reproduce your issue, by creating an instance with the 'ubuntu-minimal-1804-lts' image as you state

gcloud compute instances create \
--image-family ubuntu-minimal-1804-lts \
--image-project ubuntu-os-cloud \
--machine-type n1-standard-1 \
--boot-disk-size=15 \
--boot-disk-type=pd-ssd \
--zone=europe-west1-c \
--tags=allow-incoming-ssh \
--metadata=startup-script-url="gs://testinggrounds/startup.sh" \
ubuntu-test

and then reboot it several times

for i in {1..7}; \
do \
gcloud compute instances reset ubuntu-test --zone europe-west1-c; \
sleep 120s; \
done

but I did not manage to get this problem that you state.

Searching around for the error that you get, I found this thread where the user reporting the issue presents a highly similar problem which might be the same as you have found. The problem is supposed to be fixed, as reported by the team behind it. Can you check if the date for your Ubuntu 18.04 image is prior to the date opening of this thread I shared? Maybe by using a newer image your problem gets solved.

I saw that you said this suddenly appeared not too long ago, so probably it is not the version, but it does not hurt to check.

If this was not the case, then I would propose to just delay the execution of the startup script or try to make it run after other snapd services.

[Unit]
Description=Google Compute Engine Startup Scripts
After=local-fs.target network-online.target network.target rsyslog.service
After=google-instance-setup.service google-network-daemon.service
After=cloud-final.service multi-user.target
After=snapd.seeded.service
After=snapd.service
Wants=local-fs.target network-online.target network.target cloud-final.service
Wants=snapd.seeded.service

[Service]
ExecStartPre=/bin/sleep 20
ExecStart=/usr/bin/google_metadata_script_runner --script-type startup
KillMode=process
Type=oneshot
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

[Install]
WantedBy=multi-user.target

This file is located at /lib/systemd/system/google-startup-scripts.service. I added the After=snapd.service and ExecStartPre=/bin/sleep 20 to try and delay the execution of the service. Your file should look exactly like the one above without my additions.

If it still doesn't work, you can try to add other snap services with the After= directive, as there are a few more in the systemd services folder:

$ ls /lib/systemd/system/ | grep snap

snapd.autoimport.service
snapd.core-fixup.service
snapd.failure.service
snapd.seeded.service
snapd.service
snapd.snap-repair.service
snapd.snap-repair.timer
snapd.socket
snapd.system-shutdown.service

Hope it helps!

I ended up running snap changes until everything was Done by putting this at the top of my startup-script:

# Returns 0 if snapd is not updating google-cloud-sdk
# (nothing has happened, or all changes are Done, Hold, Error)
# Returns 1 if snapd is updating google-cloud-sdk
# (there is at least one chnage that is Doing, Abort, Undo, or Undoing)
# Echos the offending line
# https://serverfault.com/questions/967674/how-to-wait-for-snapd-auto-refresh-to-complete
function isSnapIdle {
  # For the format of snap changes, see:
  # see https://github.com/snapcore/snapd/blob/8ae45c207f3c488f0ab0dc3615661df9b0854b20/overlord/state/change.go
  # and https://github.com/snapcore/snapd/blob/8ae45c207f3c488f0ab0dc3615661df9b0854b20/cmd/snap/cmd_changes.go#L124
  local line
  local changes
  if changes=$(snap changes google-cloud-sdk); then
    :
  else
    # When snap is upgrading itself, snap changes can fail
    echo "snap changes gave error $? $changes"
    return 2
  fi
  while read line; do
    local state=$(echo "$line" | awk '{print $2}')
    if [ "$state" = Do ] || [ "$state" = Doing ] || [ "$state" = Abort ] || [ "$state" = Undo ] || [ "$state" = Undoing ]; then
      echo "$line"
      return 1
    fi
  done < <(echo "$changes" | tail -n +2)
  return 0
}
function waitUntilSnapIdle {
  local lastSnapChange=
  local snapChange
  while ! snapChange=$(isSnapIdle); do
    if [ "$snapChange" != "$lastSnapChange" ]; then
      echo >&2 "Waiting for snapd to finish installing: $snapChange"
    fi
    lastSnapChange="$snapChange"
    sleep 1
  done
  echo >&2 "Snapd does not appear to be installing google-cloud-sdk"
}

waitUntilSnapIdle

I was encountering the same issue and pin pointed the cause to snap's auto updates as well! In my case waiting for snap refresh takes way too much time and the number of jobs is still manageable for a single human so I just reduced the allowed refresh window to the calmer part of the work week.

Try for instance:

# Allows snap refreshes on Monday at 10AM and also on Friday at 3PM
sudo snap set system refresh.timer=mon,10:00,,fri,15:00