yonran's questions

Is there a way to create a Cloud SQL Postgres instance that Enforces SSL/TLS, but that does not require the use of client certificates? In pg_hba.conf, this would be a line containing hostssl but not clientcert=1.

I have tried Enforcing SSL/TLS (settings.ipConfiguration.requireSsl on the instance, or gcloud sql instances patch [INSTANCE_NAME] --require-ssl) But this option seems pretty useless, because:

1. According to Managing your client certificates , you can only create “up to 10 client certificates for each instance.” This means that we would have to share the same client certificate/private keys among multiple users or services, so the secret is not truly secret among our employees.
2. After presenting a client sslkey/sslcert in psql, postgres still asks for a password. This is evidence that the Cloud SQL sslCerts API inserts a clientcert=1 auth-option, as opposed to a cert auth-method, into pg_hba.conf. This means that each client needs both a client cert and a password.

On Google Compute Engine, I have a startup-script that uses gsutil and gcloud, which are part of the google-cloud-sdk snap that is preinstalled on the ubuntu-minimal-1804-lts public image, which is the base image for the image that I am using. But when I ran the startup script today, it failed 4 out of 8 times because snapd decided to auto-refresh at the same time that the startup-script is run.

Within the startup-script, how can I wait until snap’s auto-refresh has completed?

Here are the commands I ran to determine that snapd was updating at the same time as my startup-script:

$ snap changes --abs-time
ID   Status  Spawn                 Ready                 Summary
3    Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Auto-refresh snaps "core", "google-cloud-sdk"

$ snap tasks 3 --abs-time
Status  Spawn                 Ready                 Summary
…
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:45Z  Ensure prerequisites for "google-cloud-sdk" are available
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:46Z  Download snap "google-cloud-sdk" (82) from channel "stable/ubuntu-18.04"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:47Z  Fetch and check assertions for snap "google-cloud-sdk" (82)
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Mount snap "google-cloud-sdk" (82)
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Run pre-refresh hook of "google-cloud-sdk" snap if present
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Stop snap "google-cloud-sdk" services
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Remove aliases for snap "google-cloud-sdk"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Make current revision for snap "google-cloud-sdk" unavailable
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Copy snap "google-cloud-sdk" data
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Setup snap "google-cloud-sdk" (82) security profiles
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Make snap "google-cloud-sdk" (82) available to the system
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Automatically connect eligible plugs and slots of snap "google-cloud-sdk"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Set automatic aliases for snap "google-cloud-sdk"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Setup snap "google-cloud-sdk" aliases
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Run post-refresh hook of "google-cloud-sdk" snap if present
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Start snap "google-cloud-sdk" (82) services
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Clean up "google-cloud-sdk" (82) install
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Run configure hook of "google-cloud-sdk" snap if present

$ journalctl -u google-startup-scripts
…
May 16 18:08:39 my-instance startup-script[1469]: INFO startup-script-url: + gsutil cp gs://my-bucket/app.tar.gz /opt/my-bucket/app.tar.gz
May 16 18:08:41 my-instance startup-script[1469]: INFO startup-script-url: /snap/google-cloud-sdk/41/usr/bin/python2: relocation error: /lib/x86_64-linux-gnu/libnss_dns.so.2: symbol 
__resolv_context_get, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

On another instance, gsutil failed with a different message:

ERROR: (gsutil) /snap/google-cloud-sdk/41/usr/bin/python2: command not found

How do you write a systemd service that gracefully shuts down upon machine shutdown or reboot? In particular, it should delay machine shutdown until it exits gracefully.

I have a service that takes 10 seconds to shut down: /usr/local/bin/shutdowntest.sh:

#!/bin/bash

SHUTDOWN=0
SHUTDOWN_TIME=10
TRAPPED_SIGNAL=

function onexit() {
  TRAPPED_SIGNAL=$1
  SHUTDOWN=1
}

for SIGNAL in SIGINT SIGTERM SIGHUP SIGPIPE SIGALRM SIGUSR1 SIGUSR2; do
  trap "onexit $SIGNAL" $SIGNAL
done

echo >&2 "shutdowntest running"
while ((!SHUTDOWN || SHUTDOWN_TIME>0)); do
  if [[ -n "$TRAPPED_SIGNAL" ]]; then
    echo >&2 "shutdowntest received signal $TRAPPED_SIGNAL"
    TRAPPED_SIGNAL=
  elif ((SHUTDOWN)); then
    echo >&2 "shutdowntest Shutting down: $SHUTDOWN_TIME more sleeps"
    SHUTDOWN_TIME=$((SHUTDOWN_TIME-1))
    sleep 1
  else
    sleep 10
  fi
done
echo >&2 "shutdowntest Finished shutting down; quitting"

I set TimeoutStopSec to 15s in /etc/systemd/system/shutdowntest.service:

[Service]
ExecStart=/usr/local/bin/shutdowntest.sh
TimeoutStopSec=15

[Install]
WantedBy=multi-user.target

When I run sudo systemctl stop shutdowntest.service, the service shuts down gracefully according to /var/log/syslog:

00:57:11 shutdowntest.sh[1980]: shutdowntest received signal SIGTERM
00:57:11 shutdowntest.sh[1980]: shutdowntest Shutting down: 10 more sleeps
00:57:11 systemd[1]: Stopping shutdowntest.service...
00:57:11 shutdowntest.sh[1980]: Terminated
00:57:11 shutdowntest.sh[1980]: shutdowntest Shutting down: 9 more sleeps
00:57:12 shutdowntest.sh[1980]: shutdowntest Shutting down: 8 more sleeps
00:57:13 shutdowntest.sh[1980]: shutdowntest Shutting down: 7 more sleeps
00:57:14 shutdowntest.sh[1980]: shutdowntest Shutting down: 6 more sleeps
00:57:15 shutdowntest.sh[1980]: shutdowntest Shutting down: 5 more sleeps
00:57:16 shutdowntest.sh[1980]: shutdowntest Shutting down: 4 more sleeps
00:57:17 shutdowntest.sh[1980]: shutdowntest Shutting down: 3 more sleeps
00:57:18 shutdowntest.sh[1980]: shutdowntest Shutting down: 2 more sleeps
00:57:19 shutdowntest.sh[1980]: shutdowntest Shutting down: 1 more sleeps
00:57:20 shutdowntest.sh[1980]: shutdowntest Finished shutting down; quitting
00:57:20 systemd[1]: Stopped shutdowntest.service.

But when I sudo reboot or sudo shutdown now the machine, the service is killed without enough time to exit gracefully, and the /var/log/syslog ends only 1s later.

00:59:30 shutdowntest.sh[2014]: Terminated
00:59:30 shutdowntest.sh[2014]: shutdowntest received signal SIGTERM
00:59:30 shutdowntest.sh[2014]: shutdowntest Shutting down: 10 more sleeps
00:59:30 systemd[1]: Stopping shutdowntest.service...

How to ensure that the service is given the time (TimeoutSec or TimeoutStopSec) to exit when the machine is shutdown or rebooted?

I would like to see GCE HTTP/HTTPS Load Balancer access logs. By default, access logs go to Stackdriver. But we have to exclude the log named “cloud-http-load-balancer” because Stackdriver will start charging on March 31, 2018, and Stackdriver logs are obscenely expensive ($0.50/GiB, and we get many TiB per month so it would be over $10,000/month).

How can we get Cloud Load Balancer access logs, but without Stackdriver? In comparison, AWS allows you to setup ELB to export access logs to S3 for no added charge other than the price of S3 itself.

The documentation About the Cloud SQL Proxy contains a line, “[B]ecause the proxy always connects from a hostname that cannot be accessed except by the proxy, you can create a user account that can be used only by the proxy. The advantage of doing this is that you can specify this account without a password without compromising the security of your instance or your data.”

Is there any practical way of connecting a MySQL client through the Google Cloud Proxy which is authenticated using a Service Account that is only allowed to access a specific MySQL user?

One potential way to restrict access when using the Cloud SQL Proxy is to create a MySQL user account with Cloud SQL Proxy IP address: '[USER_NAME]'@'cloudsqlproxy~[IP_ADDRESS]'. But this would not work for a service account that runs on multiple cloud VMs. This is basically authentication of the MySQL user by IP address instead of by Service Account.

A second way to restrict a Service Account to a MySQL user might be to use a MySQL user account '[USER_NAME]'@'cloudsqlproxy~%' and rely on IAM to restrict access. The Service Account running the Cloud SQL Client requires the roles/cloudsql.client role, which is equivalent to the cloudsql.instances.connect and cloudsql.instances.get permissions. Unfortunately, if you grant multiple Service Accounts this role, then it looks like there is no way to restrict them from using each other’s MySQL user.

A third potential way to enable passwordless authentication might be to use client certificates (gcloud sql ssl-certs create [CERT_NAME] client-key.pem --instance [INSTANCE_NAME]). But client private keys are even harder to manage than passwords, since they are all invalidated at the same time in under 1 year when you refresh the server certificate: “When you refresh your server cerficate, you must also generate new client certificates; the existing client certificates are revoked.” This makes them basically impossible to use in production.

Basically, it seems like the Cloud SQL Proxy does not enable authenticating to a specific MySQL user via an IAM Service Account. Is this correct?

Is there a way to automatically apply Labels to instances that are launched by a GCE Managed Instance Group (aka. Instance Group Manager)? I see that you can add Network Tags in the Instance Template (properties.tags), but I don’t see a way to add Labels for use in searching for labeled instances using instances/list.

A workaround would be to manually call setLabels on instances from the Instance Template’s startup-script.

In AWS EC2, the equivalent would be to add Tags to an AutoScalingGroup with PropagateAtLaunch: true.

When I create a node-pool for a specific purpose, is there a way to tell Kubernetes not to put kube-system pods such as kube-proxy on those nodes (which consumes memory)? When this happens, the actual pod that is supposed to request all the memory on these nodes becomes Pending forever.

$ kubectl get pods -l app=my-app
…
my-app-2    0/2       Pending   0          2d
$ kubectl describe nodes gke-my-cluster-my-node-pool-f05270d7-7c5d
…
Non-terminated Pods:        (3 in total)
  Namespace         Name                                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                                        ------------    ----------  --------------- -------------
  kube-system           fluentd-cloud-logging-gke-my-cluster-my-node-pool-f05270d7-7c5d     100m (5%)   0 (0%)      200Mi (1%)  200Mi (1%)
  kube-system           kube-dns-3263495268-hw5bl                           260m (13%)  0 (0%)      110Mi (0%)  170Mi (1%)
  kube-system           kube-proxy-gke-my-cluster-my-node-pool-f05270d7-7c5d            100m (5%)   0 (0%)      0 (0%)      0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  460m (23%)    0 (0%)      310Mi (2%)  370Mi (2%)

What is the length limit for URLs on the Google HTTP/HTTPS Cloud Load Balancer aka HTTP/HTTPS Forwarding Rule? I have had reports of HTTP/1.1 413 Request Entity Too Large error responses from the load balancer:

Is there a way to adjust the length limit?

Is there a way to tell (from load balancer metrics or logs) how many requests are rejected due to the length limit? There seems to be no results in StackDriver logs:

Related question: Amazon AWS ELB length limit

What is the simplest way to use the gcloud command line non-interactively with a Service Account outside of GCE? Preferably without littering the file system with credentials files, which is what gcloud auth activate-service-account --key-file=... does.

There are many use cases for using gcloud with a service account. For example, on a server, I would like to test that the GOOGLE_APPLICATION_CREDENTIALS is correctly set and has the required permissions before running my application. Or, I would like to run some setup scripts or cron scripts that perform some check with the gcloud command line.

Google Cloud libraries (e.g. python, java) automatically use the environment variable GOOGLE_APPLICATION_CREDENTIALS to authenticate to Google Cloud. But unfortunately, this command line seems to have no effect on gcloud. What is a clean way to use gcloud while leaving the filesystem intact?

$ GOOGLE_APPLICATION_CREDENTIALS=/etc/my-service-account-4b4b6e63aaed.json gcloud alpha pubsub topics publish testtopic hello
ERROR: (gcloud.alpha.pubsub.topics.publish) You do not currently have an active account selected.
Please run:

  $ gcloud auth login

to obtain new credentials, or if you have already logged in with a
different account:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.

We would like to increase the size of the logs that kubectl log retrieves.

As of Container-Optimized OS (COS) VERSION_ID=56 BUILD_ID=9000.84.2 (according to /etc/os-release), the following logrotate script is used: /etc/logrotate.d/docker-containers:

/var/lib/docker/containers/*/*-json.log {
    rotate 5
    copytruncate
    missingok
    notifempty
    compress
    maxsize 10M
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}

How can I increase the maxsize from 10M?

One thing I thought would be set the user-data metadata within the nodePools creation, but the NodeConfig documentation states that this is impossible: “keys must not conflict with any other metadata keys for the project or be one of the four reserved keys: "instance-template", "kube-env", "startup-script", and "user-data"”

With EC2 auto scaling, when using a step scaling policy (as opposed to a simple scaling policy) that scales in due to an alarm that is based on an AWS/EC2 metric (e.g. CPUUtilization <= 30%) with detailed CloudWatch monitoring disabled, when my auto scaling group scales in, it scales in two instances in short succession, without waiting for the metric to update. How can I prevent the auto scaling group from scaling in too fast for the metric to update?

Edit: here was last night’s scaling history. At 5:15, 5:17, 5:19, 5:21 UTC, auto scaling scaled in due to low CPU Utilization, even though CPU Utilization only has data points at 5:10, 5:15, 5:20, and the scaling event should have ceased after the 5:15 datapoint. There appears to be no way to adjust the cooldown duration of step scaling policy scale ins (step scaling policies ignore the Default Cooldown (=600s) and only scale out policies have estimated instance warmup).