Thorsten Staerk's questions

I am following https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule. When testing the Cloud Function, I get the error message:

Error: Required 'compute.instances.list' permission for 'projects/thorstenstaerk-cert' at Function.parseHttpError (/workspace/node_modules/google-gax/build/src/googleError.js:59:37) at decodeResponse (/workspace/node_modules/google-gax/build/src/fallbackRest.js:72:49) at Promise.all.then (/workspace/node_modules/google-gax/build/src/fallbackServiceStub.js:90:42) at process._tickCallback (internal/process/next_tick.js:68:7)

Now, I recall I have to give this permission to the default service account of Cloud Functions, but where do I find it? It's not in IAM.

Trying to deploy a VM landscape using https://github.com/GoogleCloudPlatform/sap-deployment-automation, I often get the message:

FAILED - RETRYING: wait for hosts (12 retries left).

It usually repeats failing, and I resolve it by trying various things to enable ssh access. But this time I would like to get some ideas how to make it better than blindly trying.

How can I debug Terraform?

When trying to create a "hello world" Cloud Function, I get the error message:

"The request has violated one or more Org Policies. Please refer to the respective violations for more information."

Now, which org policies have been violated? In the Log Explorer I find the error message like this:

{
insertId: "XXX"
logName: "projects/XXX/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload: {10}
receiveTimestamp: "2021-11-26T11:42:16.735011108Z"
resource: {2}
severity: "ERROR"
timestamp: "2021-11-26T11:42:16.490247Z"
}

On Linux when using dsa keys I establish passwordless login with the command

ssh-copy-id -i .ssh/id_dsa.pub [email protected]

This prompts the user for user's password on target.host. My script knows the password and the user shall not be bothered, how can I do this?

My physical server hangs randomly and I do not get any log entries during that time. So I let a script run:

while true; do date >>/tmp/dates; sleep 1; done & disown

After some minutes the file /tmp/dates has "holes". Sometimes the dates are interrupted for 30 seconds, sometimes for 3 minutes in a row:

Thu Feb 13 14:54:39 CET 2014
Thu Feb 13 14:54:40 CET 2014
Thu Feb 13 14:57:45 CET 2014
Thu Feb 13 14:57:46 CET 2014

The server does not show high load or memory utilization. /var/log/messages does not show anything for the times when the server hangs. However it does have some messages about IO problems like a fibrechannel link taking a nap. I am booting from SAN, so / is on a multimapper device. If I call

while true; do date >>/tmp/dates; sleep 1; done & disown
while true; do date >>/dev/shm/dates; sleep 1; done & disown

There will be no holes in /dev/shm/dates, but there will be holes in /tmp/dates so I think it is not a kernel hang situation, but an IO hang situation.

I can see no processes in uninterruptible sleep state in top. No entries in the hardware or storage log that seem relevant.

However iostat shows high latency spikes regarding await:

03/28/14 12:32:00
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00   214.00    0.00   0.00 100.00

03/28/14 12:32:01
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00   214.00    0.00   0.00 100.00

03/28/14 12:32:02
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00   214.00    0.00   0.00 100.00

03/28/14 12:32:03
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00 1219.00     0.00     4.76     8.00    45.61 5251.48   0.82 100.40

03/28/14 12:32:04
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     2.99    0.00   0.00  99.60

03/28/14 12:32:05
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     3.01    0.00   0.00 100.40

03/28/14 12:32:06
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     3.00    0.00   0.00 100.00

Could it be that the kernel is waiting for IO from the storage and does not do anything until the block arrives? What else could it be? How can I find out?

For sizing purposes I need to understand what requirements an application has to the I/O subsystem. I would love to do what I call I/O sniffing and obtain a list of events from the block layer like this:

initiator XYZ requests block 4711 from device 0815
initiator BLA writes block 1234 to device 9876 

I have been told blktrace was what I am looking for but I cannot get this info from that tool.

I can obtain a wonderful memory map of all processes with the Linux command

pmap $(ps -A | awk '{print $1}'|grep -v PID) | sort | grep \^0

It looks like

...
00007fd6dbf45000      4K rw---  /lib/libnss_compat-2.11.1.so 
00007fd6dbf46000   1524K r-x--  /lib/libc-2.11.1.so
00007fd6dc0c3000   2044K -----  /lib/libc-2.11.1.so  
...

All processes' memory maps are merged here. But I am missing the information where the kernel has its memory pages. Is there a similar tool for the "complete memory map"?