Questions[google-stackdriver](server)

I have a VM on GCE where I run a custom Docker image. I installed it on COS (cos-stable-74-11895-125-0) to a g1-small (1 vCPU, 1.7 GB memory) instance.

It is an Elasticsearch with locked memory settings. It consumes exactly 1 GB of RAM.

The setup worked perfectly for nearly one year, but suddenly it stopped working as it runs out of memory.

On the serial console, it logs that the oom-killer was invoked and it selected the java process to be killed. After a quick restart, it works perfectly, but after about a day or two, it fails with the same out of memory error.

It turned out, that GCE installs and runs a Stackdriver Logging Agent in parallel with my container. According to the oom-killer log, a related ruby process consumes a bunch of RAM. After a reboot, it uses 50 MB of RAM and it stacks up to 300MB in a few hours. From my point of view, it seems to be a memory leak.

To make it clear: The ES server is not receiving any load, just a periodic uptime check is making requests about every 5 minutes. As a result, there are no vast amounts of logs, only a few lines per hour. A whole year of logs would not take up 300MB of storage or memory.

I tried to examine the memory footprint using the ps and top commands but the results were totally implausible. According to the system, the ruby task consumes 1-10GB of physical memory (RSS) and up to 70GB of virtual memory (VSZ). (Memory swapping is disabled according to the free command.) These figures can't be true as the VM does not have these amounts of RAM.

I tried to update the OS to a more recent version (cos-81-12871-119-0). It definitely helped as the ES was running for more than 5 days now, without a problem. But the ruby process's memory consumption is worrying. According to the ps and top commands, it uses 70-300MB of physical memory (RSS) and up to 7GB of virtual memory (VSZ).

I found that the ruby process is part of the fluentd tool which had a similar memory problem according to their GitHub. As the configuration and installation are automatically done by GCE I did not found a way to change its settings.

The easiest solution would be to disable the installation of the logging agent by changing the VM metadata. (google-logging-enabled=false). But I would like to find out the reason why this happens and how can I solve it.

I'm really curious if anybody had a similar problem, and what was the solution.

Stackdriver Logging Agent Docker image: gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1

The error can be reproduced. I have a development and a production environment in separate GCP projects. The error appeared on both, roughly at the same time. (~2020. May. 28.) According to this, I think this is caused by a change in Stackdriver but I did not found any evidence in the logs. (Both VM instances are in eu-west3 region, in c zone.)

I would appreciate any help or advice. Thanks!

I have set up the "connector account" thing in Stackdriver, and I'm able to see metrics in GCP, which is super cool! I would now like to also get all cloudwatch logs into google's logging tool. Can this be done with the connector account as well? Or will I have to write a lambda to ship those off?

Tell me whether it is possible to make a separation by severity in stackdriver. All messages in the container are displayed in stderror and in stackdriver, they are marked as errors. The task is to separate messages from logs (laravel) into categories information, error, warning.

Google Kubernetes Engine + Stackdriver logging enabled.

I'd like to set up instrumentation for OOMKilled events, which look like this when examining a pod:

Name:   pnovotnak-manhole-123456789-82l2h
Namespace:  test
Node:   test-cluster-cja8smaK-oQSR/10.x.x.x
Start Time: Fri, 03 Feb 2017 14:34:57 -0800
Labels:   pod-template-hash=123456789
    run=pnovotnak-manhole
Status:   Running
IP:   10.x.x.x
Controllers:  ReplicaSet/pnovotnak-manhole-123456789
Containers:
  pnovotnak-manhole:
    Container ID: docker://...
    Image:    pnovotnak/it
    Image ID:   docker://sha256:...
    Port:
    Limits:
      cpu:  2
      memory: 3Gi
    Requests:
      cpu:    200m
      memory:   256Mi
    State:    Running
      Started:    Fri, 03 Feb 2017 14:41:12 -0800
    Last State:   Terminated
      Reason:   OOMKilled
      Exit Code:  137
      Started:    Fri, 03 Feb 2017 14:35:08 -0800
      Finished:   Fri, 03 Feb 2017 14:41:11 -0800
    Ready:    True
    Restart Count:  1
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tder (ro)
    Environment Variables:  <none>
Conditions:
  Type    Status
  Initialized   True
  Ready   True
  PodScheduled  True
Volumes:
  default-token-46euo:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-tder
QoS Class:  Burstable
Tolerations:  <none>
Events:
  FirstSeen LastSeen  Count From                SubObjectPath       Type    Reason    Message
  --------- --------  ----- ----                -------------       --------  ------    -------
  11m   11m   1 {default-scheduler }                      Normal    Scheduled Successfully assigned pnovotnak-manhole-123456789-82l2h to test-cluster-cja8smaK-oQSR
  10m   10m   1 {kubelet test-cluster-cja8smaK-oQSR} spec.containers{pnovotnak-manhole}  Normal    Created   Created container with docker id xxxxxxxxxxxx; Security:[seccomp=unconfined]
  10m   10m   1 {kubelet test-cluster-cja8smaK-oQSR} spec.containers{pnovotnak-manhole}  Normal    Started   Started container with docker id xxxxxxxxxxxx
  11m   4m    2 {kubelet test-cluster-cja8smaK-oQSR} spec.containers{pnovotnak-manhole}  Normal    Pulling   pulling image "pnovotnak/it"
  10m   4m    2 {kubelet test-cluster-cja8smaK-oQSR} spec.containers{pnovotnak-manhole}  Normal    Pulled    Successfully pulled image "pnovotnak/it"
  4m    4m    1 {kubelet test-cluster-cja8smaK-oQSR} spec.containers{pnovotnak-manhole}  Normal    Created   Created container with docker id yyyyyyyyyyyy; Security:[seccomp=unconfined]
  4m    4m    1 {kubelet test-cluster-cja8smaK-oQSR} spec.containers{pnovotnak-manhole}  Normal    Started   Started container with docker id yyyyyyyyyyyy

All I get from the pod logs is;

{
 textPayload: "shutting down, got signal: Terminated
"
 insertId: "aaaaaaaaaaaaaaaa"
 resource: {
  type: "container"
  labels: {
   pod_id: "pnovotnak-manhole-123456789-82l2h"
   ...
  }
 }
 timestamp: "2017-02-03T22:34:48Z"
 severity: "ERROR"
 labels: {
  container.googleapis.com/container_name: "POD"
  ...
 }
 logName: "projects/myproj/logs/POD"
}

And the kublet logs;

{
 insertId: "bbbbbbbbbbbbbb"   
 jsonPayload: {
  _BOOT_ID: "ffffffffffffffffffffffffffffffff"    
  MESSAGE: "I0203 22:41:11.925928    1843 kubelet.go:1816] SyncLoop (PLEG): "pnovotnak-manhole-123456789-82l2h_test(a-uuid)", event: &pleg.PodLifecycleEvent{ID:"another-uuid", Type:"ContainerDied", Data:"..."}"
 ...

Which doesn't seem like quite enough to uniquely identify this as an OOM event. Any other ideas?

Have a cluster setup with the following permissions.

I tried creating a node pool with new permissions, which seems to be able to enable some things. I didn't find the scope required for stackdriver trace permissions in the documentation located here.

Further, trying to enable monitoring via creating a node pool, and deleting the old one didn't seem to successfully flip the switch.

gcloud container node-pools create pool-2 \
  --cluster=cluster-1 \
  --scopes=compute-rw,storage-rw,taskqueue,logging-write,monitoring-write,datastore,service-control,service-management