Home / server / Questions / 1093143
Accepted
questionto42standswithUkraine
Asked: 2022-02-10 03:24:19 +0800 CST

jsonPayload (structured logging) output from Python Google Cloud Function's logging needed to create Log-Based Metrics (LBM) in GCP

  • 772

I need the jsonPayload in the logs of a Google Cloud Function instead of the textPayload. My aim is to use the keys of the dictionary as labels (see Log-based metrics labels) for Log-Based Metrics so that these can be addressed in Grafana.

I am using Python's logging module, but I can also switch to something else if needed.

I need as output in the logs:

jsonPayload: `{'key1':value1, 'key2':value2}`

But I get a textPayload output instead, the whole next line is a string:

"2022-02-08 15:43:32,460 [INFO]: {"key1": value1, "key2": value2}"

Real example from the logs, in the middle, you see the textPayload:

enter image description here

The picture as text:

{
insertId: "000000-1b431ffd-e42d-4f83-xyz"
labels: {1}
logName: "projects/MY_PROJECT/logs/cloudfunctions.googleapis.com%2Fcloud-functions"
receiveTimestamp: "2022-02-08T15:43:41.808217166Z"
resource: {2}
textPayload: "2022-02-08 15:43:32,460 [INFO]: {"json_metadata": {"countrows": 736203, "countcolumns": 6, "size": 48261360, "gcs_stamp": "2022-02-08 15:43:32.451000+00:00", "python_stamp": "2022-02-08 15:43:31.055538"}}"
timestamp: "2022-02-08T15:43:32.460Z"
trace: "projects/MY_PROJECT/traces/dd97759176248586a3d3xyz"
}

First tries

Reading from https://cloud.google.com/logging/docs/structured-logging:

In Cloud Logging, structured logs refer to log entries that use the jsonPayload field to add structure to their payloads. Structured logging applies to user-written logs.

I tried to get this "structured logging" following Writing structured logs by

logging.info(json.dumps(json_for_gcp_lbm))

but to no avail.

Further in the links: there is a built-in Logging agent from GCP that uses fluentd as to About the Logging agent seems to be available only for Google Kubernetes Engine or the App Engine, not in a Google Cloud Function:

If you're using Google Kubernetes Engine or the App Engine flexible environment, you can write structured logs as JSON objects serialized on a single line to stdout or stderr. The Logging agent then sends the structured logs to Cloud Logging as the jsonPayload of the LogEntry structure.

How can I get the jsonPayload in this output?

logging python google-cloud-platform google-cloud-functions

1 Answers

  1. Best Answer

    You can set up structured logging following the example in the docs. Make sure your python version is 3.8 or above. Another way to achieve that is to use Logging client libraries.

    Example from the OP for the second method:

    Put into your requirements.txt:

    google-cloud-logging==3.0.0

    as the most recent version at the time of writing.

    from google.cloud import logging as gclogger

    (or name it as you like)

    and call the json logging with:

    json_for_gcp_lbm = {MY_JSON_HERE}

        from google.cloud import logging as gclogger

    ...

    logging_client = gclogger.Client()
    logger_name = "MY_LOG_NAME_OF_FREE_CHOICE" # Saved in the google cloud logs
    logger = logging_client.logger(logger_name)

    # I do not know how it would work with `import logging` as the 
    # built-in Python module, but I got it to work with adding
    # Following line does not work
    # logging.info(json.dumps(json_for_gcp_lbm, default=str))
    # Instead, we need 
    # https://cloud.google.com/logging/docs/samples/logging-write-log-entry#code-sample
    logger.log_struct(json_for_gcp_lbm)

    Which leads to a jsonPayload output:

    enter image description here

    And you can then choose any label from the json in the "Create logs metric" (Log-Based Metrics / LBM) menu:

    enter image description here

    • 1