Questions[google-cloud-functions](server)

I am deploying a Cloudfunction with VPC network as follows:

gcloud beta functions deploy my-function
      --trigger-http
      --region europe-west1
      --memory 128MB
      --runtime nodejs16
      --entry-point entrypoint
      --allow-unauthenticated
      # needed to access compute instances
      # https://console.cloud.google.com/networking/connectors/list
      --vpc-connector cloud-function-connector
      # vpc connector should be used only to access private network
      --egress-settings private-ranges-only

Now, if my Cloudfunction uses IP address of compute resources, I can easily access them. However when I use their hostnames, the DNS are not resolved which ends up with:

Error: getaddrinfo ENOTFOUND my-compute-resource

What do I need to do to be able to use DNS for my compute instances?

I need the jsonPayload in the logs of a Google Cloud Function instead of the textPayload. My aim is to use the keys of the dictionary as labels (see Log-based metrics labels) for Log-Based Metrics so that these can be addressed in Grafana.

I am using Python's logging module, but I can also switch to something else if needed.

I need as output in the logs:

jsonPayload: `{'key1':value1, 'key2':value2}`

But I get a textPayload output instead, the whole next line is a string:

"2022-02-08 15:43:32,460 [INFO]: {"key1": value1, "key2": value2}"

Real example from the logs, in the middle, you see the textPayload:

The picture as text:

{
insertId: "000000-1b431ffd-e42d-4f83-xyz"
labels: {1}
logName: "projects/MY_PROJECT/logs/cloudfunctions.googleapis.com%2Fcloud-functions"
receiveTimestamp: "2022-02-08T15:43:41.808217166Z"
resource: {2}
textPayload: "2022-02-08 15:43:32,460 [INFO]: {"json_metadata": {"countrows": 736203, "countcolumns": 6, "size": 48261360, "gcs_stamp": "2022-02-08 15:43:32.451000+00:00", "python_stamp": "2022-02-08 15:43:31.055538"}}"
timestamp: "2022-02-08T15:43:32.460Z"
trace: "projects/MY_PROJECT/traces/dd97759176248586a3d3xyz"
}

First tries

Reading from https://cloud.google.com/logging/docs/structured-logging:

In Cloud Logging, structured logs refer to log entries that use the jsonPayload field to add structure to their payloads. Structured logging applies to user-written logs.

I tried to get this "structured logging" following Writing structured logs by

logging.info(json.dumps(json_for_gcp_lbm))

but to no avail.

Further in the links: there is a built-in Logging agent from GCP that uses fluentd as to About the Logging agent seems to be available only for Google Kubernetes Engine or the App Engine, not in a Google Cloud Function:

If you're using Google Kubernetes Engine or the App Engine flexible environment, you can write structured logs as JSON objects serialized on a single line to stdout or stderr. The Logging agent then sends the structured logs to Cloud Logging as the jsonPayload of the LogEntry structure.

How can I get the jsonPayload in this output?

I get the warning

OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k

in the logs of the "LOGS" tab of the Cloud Function.

There is already a Q/A on this on Stack Overflow at AppEngine warning - OpenBLAS WARNING - could not determine the L2 cache size on this system, but it is not asked for a Google Cloud Function, only for the Google App Engine.

I wonder how I can get rid of this warning in a Google Cloud Function and whether I should care at all?

• It is just a warning anyway and the quite heavy Cloud Function (with a lot of calculations + saving 50 MB csv, 700 MB needed RAM, 1 GB allocated, 540s timeout time) runs through anyway. Doing nothing can be an accepted answer.
• Perhaps I can find the L2 cache size somewhere in the settings, which system is meant here?

I get this error FileNotFoundError: [Errno 2] No such file or directory when I try to write a csv file to the bucket, using a csv writer that loops over batches of data. The full insight into the Cloud Function logs around that error:

File "/workspace/main.py", line 299, in write_to_csv_file with
open(filepath, "w") as outcsv: FileNotFoundError: [Errno 2] No such
file or directory: 'gs://MY_BUCKET/MY_CSV.csv'

Function execution took 52655 ms, finished with status: 'crash' 

OpenBLAS WARNING - could not determine the L2 cache size on this
system, assuming 256k  ```

And this, although this bucket_filepath definitely exists: I can upload an empty dummy file and get its "gsutils URI" (right-click on the three dots at the right side of the file) and the bucket_filepath will look the same: 'gs://MY_BUCKET/MY_CSV.csv'.

I checked saving a dummy pandas dataframe instead using pd.to_csv and it worked with the same bucket_filepath (!).

Therefore, there must be another reason, likely the writer is not accepted, or the with statement that opens the file.

The code that throws the error is as follows. It is with the same code working outside of Google Cloud Function in a normal cron job on a local server. I have added two debug prints around the line that throws the error, the print("Right after opening the file ...") does not show up anymore. The subfunction query_execute_batch() that write_to_csv_file() is calling for each batch is also shown but likely not the problem here since the error happens already at the very start when write-opening the csv file.

requirements.txt (which are then imported as modules):

SQLAlchemy>=1.4.2
google-cloud-storage>=1.16.1
mysqlclient==2.1.0
pandas==1.2.3
fsspec==2021.11.1
gcsfs==2021.11.1
unicodecsv==0.14.1

And from the main.py:

def query_execute_batch(connection):
    """Function for reading data from the query result into batches
    :yield: each result in a loop is a batch of the query result
    """
    results = execute_select_batch(connection, SQL_QUERY)
    print(f"len(results): {len(results)}")
    for result in results:
        yield result

def write_to_csv_file(connection, filepath):
    """Write the data in a loop over batches into a csv.
    This is done in batches since the query from the database is huge.
    :param connection: mysqldb connection to DB
    :param filepath: path to csv file to write data
    returns: metadata on rows and time
    """
    countrows = 0
    print("Right before opening the file ...")    
    with open(filepath, "w") as outcsv:
        print("Right after opening the file ...")        
        writer = csv.DictWriter(
            outcsv,
            fieldnames=FIELDNAMES,
            extrasaction="ignore",
            delimiter="|",
            lineterminator="\n",
        )
        # write header according to fieldnames
        writer.writeheader()

        for batch in query_execute_batch(connection):
            writer.writerows(batch)
            countrows += len(batch)
        datetime_now_save = datetime.now()
    return countrows, datetime_now_save

Mind that for the above script to work, I import gcsfs which makes the bucket read-write-available. Else I would likely need a google cloud storage object like for example:

storage_client = storage.Client()
bucket = storage_client.bucket(BUCKET_NAME)

and then make the file in that bucket with further functions, but that is not the aim here.

In the following, the pd.to_csv code that works, it uses the output of a dummy SQL query SELECT 1 as the input of a dataframe. This can be saved to the same bucket_filepath, of course the reason might not just be pd.to_csv() as such, but also that the dataset is a dummy instead of complex unicode strings from a huge SELECT query. Or there is another reason, I am just guessing.

if records is not None:
    df = pd.DataFrame(records.fetchall())
    df.columns = records.keys()
    df.to_csv(filepath,
        index=False,
    )
    datetime_now_save = datetime.now()
    countrows = df.shape[0]

I would like to use the csv writer to have the chance to write in unicode with the unicodecsv module and the chance to use the batches.

I might be willing to change to batches (loop + append mode or chunksize) in pandas like in Writing large Pandas Dataframes to CSV file in chunks to get rid of this bucket filepath problem, but I would like to rather use the ready code (never touch a running system).

How can I get the saving of that csv done with the csv writer so that it can open a new file in the bucket in write mode = with open(filepath, "w") as outcsv:?

The given function write_to_csv_file() is just a tiny part of the Cloud Function which uses a wide range of functions and cascaded functions. I cannot show the whole reproducible case here and hope that it can be answered by experience or easier examples.

I need something like Cloud Storage for Firebase: download metadata of all files, just not in Angular but in Python and just for a chosen file instead.

The aim is to return this information when the Cloud Function finishes with the return statement or just to log it during the run of the Cloud Function as soon as the file is saved in the Google Storage bucket. With that information at hand, another job can be started after the given timestamp. The pipeline is synchronous.

I have found Q/A's on loading a file or its data into the Cloud Function

• using /tmp directory like at Reading Data From Cloud Storage Via Cloud Functions
• or load the data (and not the file) using storage.Client() or pandas df.read_csv() like at How to load a file from google cloud storage to google cloud function

to extract data stats into the running Cloud Function from the external file.

Since I do not want to save the large file or its data in memory at any time only to get some metadata, I want to download only the metadata from that file that is stored in a bucket in Google Storage, meaning timestamp and size.

How can I fetch only the metadata of a csv file in a Google Cloud Storage bucket to the Google Cloud Function?