Parse nested JSON logs in Grafana Loki (forwarded with Fluentd)

My lab has a few Docker containers as follows:

My goal is to collect Caddy logs and visualize them in Grafana.

Scenario: Fluent-bit tails the logs and sends them to Fluentd. Then Fluentd pushes the logs to Loki. My aim is to use Fluentd as the central log collector.

The problem is the parsing of those logs Grafana-side.

The Caddy logs are in (nested) JSON format. Sample:

{"level":"info","ts":1712949034.535184,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"39664","client_ip":"172.18.0.1","proto":"HTTP/1.1","method":"POST","host":"grafana.darknet.com","uri":"/api/short-urls","headers":{"Content-Length":["580"],"Origin":["http://grafana.darknet.com"],"Content-Type":["application/json"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"],"Accept":["application/json, text/plain, */*"],"X-Grafana-Org-Id":["1"],"Connection":["keep-alive"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate"],"Referer":["http://grafana.darknet.com/explore?schemaVersion=1&panes=%7B%22Efb%22:%7B%22datasource%22:%22f779c221-7bd2-468d-9f9c-96e069b869f8%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bjob%3D%5C%22caddy.log.loki%5C%22%7D%20%7C%20json%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22f779c221-7bd2-468d-9f9c-96e069b869f8%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-1m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1"],"X-Grafana-Device-Id":["f343e938e74b3a57997faff69d24de8a"],"Cookie":[]}},"bytes_read":580,"user_id":"","duration":0.011267887,"size":72,"status":200,"resp_headers":{"X-Xss-Protection":["1; mode=block"],"Date":["Fri, 12 Apr 2024 19:10:34 GMT"],"Content-Length":["72"],"Server":["Caddy"],"Cache-Control":["no-store"],"Content-Type":["application/json"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["deny"]}}

I have tried two different configurations so far:

1. Have Fluent-bit send the logs to Fluentd, then Fluentd forwards the logs to Loki (tagged as caddy.log)
   Schema: Cady --> Fluent-bit --> Fluentd --> Loki
   

2. Have Fluent-bit send the logs straight to Loki (tagged as caddy.log.loki)
   Schema: Cady --> Fluent-bit --> Loki
   

Here I have the following Fluent-bit config to send logs to both Loki and Fluentd at the same time, with different tags:

[INPUT]
    Name tail
    Path /var/log/caddy/*.log
    Parser json
    Tag caddy.log
    Path_Key log_filename

# send logs to Fluentd
[OUTPUT]
    Name forward
    Host fluentd
    Port 24224
    Match caddy.*

# send logs straight to Loki
[OUTPUT]
    name                   loki
    match                  caddy.*
    host                   loki
    port                   3100
    labels                 job=caddy.log.loki

Fluentd config:

<source>
  @type  forward
</source>

<match caddy.*>
  @type loki
  url "http://loki:3100"
  extra_labels {"job": "caddy.log"}
  <buffer>
    flush_interval 5s
    flush_at_shutdown true
  </buffer>
</match>

Then in Grafana I can browse the logs and I have the two labels available in the Explore window.

If I choose the tag caddy.log.loki the logs are displayed in plain JSON as shown below. With this expression I can parse them: {job="caddy.log.loki"} | json. Some of the nested JSON is extracted eg: request_client_ip but not all of it, for example request.headers is missing but I can live with that.

If I choose the tag caddy.log then the logs are displayed in a "mixed" format:

It appears that some transformation took place but I am not sure where. I can use logfmt to parse the lines. But I am still left with some unparsed fields (request, resp_headers) as shown below:

Questions:

• why is that the logs are not rendered in plain JSON anymore if I add the Fluentd step?
• what would be the best way to ship and parse nested JSON logs in Loki/Grafana with Fluentd?