0

I'm following the guide https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/ to install loki in my k8s cluster:

deploymentMode: SingleBinary
loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: 'filesystem'
  schemaConfig:
    configs:
    - from: "2024-01-01"
      store: tsdb
      index:
        prefix: loki_index_
        period: 24h
      object_store: filesystem # we're storing on filesystem so there's no real persistence here.
      schema: v13
singleBinary:
  replicas: 1
read:
  replicas: 0
backend:
  replicas: 0
write:
  replicas: 0

The installation works "fine" but if I execute kubectl describe pod loki-chunks-cache-0. I see the following error in the loki-chunks-cache-0:

Warning FailedScheduling 59s (x297 over 24h) default-scheduler 0/2 nodes are available: 1 Too many pods, 2 Insufficient memory. preemption: 0/2 nodes are

And the following resource requests:

memcached:
    Image:      memcached:1.6.23-alpine
    Port:       11211/TCP
    Host Port:  0/TCP
    Args:
      -m 8192
      --extended=modern,track_sizes
      -I 5m
      -c 16384
      -v
      -u 11211
    Limits:
      memory:  9830Mi
    Requests:
      cpu:        500m
      memory:     9830Mi

I understand that the problem is that I've not enough memory in my k8s cluster. But is there anyway that I can indicate in my yaml the limits/requests for this cache to reduce these needs or to disable it entirely?

My lab has a few Docker containers as follows:

My goal is to collect Caddy logs and visualize them in Grafana.

Scenario: Fluent-bit tails the logs and sends them to Fluentd. Then Fluentd pushes the logs to Loki. My aim is to use Fluentd as the central log collector.

The problem is the parsing of those logs Grafana-side.

The Caddy logs are in (nested) JSON format. Sample:

{"level":"info","ts":1712949034.535184,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"39664","client_ip":"172.18.0.1","proto":"HTTP/1.1","method":"POST","host":"grafana.darknet.com","uri":"/api/short-urls","headers":{"Content-Length":["580"],"Origin":["http://grafana.darknet.com"],"Content-Type":["application/json"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"],"Accept":["application/json, text/plain, */*"],"X-Grafana-Org-Id":["1"],"Connection":["keep-alive"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate"],"Referer":["http://grafana.darknet.com/explore?schemaVersion=1&panes=%7B%22Efb%22:%7B%22datasource%22:%22f779c221-7bd2-468d-9f9c-96e069b869f8%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bjob%3D%5C%22caddy.log.loki%5C%22%7D%20%7C%20json%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22f779c221-7bd2-468d-9f9c-96e069b869f8%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-1m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1"],"X-Grafana-Device-Id":["f343e938e74b3a57997faff69d24de8a"],"Cookie":[]}},"bytes_read":580,"user_id":"","duration":0.011267887,"size":72,"status":200,"resp_headers":{"X-Xss-Protection":["1; mode=block"],"Date":["Fri, 12 Apr 2024 19:10:34 GMT"],"Content-Length":["72"],"Server":["Caddy"],"Cache-Control":["no-store"],"Content-Type":["application/json"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["deny"]}}

I have tried two different configurations so far:

1. Have Fluent-bit send the logs to Fluentd, then Fluentd forwards the logs to Loki (tagged as caddy.log)
   Schema: Cady --> Fluent-bit --> Fluentd --> Loki
   

2. Have Fluent-bit send the logs straight to Loki (tagged as caddy.log.loki)
   Schema: Cady --> Fluent-bit --> Loki
   

Here I have the following Fluent-bit config to send logs to both Loki and Fluentd at the same time, with different tags:

[INPUT]
    Name tail
    Path /var/log/caddy/*.log
    Parser json
    Tag caddy.log
    Path_Key log_filename

# send logs to Fluentd
[OUTPUT]
    Name forward
    Host fluentd
    Port 24224
    Match caddy.*

# send logs straight to Loki
[OUTPUT]
    name                   loki
    match                  caddy.*
    host                   loki
    port                   3100
    labels                 job=caddy.log.loki

Fluentd config:

<source>
  @type  forward
</source>

<match caddy.*>
  @type loki
  url "http://loki:3100"
  extra_labels {"job": "caddy.log"}
  <buffer>
    flush_interval 5s
    flush_at_shutdown true
  </buffer>
</match>

Then in Grafana I can browse the logs and I have the two labels available in the Explore window.

If I choose the tag caddy.log.loki the logs are displayed in plain JSON as shown below. With this expression I can parse them: {job="caddy.log.loki"} | json. Some of the nested JSON is extracted eg: request_client_ip but not all of it, for example request.headers is missing but I can live with that.

If I choose the tag caddy.log then the logs are displayed in a "mixed" format:

It appears that some transformation took place but I am not sure where. I can use logfmt to parse the lines. But I am still left with some unparsed fields (request, resp_headers) as shown below:

Questions:

• why is that the logs are not rendered in plain JSON anymore if I add the Fluentd step?
• what would be the best way to ship and parse nested JSON logs in Loki/Grafana with Fluentd?

In a setup with promtail, loki and grafana, no data shows up in grafana explore. All three parts are running on the same machine.

Question is: where to look for an error in the setup?

Promtail

promtail.yaml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/db/promtail/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/**/*log

This seems to be fine. Testing:

promtail --config.file=promtail.yaml --log.level=info -print-config-stderr

---
# Loki Config
# (version=, branch=, revision=unknown)
global:
  file_watch_config:
    min_poll_frequency: 250ms
    max_poll_frequency: 250ms
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 9080
[...]
level=info ts=2023-12-13T16:00:24.711575585Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=169d8cc02367d196d7ba24694ea90c5d
level=info ts=2023-12-13T16:00:24.712927375Z caller=server.go:322 http=[::]:9080 grpc=[::]:32729 msg="server listening on addresses"
level=info ts=2023-12-13T16:00:24.713264874Z caller=main.go:174 msg="Starting Promtail" version="(version=, branch=, revision=unknown)"
level=warn ts=2023-12-13T16:00:24.713474024Z caller=promtail.go:263 msg="enable watchConfig"
level=info ts=2023-12-13T16:00:29.714075081Z caller=filetargetmanager.go:361 msg="Adding target" key="/var/log/**/*log:{job=\"varlogs\"}"
level=info ts=2023-12-13T16:00:29.716662721Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/loki
level=info ts=2023-12-13T16:00:29.716703277Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/promtail
level=info ts=2023-12-13T16:00:29.716812088Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/gitea
level=info ts=2023-12-13T16:00:29.716857702Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/grafana
level=info ts=2023-12-13T16:00:29.717132151Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/redis
level=info ts=2023-12-13T16:00:29.717201082Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/caddy
level=info ts=2023-12-13T16:00:29.717280021Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log
ts=2023-12-13T16:00:29.720424446Z caller=log.go:168 level=info msg="Seeked /var/log/caddy/caddy.log - &{Offset:606936482 Whence:0}"
level=info ts=2023-12-13T16:00:29.720534804Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/caddy/caddy.log
ts=2023-12-13T16:00:29.720631282Z caller=log.go:168 level=info msg="Seeked /var/log/gitea/gitea.log - &{Offset:8194 Whence:0}"
level=info ts=2023-12-13T16:00:29.72066379Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/gitea/gitea.log
ts=2023-12-13T16:00:29.720728343Z caller=log.go:168 level=info msg="Seeked /var/log/grafana/grafana.log - &{Offset:38901 Whence:0}"
level=info ts=2023-12-13T16:00:29.720756759Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/grafana/grafana.log
[...]

Loki

loki.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/db/loki
  storage:
    filesystem:
      chunks_directory: /var/db/loki/chunks
      rules_directory: /var/db/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

The loki application / service is up and running. Anything else I could be testing here?

Grafana

Data source for loki created with:

• Type: Loki
• URL: http://127.0.0.1:3100 (everything else: left at default settings)

Looking at grafana's /explore only leads to no data being shown, even without any filter criteria. Pressing Run query just doesn't seem to have any effect.

The grafana log looks good to me:

tail /var/log/grafana/grafana.log

logger=grafana.update.checker t=2023-12-13T16:31:16.759743533+01:00 level=info msg="Update check succeeded" duration=486.885735ms
logger=cleanup t=2023-12-13T16:41:16.24610575+01:00 level=info msg="Completed cleanup jobs" duration=13.483743ms
logger=plugins.update.checker t=2023-12-13T16:41:16.617294501+01:00 level=info msg="Update check succeeded" duration=334.848905ms
logger=grafana.update.checker t=2023-12-13T16:41:16.657803737+01:00 level=info msg="Update check succeeded" duration=387.307337ms
logger=cleanup t=2023-12-13T16:51:16.240282272+01:00 level=info msg="Completed cleanup jobs" duration=7.750284ms
logger=grafana.update.checker t=2023-12-13T16:51:16.292110778+01:00 level=info msg="Update check succeeded" duration=21.261137ms
logger=plugins.update.checker t=2023-12-13T16:51:16.313163122+01:00 level=info msg="Update check succeeded" duration=31.295959ms
logger=cleanup t=2023-12-13T17:01:16.220217724+01:00 level=info msg="Completed cleanup jobs" duration=4.195714ms
logger=plugins.update.checker t=2023-12-13T17:01:16.312587838+01:00 level=info msg="Update check succeeded" duration=30.754461ms
logger=grafana.update.checker t=2023-12-13T17:01:16.333881656+01:00 level=info msg="Update check succeeded" duration=63.366615ms

Pointers on what might be missing here are appreciated.

I'm trying to install loki so that I can read logs on Grafana, but I keep receiving Data source connected, but no labels received. Verify that Loki and Promtail is configured properly. (nothing from the logs).

I'm using docker images for each service (grafana, loki, promtail) on a Raspberry Pi 4 8Gb.

Each service are working properly. I can connect to Grafana (localhost:3000) and see node_exporter data. I can connect to loki itself and see the metrics (localhost:3100/metrics), and I can also connect to promtail (localhost:9080/targets).

On Grafana, I tried pointing loki to 10.88.0.1:3100/metrics (since Grafana is a docker container, I have to use 10.88.0.1 to connect to the host) but I get a new error Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details

Here is what I get from the logs for that error:

logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204508553Z level=error msg="Failed to call resource" error="404 page not found\n" traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204758623Z level=error msg="Request Completed" method=GET path=/api/datasources/2/resources/labels status=500 remote_addr=192.168.1.20 time_ms=7 duration=7.563973ms size=51 referer=http://raspberry.local:3000/datasources/edit/sgEZvn14k handler=/api/datasources/:id/resources/*

It's weird because without the /metrics Grafana does find loki.

I'm using ansible to provision my rasp :

- name: Grafana - Run container using podman
  containers.podman.podman_container:
      name: grafana
      image: grafana/grafana-oss:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports: "3000:3000"
      memory: "2048m"
      net:
          - host
      volume:
          - "grafana-storage:/var/lib/grafana"

- name: Loki - Run container using podman
  containers.podman.podman_container:
      name: loki
      image: grafana/loki:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports: "3100:3100"
      memory: "2048m"
      net:
          - host
      volume:
          - "loki-storage:/loki"

- name: Promtail - Run container using podman
  containers.podman.podman_container:
      name: promtail
      image: grafana/promtail:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports:
          - "9080:9080"
          - "1514:1514"
      memory: "2048m"
      net:
          - host
      volume:
          - "{{ lokiData }}/promtail.yml:/etc/promtail/promtail-config.yml"
      command: -config.file=/etc/promtail/promtail-config.yml

My promtail config:

server:
    http_listen_port: 9080
    grpc_listen_port: 0

positions:
    filename: /tmp/positions.yaml

clients:
    - url: http://10.88.0.1:3100/loki/api/v1/push

scrape_configs:
    - job_name: syslog
      syslog:
        listen_address: 0.0.0.0:1514
        labels:
          job: "syslog"
      relabel_configs:
        - source_labels: ['__syslog_message_hostname']
          target_label: 'host'

my loki config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

I really don't know what to do now, I'm new to Grafana, so maybe I'm missing something. I googled the error, but I found nothing interesting (mainly config mistakes).

Since my last reboot, I am seeing the following every 1-2 minutes:

Aug 02 13:53:00 monitor systemd[1]: influxdb.service: start operation timed out. Terminating.
Aug 02 13:53:00 monitor systemd[1]: influxdb.service: Failed with result 'timeout'.
Aug 02 13:53:00 monitor systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
Aug 02 13:53:00 monitor systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 4.
Aug 02 13:53:00 monitor systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Aug 02 13:53:00 monitor systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Aug 02 13:53:00 monitor influxd-systemd-start.sh[3539]: Merging with configuration at: /etc/influxdb/influxdb.conf

on 29/07/2021 influx was updated from 1.8.6-1 to 1.8.7-1. The OS is Ubuntu 20.04 server. The first reboot after this is when the issues started.
Initially there was a permissions issue with /usr/lib/influxdb/scripts/influxd-systemd-start.sh, which prevented it starting. I changed the perms to 0755 and it started, but keeps restarting. It seems that it is accepting connections and data between the restarts, as telegraf is still populating the database, and Grafana is able to display the stats, so long as it doesn't coincide with the restart.

I am also seeing the message

influxd-systemd-start.sh[12171]: [tcp] 2021/08/02 14:21:40 tcp.Mux: Listener at 127.0.0.1:8088 failed failed to accept a connection, closing all listeners

It is listening on those ports

root@monitor$ ss -ilpn | grep influx
tcp     LISTEN   0        4096                                        127.0.0.1:8088                                              0.0.0.0:*                      users:(("influxd",pid=15115,fd=3))
tcp     LISTEN   0        4096                                                *:8086                                                    *:*                      users:(("influxd",pid=15115,fd=32))

As far as I am aware there have been no config changed. There is no firewall rules active.

Anybody have any idea why it started misbehaving?