Home / user-1004743

tholeb's questions

Martin Hope
tholeb
Asked: 2023-02-13 05:31:02 +0800 CST
  • 5

I'm trying to install loki so that I can read logs on Grafana, but I keep receiving Data source connected, but no labels received. Verify that Loki and Promtail is configured properly. (nothing from the logs).

I'm using docker images for each service (grafana, loki, promtail) on a Raspberry Pi 4 8Gb.

Each service are working properly. I can connect to Grafana (localhost:3000) and see node_exporter data. I can connect to loki itself and see the metrics (localhost:3100/metrics), and I can also connect to promtail (localhost:9080/targets).

On Grafana, I tried pointing loki to 10.88.0.1:3100/metrics (since Grafana is a docker container, I have to use 10.88.0.1 to connect to the host) but I get a new error Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details

Here is what I get from the logs for that error:

logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204508553Z level=error msg="Failed to call resource" error="404 page not found\n" traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204758623Z level=error msg="Request Completed" method=GET path=/api/datasources/2/resources/labels status=500 remote_addr=192.168.1.20 time_ms=7 duration=7.563973ms size=51 referer=http://raspberry.local:3000/datasources/edit/sgEZvn14k handler=/api/datasources/:id/resources/*

It's weird because without the /metrics Grafana does find loki.

I'm using ansible to provision my rasp :

- name: Grafana - Run container using podman
  containers.podman.podman_container:
      name: grafana
      image: grafana/grafana-oss:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports: "3000:3000"
      memory: "2048m"
      net:
          - host
      volume:
          - "grafana-storage:/var/lib/grafana"

- name: Loki - Run container using podman
  containers.podman.podman_container:
      name: loki
      image: grafana/loki:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports: "3100:3100"
      memory: "2048m"
      net:
          - host
      volume:
          - "loki-storage:/loki"

- name: Promtail - Run container using podman
  containers.podman.podman_container:
      name: promtail
      image: grafana/promtail:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports:
          - "9080:9080"
          - "1514:1514"
      memory: "2048m"
      net:
          - host
      volume:
          - "{{ lokiData }}/promtail.yml:/etc/promtail/promtail-config.yml"
      command: -config.file=/etc/promtail/promtail-config.yml

My promtail config:

server:
    http_listen_port: 9080
    grpc_listen_port: 0

positions:
    filename: /tmp/positions.yaml

clients:
    - url: http://10.88.0.1:3100/loki/api/v1/push

scrape_configs:
    - job_name: syslog
      syslog:
        listen_address: 0.0.0.0:1514
        labels:
          job: "syslog"
      relabel_configs:
        - source_labels: ['__syslog_message_hostname']
          target_label: 'host'

my loki config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

I really don't know what to do now, I'm new to Grafana, so maybe I'm missing something. I googled the error, but I found nothing interesting (mainly config mistakes).

grafana