I'm trying to install loki so that I can read logs on Grafana, but I keep receiving Data source connected, but no labels received. Verify that Loki and Promtail is configured properly.
(nothing from the logs).
I'm using docker images for each service (grafana, loki, promtail) on a Raspberry Pi 4 8Gb.
Each service are working properly. I can connect to Grafana (localhost:3000
) and see node_exporter data. I can connect to loki itself and see the metrics (localhost:3100/metrics
), and I can also connect to promtail (localhost:9080/targets
).
On Grafana, I tried pointing loki to 10.88.0.1:3100/metrics
(since Grafana is a docker container, I have to use 10.88.0.1
to connect to the host) but I get a new error Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details
Here is what I get from the logs for that error:
logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204508553Z level=error msg="Failed to call resource" error="404 page not found\n" traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204758623Z level=error msg="Request Completed" method=GET path=/api/datasources/2/resources/labels status=500 remote_addr=192.168.1.20 time_ms=7 duration=7.563973ms size=51 referer=http://raspberry.local:3000/datasources/edit/sgEZvn14k handler=/api/datasources/:id/resources/*
It's weird because without the /metrics
Grafana does find loki.
I'm using ansible to provision my rasp :
- name: Grafana - Run container using podman
containers.podman.podman_container:
name: grafana
image: grafana/grafana-oss:latest
state: started
recreate: true
restart_policy: on-failure
ports: "3000:3000"
memory: "2048m"
net:
- host
volume:
- "grafana-storage:/var/lib/grafana"
- name: Loki - Run container using podman
containers.podman.podman_container:
name: loki
image: grafana/loki:latest
state: started
recreate: true
restart_policy: on-failure
ports: "3100:3100"
memory: "2048m"
net:
- host
volume:
- "loki-storage:/loki"
- name: Promtail - Run container using podman
containers.podman.podman_container:
name: promtail
image: grafana/promtail:latest
state: started
recreate: true
restart_policy: on-failure
ports:
- "9080:9080"
- "1514:1514"
memory: "2048m"
net:
- host
volume:
- "{{ lokiData }}/promtail.yml:/etc/promtail/promtail-config.yml"
command: -config.file=/etc/promtail/promtail-config.yml
My promtail config:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://10.88.0.1:3100/loki/api/v1/push
scrape_configs:
- job_name: syslog
syslog:
listen_address: 0.0.0.0:1514
labels:
job: "syslog"
relabel_configs:
- source_labels: ['__syslog_message_hostname']
target_label: 'host'
my loki config:
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
I really don't know what to do now, I'm new to Grafana, so maybe I'm missing something. I googled the error, but I found nothing interesting (mainly config mistakes).
I forgot to add the journal/syslogs volumes. In fact, everything was working, but promtail had no data to send. Make sure to read the
positions.yaml
file (located by promtail's config).