In a setup with promtail, loki and grafana, no data shows up in grafana explore. All three parts are running on the same machine.

Question is: where to look for an error in the setup?

Promtail

promtail.yaml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/db/promtail/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/**/*log

This seems to be fine. Testing:

promtail --config.file=promtail.yaml --log.level=info -print-config-stderr

---
# Loki Config
# (version=, branch=, revision=unknown)
global:
  file_watch_config:
    min_poll_frequency: 250ms
    max_poll_frequency: 250ms
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 9080
[...]
level=info ts=2023-12-13T16:00:24.711575585Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=169d8cc02367d196d7ba24694ea90c5d
level=info ts=2023-12-13T16:00:24.712927375Z caller=server.go:322 http=[::]:9080 grpc=[::]:32729 msg="server listening on addresses"
level=info ts=2023-12-13T16:00:24.713264874Z caller=main.go:174 msg="Starting Promtail" version="(version=, branch=, revision=unknown)"
level=warn ts=2023-12-13T16:00:24.713474024Z caller=promtail.go:263 msg="enable watchConfig"
level=info ts=2023-12-13T16:00:29.714075081Z caller=filetargetmanager.go:361 msg="Adding target" key="/var/log/**/*log:{job=\"varlogs\"}"
level=info ts=2023-12-13T16:00:29.716662721Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/loki
level=info ts=2023-12-13T16:00:29.716703277Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/promtail
level=info ts=2023-12-13T16:00:29.716812088Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/gitea
level=info ts=2023-12-13T16:00:29.716857702Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/grafana
level=info ts=2023-12-13T16:00:29.717132151Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/redis
level=info ts=2023-12-13T16:00:29.717201082Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/caddy
level=info ts=2023-12-13T16:00:29.717280021Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log
ts=2023-12-13T16:00:29.720424446Z caller=log.go:168 level=info msg="Seeked /var/log/caddy/caddy.log - &{Offset:606936482 Whence:0}"
level=info ts=2023-12-13T16:00:29.720534804Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/caddy/caddy.log
ts=2023-12-13T16:00:29.720631282Z caller=log.go:168 level=info msg="Seeked /var/log/gitea/gitea.log - &{Offset:8194 Whence:0}"
level=info ts=2023-12-13T16:00:29.72066379Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/gitea/gitea.log
ts=2023-12-13T16:00:29.720728343Z caller=log.go:168 level=info msg="Seeked /var/log/grafana/grafana.log - &{Offset:38901 Whence:0}"
level=info ts=2023-12-13T16:00:29.720756759Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/grafana/grafana.log
[...]

Loki

loki.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/db/loki
  storage:
    filesystem:
      chunks_directory: /var/db/loki/chunks
      rules_directory: /var/db/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

The loki application / service is up and running. Anything else I could be testing here?

Grafana

Data source for loki created with:

• Type: Loki
• URL: http://127.0.0.1:3100 (everything else: left at default settings)

Looking at grafana's /explore only leads to no data being shown, even without any filter criteria. Pressing Run query just doesn't seem to have any effect.

The grafana log looks good to me:

tail /var/log/grafana/grafana.log

logger=grafana.update.checker t=2023-12-13T16:31:16.759743533+01:00 level=info msg="Update check succeeded" duration=486.885735ms
logger=cleanup t=2023-12-13T16:41:16.24610575+01:00 level=info msg="Completed cleanup jobs" duration=13.483743ms
logger=plugins.update.checker t=2023-12-13T16:41:16.617294501+01:00 level=info msg="Update check succeeded" duration=334.848905ms
logger=grafana.update.checker t=2023-12-13T16:41:16.657803737+01:00 level=info msg="Update check succeeded" duration=387.307337ms
logger=cleanup t=2023-12-13T16:51:16.240282272+01:00 level=info msg="Completed cleanup jobs" duration=7.750284ms
logger=grafana.update.checker t=2023-12-13T16:51:16.292110778+01:00 level=info msg="Update check succeeded" duration=21.261137ms
logger=plugins.update.checker t=2023-12-13T16:51:16.313163122+01:00 level=info msg="Update check succeeded" duration=31.295959ms
logger=cleanup t=2023-12-13T17:01:16.220217724+01:00 level=info msg="Completed cleanup jobs" duration=4.195714ms
logger=plugins.update.checker t=2023-12-13T17:01:16.312587838+01:00 level=info msg="Update check succeeded" duration=30.754461ms
logger=grafana.update.checker t=2023-12-13T17:01:16.333881656+01:00 level=info msg="Update check succeeded" duration=63.366615ms

Pointers on what might be missing here are appreciated.