In a setup with promtail, loki and grafana, no data shows up in grafana explore. All three parts are running on the same machine.

Question is: where to look for an error in the setup?

Promtail

promtail.yaml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/db/promtail/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/**/*log

This seems to be fine. Testing:

promtail --config.file=promtail.yaml --log.level=info -print-config-stderr

---
# Loki Config
# (version=, branch=, revision=unknown)
global:
  file_watch_config:
    min_poll_frequency: 250ms
    max_poll_frequency: 250ms
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 9080
[...]
level=info ts=2023-12-13T16:00:24.711575585Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=169d8cc02367d196d7ba24694ea90c5d
level=info ts=2023-12-13T16:00:24.712927375Z caller=server.go:322 http=[::]:9080 grpc=[::]:32729 msg="server listening on addresses"
level=info ts=2023-12-13T16:00:24.713264874Z caller=main.go:174 msg="Starting Promtail" version="(version=, branch=, revision=unknown)"
level=warn ts=2023-12-13T16:00:24.713474024Z caller=promtail.go:263 msg="enable watchConfig"
level=info ts=2023-12-13T16:00:29.714075081Z caller=filetargetmanager.go:361 msg="Adding target" key="/var/log/**/*log:{job=\"varlogs\"}"
level=info ts=2023-12-13T16:00:29.716662721Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/loki
level=info ts=2023-12-13T16:00:29.716703277Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/promtail
level=info ts=2023-12-13T16:00:29.716812088Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/gitea
level=info ts=2023-12-13T16:00:29.716857702Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/grafana
level=info ts=2023-12-13T16:00:29.717132151Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/redis
level=info ts=2023-12-13T16:00:29.717201082Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/caddy
level=info ts=2023-12-13T16:00:29.717280021Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log
ts=2023-12-13T16:00:29.720424446Z caller=log.go:168 level=info msg="Seeked /var/log/caddy/caddy.log - &{Offset:606936482 Whence:0}"
level=info ts=2023-12-13T16:00:29.720534804Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/caddy/caddy.log
ts=2023-12-13T16:00:29.720631282Z caller=log.go:168 level=info msg="Seeked /var/log/gitea/gitea.log - &{Offset:8194 Whence:0}"
level=info ts=2023-12-13T16:00:29.72066379Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/gitea/gitea.log
ts=2023-12-13T16:00:29.720728343Z caller=log.go:168 level=info msg="Seeked /var/log/grafana/grafana.log - &{Offset:38901 Whence:0}"
level=info ts=2023-12-13T16:00:29.720756759Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/grafana/grafana.log
[...]

Loki

loki.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/db/loki
  storage:
    filesystem:
      chunks_directory: /var/db/loki/chunks
      rules_directory: /var/db/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

The loki application / service is up and running. Anything else I could be testing here?

Grafana

Data source for loki created with:

• Type: Loki
• URL: http://127.0.0.1:3100 (everything else: left at default settings)

Looking at grafana's /explore only leads to no data being shown, even without any filter criteria. Pressing Run query just doesn't seem to have any effect.

The grafana log looks good to me:

tail /var/log/grafana/grafana.log

logger=grafana.update.checker t=2023-12-13T16:31:16.759743533+01:00 level=info msg="Update check succeeded" duration=486.885735ms
logger=cleanup t=2023-12-13T16:41:16.24610575+01:00 level=info msg="Completed cleanup jobs" duration=13.483743ms
logger=plugins.update.checker t=2023-12-13T16:41:16.617294501+01:00 level=info msg="Update check succeeded" duration=334.848905ms
logger=grafana.update.checker t=2023-12-13T16:41:16.657803737+01:00 level=info msg="Update check succeeded" duration=387.307337ms
logger=cleanup t=2023-12-13T16:51:16.240282272+01:00 level=info msg="Completed cleanup jobs" duration=7.750284ms
logger=grafana.update.checker t=2023-12-13T16:51:16.292110778+01:00 level=info msg="Update check succeeded" duration=21.261137ms
logger=plugins.update.checker t=2023-12-13T16:51:16.313163122+01:00 level=info msg="Update check succeeded" duration=31.295959ms
logger=cleanup t=2023-12-13T17:01:16.220217724+01:00 level=info msg="Completed cleanup jobs" duration=4.195714ms
logger=plugins.update.checker t=2023-12-13T17:01:16.312587838+01:00 level=info msg="Update check succeeded" duration=30.754461ms
logger=grafana.update.checker t=2023-12-13T17:01:16.333881656+01:00 level=info msg="Update check succeeded" duration=63.366615ms

Pointers on what might be missing here are appreciated.

Caddy v2.2.1 doesn't seem to respect the port of a given address in this Caddyfile:

myfinedomain.com:9999 {
    respond "Nothing to see here. Srsly."
}

Starting up caddy simply results in the following error (taken from the log):

{"level":"info","ts":1614777947.500496,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1614777947.5004952,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002d2380"}
run: loading initial config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: address already in use
start: caddy process exited with error: exit status 1
{"level":"info","ts":1614778157.9187949,"msg":"using provided configuration","config_file":"/usr/local/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1614778157.9210048,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
{"level":"info","ts":1614778157.9213488,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1614778157.92195,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002be3f0"}
run: loading initial config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: address already in use
start: caddy process exited with error: exit status 1

Thus, caddy seems to try binding to port 80, which it shouldn't do.

The documentation lists the port as being a valid option for addresses.

Ideas on what might be missing here are appreciated.

When I have a kubernetes cluster with multiple control nodes and delete one of these, the whole API server does not seem to be available anymore.

In this setup I want to scale down from two to one control node but end up rendering the cluster unusable:

$ kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
master1   Ready    master   5d20h   v1.18.6
worker1   Ready    <none>   5d19h   v1.18.6
master2   Ready    master   19h     v1.18.6
$ kubectl drain master2 --ignore-daemonsets
node/master2 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-hns7p, kube-system/kube-proxy-vk6t7
node/master2 drained
$ kubectl get nodes
NAME      STATUS                     ROLES    AGE     VERSION
master1   Ready                      master   5d20h   v1.18.6
worker1   Ready                      <none>   5d20h   v1.18.6
master2   Ready,SchedulingDisabled   master   19h     v1.18.6
$ kubectl delete node master2
node "master2" deleted
$ kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
master1   Ready    master   5d20h   v1.18.6
worker1   Ready    <none>   5d20h   v1.18.6
$ ssh master2
$ sudo kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W0811 10:24:49.750898    7159 reset.go:99] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get node registration: failed to get corresponding node: nodes "master2" not found
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0811 10:24:51.487912    7159 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
$ exit
$ kubectl get nodes
Error from server: etcdserver: request timed out
$ kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server master1:6443 was refused - did you specify the right host or port?

What's missing here? Or how else is removing a control plane node different from removing a worker node? Pointers are appreciated.