Does Prometheus expose any metrics on itself? i.e. how much disk space it's using etc.
I want to start fine tuning our prometheus server, so need to monitor what's currently there. I want to be able to see how much disk space it using.
Prometheus v2.31 installed via apt on Ubuntu Linux 22.04 LTS.
While reading Prometheus Configuration documentation and some sample scrape configurations, I found some kubernetes_sd_configs with role service & role endpoints & role pod
- job_name: kube-state-metrics
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: keep
regex: prometheus;kube-state-metrics
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_component
- honor_labels: true
job_name: prometheus-pushgateway
kubernetes_sd_configs:
- role: service
relabel_configs:
- action: keep
regex: pushgateway
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- job_name: node-exporter
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_container_port_number
target_label: __address__
Looks like all these roles can scrape the metrics from instances,
When shall we use endpoints & when shall we use pod?
EDIT: my real question is, which role (endpoints or pod) shall I use if they both discover the scraping target?
what do you prefer? any examples?
Our setup includes:
Basically, our load could be volatile so we'd like to capture details so currently we are scraping metrics every 10 seconds, and display 1-minute rates in Grafana, with queries like that:
rate(node_network_receive_bytes_total{instance=~'$node',device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[1m])*8
In Grafana we see huge spikes, for network instances with average throughput under 100Mbit/s, spikes exceed hundreds of Gigabits per second which obviously is not exactly technically possible. Same happens for CPU load, CPU wait time, disks iops and other node_exporter metrics as well, generally it looks like that, see the dramatic difference between averages and maximums:
Apparently this happens because Prometheus seems to 'miss' single points of data, and based on how rate works it compares 'last' point with zero to current node_network_receive_bytes_total accumulated since last startup and rockets up the output. If we try to switch to irate the spikes just jump even higher which seems to prove our guess.
Querying our Prometheus collerting server for datapoints in particular time ranges where our rate is having spikes, we do not see any zeroed points, data in 'spiky' time range looks like consecutive increase:
node_network_receive_bytes_total{device="ens8",instance="cassandra-xxxxxxxxx0:9100",job="cassandra-xxxxxxxxx"}
3173659836137 @1585311247.489
3173678570634 @1585311257.49
3173696782823 @1585311267.491
3173715943503 @1585311277.492
3173715937480 @1585311277.493
3173731328095 @1585311287.495
3173743034248 @1585311297.502
3173756482486 @1585311307.497
3173775999916 @1585311317.497
3173796096167 @1585311327.498
3173814354877 @1585311337.499
3173833456218 @1585311347.499
3173852345655 @1585311357.501
Same on graph:
rate query rate(node_network_receive_bytes_total{instance="cassandra-xxxxxxxxx0:9100",device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[1m])*8 is displaying surprisingly different picture in the same time range:
While Prometheus documentation states that it should extrapolate missing datapoints, as well as certain issues with rate/irate are widely recognized, for now we're pretty confused with the above.
Our biggest problem is that the spikes make both visualizing and, more importantly, setting up the limits/alerts impossible.
For now we're only certain that Grafana is out of question and the issue lies within our Prometheus, and the question goes like - have you bumped into something similar, maybe? If yes, how do you deal with that?
If not, maybe you could suggest some further dignostic approach?
Anyway, thank you everybody at least for reading until this point.
I'm trying to write a general rule to fire alert when a discovered target goes missing. In particular kubernetes pods annotated for scraping and auto-discovered using kubernetes_sd_configs.
Expressions of the form: absent(up{job="kubernetes-pods"}==1) do not return any additional labels which were available as part of the up time series. If a pod is deleted (say by mistake), it disappears as a target from prometheus. An alert based on absent() is fired, but I have no information about what pod has gone missing.
I think the same happens for auto-discovered kubernetes services. If it's deleted by mistake, it just disappears as a monitored target. I'm not sure if the behavior is the same for target_groups (https://prometheus.io/blog/2015/06/01/advanced-service-discovery/) with ip range - that is if the physical node is turned off the metrics just stop and up == 0 is not available.
What is the correct way to detect when an auto-discovered target is gone in a general way? Or do I need to hard code rules for each service/node/pod explicitly, even though it was auto discovered?
I've setup Prometheus and node_exporter. All example graphs I've found utilize node_cpu but I don't have this metric present. The closest I've come is node_cpu_seconds_total. Am I missing something or is my version of Prometheus incompatible with these graphs?