Home / server / Questions / 1025552
In Process
jiangwei
Asked: 2020-07-17 11:24:36 +0800 CST

Prometheus - Use case of service discovery by role endpoints and role pod in Kubernetes

  • 772

While reading Prometheus Configuration documentation and some sample scrape configurations, I found some kubernetes_sd_configs with role service & role endpoints & role pod

- job_name: kube-state-metrics
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - action: keep
    regex: prometheus;kube-state-metrics
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_component

- honor_labels: true
  job_name: prometheus-pushgateway
  kubernetes_sd_configs:
  - role: service
  relabel_configs:
  - action: keep
    regex: pushgateway
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_probe

- job_name: node-exporter
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    source_labels:
    - __address__
    - __meta_kubernetes_pod_container_port_number
    target_label: __address__

Looks like all these roles can scrape the metrics from instances,

When shall we use endpoints & when shall we use pod?

EDIT: my real question is, which role (endpoints or pod) shall I use if they both discover the scraping target?

what do you prefer? any examples?

prometheus kubernetes

1 Answers

  1. Kubernetes SD in Prometheus has a collection of so-called “roles”, which defines how to collect and display metrics. Each Role has its own set of labels which you already know from the official documentation. The ones that interest you are:

    • service: will find and return each Service and its Port

    • pod: will find pods and return its containers as targets to grab metrics from

    • endpoints: will create targets from each Endpoint for each Service found in a cluster

    Some examples can be found below:

    Regarding you additional questions:

    1. role service is used as a black box probing. It won't report how many backends behind this service

    2. role endpoints is used for services exposing web services (have an http endpoint to connect to)

    3. pod is generally used

    Pretty much yes. Note that the pod role discovers all pods and exposes their containers as targets.

    I also strongly recommend you to check out the below sources:

    Please let me know if that helped.

    • 2