Prometheus Breadcrumbs 2

Prometheus breadcrumbs 2

Last week I explained the types of metrics Prometheus can handle. This week I'll just speak briefly about labels.

What are labels?

Every time Prometheus grabs a metric, such a metric has labels associated with it so we can tag and identify the metric. Imagine the metric is memory used, how do you know if a particular metric is from your service X or Y? That's what labels are mainly used for. However, we have also target labels, which are labels to identify potential targets to get metrics from.

Grabbing around

Evil is in the details. Did you notice I said Prometheus "grabs" a metric? Why is that? Well, Prometheus is actually doing two things. One is to store metrics, so it's a database. But also, it scrapes the metrics from targets.

When you configure Prometheus, you can tell it what to scrape e.g. you can specify to scrape services with only certain labels (scrape metrics from targets with the label "service=servicex") or you can use service discovery e.g. by using Kubernetes API. As you can imagine, when you are scraping metrics it happens the same as with web crawling, you can get a lot of useless stuff a.k.a junk. Here is when relabelling enters into action:

• by using relabel_configs field: You can tell Prometheus to relabel before the scrape. So Prometheus sees a target, applies relabelling on target labels and then Prometheus scrapes and stores the metrics with the precise labels IF it matches certain criteria.
• by using metric_relabel_configs field: Prometheus scrapes a metric and then relabelling happens after the scrape. So the metric is grabbed, then relabel happens to the metric and then finally stored.

What relabel is used for?

So we have two types of relabelling, but in reality it's because we also have two types of labels. We have target labels and target relabelling, which is used to be selective with what to scrape, so you avoid Prometheus scraping stuff you might not need later or junk, this is relabel_configs field. The other one is metrics labels, which uses metric_relabel_configs field. It's used to organize information/metrics in a better way by adding useful labels or by discarding labels that are not useful for you, for example, you could remove user IP label from some metrics.

And that's all for this week's breadcrumbs. As always my goal with these breadcrumbs is to share my knowledge with easy and short posts. If you want to stay tuned with Backend and DevOps stuff, feel free to follow me and if you feel extra generous please give some love to the post by commenting on it or reacting.