Prometheus breadcrumbs 4

Prometheus aggregation

Following my Prometheus breadcrumbs series I'm gonna talk about aggregation and vector matching, this is my forth breadcrumb, and the next one will wrap up the series with rate and irate operators as well as Push Gateway.

Why to aggregate, and what is it?

Prometheus allows you to aggregate metrics, basically play around with data so you can get more insights about what's going on in your system. For example, you could be grabbing cpu metrics for several systems and maybe get the average of it. Collecting metrics without doing maths after it's not going to give you any insights, normally you want to do averages, sum stuff and so on.

However, there is a problem, we have explained in the past that metrics have labels associated to them. So a CPU metric might be something like this {server="red", zone="Europe", instance="1"} but another CPU metric without all those three labels is not going to be considered the same metric for Prometheus. If you want to aggregate all the cpu metrics, you need to tell Prometheus how to aggregate the metrics with different labels so it can perform the average operation.

How to tell Prometheus how to aggregate things?

There are two clauses to tell Prometheus how to aggregate metrics:

  • By clause, e.g. sum by(label)(metric): This tells Prometheus to aggregate using the specified label or labels.
  • Without clause, e.g. sum without(label)(metric): This is the opposite, kind of "not" operation. We aggregate by using the labels not specified in the without clause.

Now, this is only applicable to aggregation functions like sum, avg, count, etc. What if you want to perform "normal" operations like divide? For example, http_errors_total / http_requests_total. Here is when vector matching comes in:

  • on clause, e.g. on(label) metric: It matches metrics using the specified labels. So you can make the division using http_errors_total / on(instance) http_requests_total. So Prometheus will divide the errors with the requests if the label instance matches.
  • ignoring clause e.g. ignoring(label) metric. The opposite of on. It matches on all labels less the specified one.

Ok, so I can tell Prometheus how to match metrics with different labels. So far, all we have explained is going to work if http_errors_total has the same number of metrics as http_requests_total after matching the labels. So one http_errors_total with label blue is going to match one http_requests_total with label blue. This is one to one matching. But what if one side has more metrics than the other one? Here comes one to many and many to one matching which is done with the following operators that can accompany on/ignoring clauses:

  • group_left: It tells Prometheus that the left side can have more metrics than the right one.
  • group_right: It tells Prometheus that the right side can have more metrics than the left one.

So, if you have these two metrics for errors total:

http_errors_total{instance="blue", method="GET"}
http_errors_total{instance="blue", method="POST"}

And only one request metric:

http_requests_total{instance="blue"}

When you do:

http_errors_total / on(instance) group_left http_requests_total

This will give you two results, one dividing GET errors by total requests for blue, and one dividing POST errors by total requests for blue.

Conclusion

In Prometheus it doesn't matter if you are playing around with metrics with different labels and different number of metrics. Prometheus gives you the tools to aggregate metrics with by/without, and combine different metrics together with vector matching using on/ignoring and group_left/group_right. With these functions you can get more meaningful insights out of your raw metrics.

And that's all for this week's breadcrumbs. As always my goal with these breadcrumbs is to share my knowledge with easy and short posts. If you want to stay tuned with Backend and DevOps stuff, feel free to follow me and if you feel extra generous please give some love to the post by commenting on it or reacting.

p.s. Amazing blog post image I know...

Prometheus breadcrumbs 4 - Javier Guzman