Prometheus breadcrumbs 5

This is the last of my Prometheus breadcrumbs series where I'm gonna briefly talk about rate operators, subqueries and push gateway. Just to have the cherry on the top of the series.

Pushgateway

Remember we mentioned in breadcrumb number two that Prometheus was the one grabbing metrics? By the time you read that, you might be left wondering what happens with cron jobs or batch jobs? They normally live for a period of time and then disappear, so Prometheus cannot scrape metrics from them because maybe their lifetime is too short for Prometheus to even realize there was something running. That's when Pushgateway comes into action. With Pushgateway, the data workflow is the opposite, it's the service that pushes data to Pushgateway, and then, Prometheus scrapes the Pushgateway. One tip though, use honor_labels: true in the Prometheus scrape config, so the original labels from the service are kept and not replaced when Prometheus grabs the metrics.

Rate operators

There are a lot of operators in Prometheus but rate and irate are two of the most used ones:

rate operator follows this pattern rate(metric[time]). Let's imagine rate(cpu[5m]) it groups metrics in buckets or windows, in this case buckets of 5 minutes. Then we calculate the average of that window using the last data point and the first one of the bucket (which is the oldest, almost 5 minutes ago).

irate operator is the same, but instead of using last data point and the first one of that window, we use the last two data points.

irate looks into newer data as you can see, so it's normally used for graphing fast-moving/volatile counters. It's very sensitive so avoid it for alerts. On the other hand, rate represents slow moving counters, so spikes are smoothed and therefore can be used for alerts.

Subqueries

One thing to notice with rate and irate is that we have not put any time limit. When we do rate(cpu[5m]) we are just saying to work with 5m windows/buckets. But, how much data should we take for the calculation? Should we grab all our data since inception? Last month data? Just last hour? That's when subqueries come handy, they are used like this for example:

rate(http_requests_total[5m])[60m:60s]

In this example, we are getting the average using 5 minutes buckets. We get the average every minute (60 seconds) during 1 hour (60 minutes). We should get a result with 60 averages.

Conclusion

In this last breadcrumb about Prometheus we have explained briefly that pushgateway is used for sporadic services, that rate/irate are for calculating the average in a timeframe and subqueries to specify the time range of that calculation.

That's all for this week's breadcrumbs. As always my goal with these breadcrumbs is to share my knowledge with easy and short posts. Do you want me to talk about something in particular for my next breadcrumb series? Feel free to let me know in the comments, and I'll make my best to share what I know about that.