Prometheus Breadcrumbs 3

Last week we spoke about labels in Prometheus. This week I'm gonna talk about scalability, that's where all the meat is when we talk about infrastructure.

So when we talk about observability, at the end of the day it's just one entity grabbing logs and metrics from every element of your system. This of course consumes CPU and memory as well as network bandwidth. If the resources available are low or the system is huge, we have a problem as we might run out of resources. Actually, even OpenAI had a post-mortem when they rolled out new telemetry, the link is here in case you are curious https://status.openai.com/incidents/01JMYB483C404VMPCW726E8MET/write-up

This is the same problem as databases, you can use them until they start to struggle with the current load, and then you have to come up with a solution for scaling this up.

Prometheus comes with three different ways to scale:

1. Federation: In non-intensive setups, we have just a Prometheus server. With federation, we create a bunch of Prometheus servers to grab and store the information. Then we have a Global Prometheus server that aggregates data from all these servers. For organization, it is recommended that each server adds a label so the Global server knows where the data is coming from.
2. Vertical scaling: The old way of just increasing resources i.e. cpu, memory, etc. of the Prometheus server.
3. Horizontal scaling: a.k.a sharding, basically you have many servers running, and each one is in charge of a subset of your system. For example, server 1 takes care of service A,B,C while server 2 takes care of service X,Y,Z. In addition, instead of splitting your system in subsets, you can make all servers take care of all services but using hashes in order to dictate which server takes care of that information.

Federation is best for data aggregation; you keep local data on each server, but then you can aggregate data in the global one and query the aggregations there. Vertical is the easy thing you can do for scaling things in the short term. Horizontal is when things get serious. If your system is huge, you want to go with federation plus horizontal scaling.

And that's all for this week's breadcrumbs. As always my goal with these breadcrumbs is to share my knowledge with easy and short posts. If you want to stay tuned with Backend and DevOps stuff, feel free to follow me and if you feel extra generous please give some love to the post by commenting on it or reacting.