Converting CPU Counters to Usage Percentage in Prometheus: From Raw Metrics to Actionable Insights

Fundamental Principles of Counter-to-Percentage Conversion

In the Prometheus monitoring ecosystem, CPU usage calculation revolves around a core concept: counters. Metrics like container_cpu_user_seconds_total and container_cpu_system_seconds_total are typical counter types, recording CPU time consumed by containers in user space and kernel space respectively (measured in seconds). These counters are monotonic, meaning their values only increase over time and never decrease. This design ensures accurate cumulative tracking, but for real-time monitoring, raw cumulative values lack intuitive meaning.

The Central Role of the rate() Function

Transforming these counters into meaningful CPU usage percentages requires understanding Prometheus's rate() function. This function calculates the average per-second increase rate of a counter over a specified time window. For example, rate(container_cpu_user_seconds_total[30s]) computes the average per-second increment of this counter over the past 30 seconds. Physically, this value represents how many seconds of CPU time the container used per second on average during the last 30 seconds. Since CPU time is measured in seconds, the result essentially indicates the average number of CPU cores used per second.

To convert this to a percentage format, a simple mathematical transformation is applied. If the rate() function returns a value representing CPU cores used per second, multiplying by 100 yields a percentage. For instance, if a container uses 0.5 CPU cores per second on average over the past 30 seconds, its CPU usage is 50%. Thus, the basic conversion formula is:

rate(container_cpu_user_seconds_total[30s]) * 100

This query returns the percentage of CPU usage in user space for the container. Similarly, system space usage can be calculated using container_cpu_system_seconds_total. For total CPU usage (user + system), both counters can be combined:

rate(container_cpu_user_seconds_total[30s] + container_cpu_system_seconds_total[30s]) * 100

Time Window Selection and Optimization

Choosing an appropriate time window is crucial for calculation accuracy. Shorter windows (e.g., 30 seconds) respond faster to CPU usage changes but are more sensitive to transient fluctuations; longer windows (e.g., 5 minutes) smooth out short-term variations, providing a more stable trend view but potentially masking rapid changes. In practice, the window can be adjusted based on monitoring needs. For example, alerts requiring quick response might use shorter windows, while performance analysis and capacity planning benefit from longer windows.

Additionally, Prometheus offers the irate() function, which calculates instantaneous growth rates rather than average rates. irate() performs differentiation on the last two data points, making it more sensitive to rapid changes but potentially introducing more noise. For CPU monitoring, rate() is generally preferred as it provides smoother, more interpretable results.

Extended Applications at Aggregation Levels

In production environments, especially within Kubernetes clusters, CPU monitoring often requires aggregation at different levels. While individual container monitoring is important, aggregated views at the Pod, node, and namespace levels are equally critical for resource management and troubleshooting. Here are some practical aggregation query examples:

Calculate average CPU usage per Pod over the last 5 minutes:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod) * 100

Calculate average CPU usage per node:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (node) * 100

Calculate average CPU usage per namespace:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace) * 100

In these queries, the container!="" filter excludes redundant metrics related to cgroups hierarchy, ensuring calculation accuracy. Using sum() and by() clauses, CPU usage from multiple containers can be aggregated by specified labels to form higher-level views.

Comparison with Alternative Monitoring Approaches

Beyond container-based counter calculations, Prometheus supports deriving CPU usage from node-level metrics. For example, by monitoring idle CPU time at the node level, overall CPU usage can be indirectly calculated:

100 - (avg by (instance) (irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) * 100)

This method is suitable for node-level monitoring but not for fine-grained container analysis. In actual deployments, appropriate data sources and calculation methods should be selected based on monitoring objectives.

Common Issues and Best Practices

When implementing CPU monitoring, several common issues should be noted. First, since the rate() function calculates the average number of CPU cores used per second, results can exceed 100% when containers utilize multiple cores. This is not an error but reflects the actual CPU resources consumed. For instance, if a container uses 2 cores on a 4-core node, its CPU usage will show as 200%.

Second, time series cardinality can become a performance bottleneck. Particularly in large-scale Kubernetes clusters, container_cpu_usage_seconds_total may generate numerous time series. Through proper label filtering and aggregation, cardinality can be effectively controlled to improve query performance.

Finally, it is advisable to combine CPU monitoring with other metrics (e.g., memory usage, network traffic) for a comprehensive system overview. For example, high CPU usage with low memory usage might indicate compute-intensive tasks, while high CPU usage with high memory usage could suggest memory leaks or garbage collection issues.

By applying these methods, an accurate and efficient CPU monitoring system can be built in Prometheus, providing robust support for performance management and troubleshooting in containerized environments.