Kubernetes 1.37 Elevates Native Histogram Support to Beta

Kubernetes 1.37 brings a significant upgrade in observability by promoting native histogram support to beta. This change aims to enhance metrics accuracy and drastically reduce the overhead associated with telemetry storage and scraping. Native histograms, previously introduced as an alpha feature in Kubernetes 1.36 under KEP-5808, are now enabled by default in the latest release. The shift from classic histograms to native ones marks a substantial improvement in how Kubernetes metrics are handled.

What Changed with Native Histograms?

The introduction of native histograms in Kubernetes addresses several challenges associated with classic histograms. Classic histograms required predefined bucket boundaries, which could lead to visibility gaps if the workload's latency profile changed. For instance, if latencies suddenly shifted into microsecond ranges or experienced long-tail latencies beyond the highest bucket, the histogram would lose visibility on those metrics. Specifying bucket boundaries upfront required knowing the distribution before observing it, which was often impractical.

This approach also resulted in high cardinality and increased storage costs because each bucket boundary was exported as a separate time series. A histogram with 10 buckets across multiple labels could multiply the number of time series by 10, thereby increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs. Moreover, classic histograms suffered from interpolation errors when calculating quantiles due to static bucket boundaries. Calculating percentiles using histogram_quantile() relied on linear interpolation between static bucket boundaries, which could lead to significant estimation errors when bucket spans were coarse.

In contrast, Prometheus native histograms use dynamic, exponential buckets, which automatically adjust to any value range. This flexibility eliminates the need for predefined boundaries and reduces the number of time series by up to 90%. Quantile calculations also become more accurate, with a worst-case relative error of approximately 5% under default settings.

How Native Histograms Work in Kubernetes

In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem, k8s.io/component-base/metrics. This integration ensures broad component support, including major components like kube-apiserver, which exposes metrics like apiserver_request_duration_seconds, kube-scheduler, and kubelet. The design focuses on zero disruption for existing observability stacks by utilizing dual exposition. This means that classic buckets are emitted alongside native spans, allowing existing Prometheus setups to continue functioning without modification.

The default exponential configuration is tuned with a BucketFactor of 1.1 and a MaxBucketNumber of 160. The BucketFactor ensures that each exponential bucket is at most 10% wider than the preceding one, guaranteeing a mathematically bounded worst-case relative error of at most ~5% for quantile calculations. The MaxBucketNumber caps the maximum number of buckets per histogram to 160, following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation. This limit protects component memory usage even under extreme outlier distributions.

Practical Steps to Implement Native Histograms

To leverage native histograms in your Kubernetes environment, follow these steps:

• Upgrade to Kubernetes 1.37: This version enables native histograms by default, so simply upgrading your cluster is the first step.
• Configure Prometheus: If you're using Prometheus 3.0 or higher, update your scrape configurations to include scrape_native_histograms: true and always_scrape_classic_histograms: true. This ensures both formats are collected safely during the transition. For Prometheus 2.40 to 2.x, enable Native Histograms globally by starting Prometheus with the feature flag --enable-feature=native-histograms.
• Migrate Queries: Transition your Grafana dashboards and Prometheus alerting rules from classic histogram queries to native histogram queries. Update references to classic _bucket, _count, and _sum series with histogram_count(...) and histogram_sum(...).
• Verify in Staging/Production: Test your updated queries in a staging environment before rolling them out to production to ensure dashboards and alerts function correctly. You can verify that a Kubernetes component is exporting native histograms using curl with an Accept header specifying Protobuf, checking for both traditional bucket entries and populated schema/positive_span fields.
• Reduce Storage Costs: Once migration is complete, disable classic histogram scraping by setting always_scrape_classic_histograms: false in your Prometheus configuration to unlock significant storage savings.

Limitations and Considerations

While the move to native histograms offers significant advantages, it is not without its trade-offs. The dual exposition approach means there is no immediate reduction in storage until you fully transition away from classic histograms. Additionally, some older versions of Prometheus may not support native histograms, requiring an upgrade.

Administrators must also be mindful of the need to restart components if disabling the NativeHistograms feature gate. While the transition process is designed to be smooth, careful planning and testing are essential to avoid disrupting existing observability setups. It's also important to note that while native histograms reduce the number of time series, they do not eliminate the need for careful monitoring of resource usage, especially in large-scale deployments where telemetry data can be voluminous.

Querying Native Histograms in PromQL

Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes. For example, to calculate the 99th percentile latency for a single target, you would use:

# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

Aggregating across multiple instances, such as all API servers, becomes simpler with native histograms as there's no need for grouping by le:

# Classic histogram (requires sum by (le) to preserve bucket boundaries):
histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket[5m])))

# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))

With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

Dashboard Migration & Rollback Strategy

The recommended migration workflow involves a careful transition to avoid breaking existing alerts or dashboards. Start by maintaining dual exposition to ensure compatibility. As you migrate your dashboards and alerts, validate each change in a staging environment. Once confident, apply the changes to production. If any issues arise, rollback to the classic setup by re-enabling classic histogram scraping. This strategy allows for a controlled and reversible transition.

For more insights on Kubernetes and its evolving features, you might find our posts on scheduler preemption and node maintenance enhancements useful.