Loki Stream Rate Limits: OpenShift Proactive Alert

I will demonstrate how Loki per-stream rate limiting works in practice. I created a simple workload that floods the logging pipeline until it exceeds the recommended 5 MB/s stream ingestion threshold.

The goal of this test is to reproduce the issue in a controlled environment and build a proactive alert before the limit is reached.

Configuring LokiStack Limits

Before starting the test, configure your LokiStack with explicit ingestion limits.

The following example sets the recommended per-stream rate limit values:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  limits:
    global:
      ingestion:
        ingestionRate: 10
        ingestionBurstSize: 20
        perStreamRateLimit: 5
        perStreamRateLimitBurst: 20

This configuration means:

• ingestionRate: Total ingestion rate allowed per tenant (10 MB/s)
• ingestionBurstSize: Maximum burst allowed across ingestion (20 MB)
• perStreamRateLimit: Maximum sustained rate per stream (5 MB/s)
• perStreamRateLimitBurst: Maximum temporary burst per stream (20 MB)

Note: The ingestionRate must always be higher than perStreamRateLimit.

A long explanation of those values can be verified at :

Understanding Loki Rate Limits in OpenShift Logging

Deploying a Log Flooder Application

The following deployment continuously generates random data to stdout, producing enough log traffic to trigger Loki throttling.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-flooder
  namespace: test-logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-flooder
  template:
    metadata:
      labels:
        app: log-flooder
    spec:
      containers:
      - name: flooder
        image: registry.access.redhat.com/ubi9/ubi-minimal
        command:
        - /bin/bash
        - -c
        - |
          while true; do
            head -c 60000 /dev/urandom | base64
            sleep 0.01
          done

This workload generates approximately 6 MB/s of logs, enough to exceed Loki’s per-stream rate limit in most default/recommended configurations.

Verifying the Generated Log Volume

You can validate the log throughput directly from the OpenShift metrics dashboard or by querying:

sum by (container_name, pod_namespace, pod_name) (
  rate(vector_component_received_event_bytes_total{component_type="kubernetes_logs"}[2m])
)

Once the value crosses 5 MB/s, Loki may begin returning:

per stream rate limit exceeded

Creating a Proactive Alert

Instead of waiting until the limit is hit, it is recommended to trigger an alert before reaching the maximum threshold.

The following alert fires at 4 MB/s, giving enough time for investigation before throttling begins.

Full PrometheusRule YAML

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: loki-log-ingestion-alerts
  namespace: openshift-monitoring
spec:
  groups:
    - name: container-log-alerts
      rules:
        - alert: HighContainerLogVolume
          annotations:
            summary: High log volume detected for container {{ $labels.container_name }}
            description: >
              Container {{ $labels.container_name }} in namespace
              {{ $labels.pod_namespace }} from pod {{ $labels.pod_name }}
              is generating more than 4 MB/s of logs for the last 15 minutes,
              approaching Loki's recommended per-stream ingestion limit.
          expr: |
            sum by (container_name, pod_namespace, pod_name) (
              rate(
                vector_component_received_event_bytes_total{
                  component_type="kubernetes_logs"
                }[2m]
              )
            ) > 4194304
          for: 15m
          labels:
            severity: warning
            team: observability

Why 4 MB/s?

Although Loki’s recommended limit is 5 MB/s, alerting exactly at 5 MB is often too late.

Using 4 MB/s provides a safety margin to:

• Detect noisy applications before throttling starts
• Investigate sudden spikes
• Prevent dropped logs during burst scenarios

Conclusion

After testing, remove the flooder deployment:

oc delete deployment log-flooder -n test-logging

This setup provides a straightforward way to simulate Loki ingestion throttling in OpenShift and validate that your alerting pipeline can detect excessive log generation before logs are dropped.