Openshift ARO: Configuring Alert Manager and Remote Write with PVC

Objective

In this post, I will show how to make the configuration of Prometheus stack with PVC in an OpenShift cluster running on Azure (ARO).

ARO setup:

• Prometheus: PVC configuration
• Alert Manager: Additional target without changing the default

Microsoft References:

Remote writers

Set up remote write of Prometheus monitoring data to an Azure Monitor workspace - Azure Red Hat…

Receivers
https://learn.microsoft.com/en-us/azure/openshift/support-policies-v4#logging-and-monitoring

Setup

My setup is considering the infrastructure nodes and importing a custom CA, since my environment needs to send data to a custom environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

    prometheusK8s:
      nodeSelector:
        node-role.kubernetes.io/infra: ""

      tolerations:
        - key: node-role.kubernetes.io/infra
          operator: Exists
          effect: NoSchedule
        - key: dedicated
          operator: Equal
          value: infra
          effect: NoSchedule

      externalLabels:
        cluster: openshift-aro01

      remoteWrite:
        - url: https://remote.example.com:8427/api/v1/write # remote victoria metrics 
          tlsConfig:
            ca:
              secret:
                name: remote-write-ca
                key: ca.crt
          basicAuth:
            username:
              name: monitoring-secret 
              key: username
            password:
              name: monitoring-secret 
              key: password
      retention: 7d

      volumeClaimTemplate:
        metadata:
          name: prometheusdb
        spec:
          storageClassName: managed-csi
          resources:
            requests:
              storage: 150Gi

    alertmanagerMain:
      enableUserAlertmanagerConfig: true

      nodeSelector:
        node-role.kubernetes.io/infra: ""

      tolerations:
        - key: node-role.kubernetes.io/infra
          operator: Exists
          effect: NoSchedule
        - key: dedicated
          operator: Equal
          value: infra
          effect: NoSchedule

      volumeClaimTemplate:
        metadata:
          name: alertmanager
        spec:
          storageClassName: managed-csi
          resources:
            requests:
              storage: 80Gi

Secret:

apiVersion: v1
kind: Secret
metadata:
  name: monitoring-secret
  namespace: openshift-monitoring
type: Opaque
data:
  username: xxxxxxxx
  password: yyyyyyyy

After that, we can see the pods going up, and the PVC will be created:

In the past, this configuration was not possible in OpenShift ARO deployments.

After that let’s go to the console Administration > Cluster Settings > Configuration > Alertmanager

global:
  http_config:
    proxy_from_environment: true

inhibit_rules:
- equal:
  - namespace
  - alertname
  source_matchers:
  - severity = critical
  target_matchers:
  - severity =~ warning|info

- equal:
  - namespace
  - alertname
  source_matchers:
  - severity = warning
  target_matchers:
  - severity = info

receivers:
- name: Default
  webhook_configs:
  - url: http://aro-operator-master.openshift-azure-operator.svc.cluster.local:8080/healthz/ready

- name: Watchdog

- name: Critical

- name: my-example
  webhook_configs:
  - send_resolved: true
    url: https://mywebhook.example.com/events/alertmanager
    http_config:
      tls_config:
        insecure_skip_verify: true

route:
  group_by:
  - namespace
  group_interval: 5m
  group_wait: 30s
  receiver: Default
  repeat_interval: 12h

  routes:
  - matchers:
    - alertname = Watchdog
    receiver: Watchdog

  - matchers:
    - severity = critical
    receiver: Critical

  - receiver: my-example

As I mentioned at the start of the article, I’m only adding a new webhook_config and not changing from the default configuration.