Prometheus Setup

Bloodraven exposes Prometheus metrics on the operator metrics Service, port 8080 named metrics.

Helm values

metrics:
  service:
    enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s
    labels:
      release: kube-prometheus-stack

helm upgrade --install bloodraven bloodraven/bloodraven \
  --namespace bloodraven \
  --create-namespace \
  --values bloodraven-values.yaml

ServiceMonitor

The chart renders a ServiceMonitor when metrics.serviceMonitor.enabled=true. It selects the operator metrics Service in the release namespace.

If you manage the monitor yourself:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: bloodraven
  namespace: bloodraven
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - bloodraven
  selector:
    matchLabels:
      app.kubernetes.io/name: bloodraven
  endpoints:
    - port: metrics
      interval: 30s
      scrapeTimeout: 10s

Plain Prometheus scrape config

scrape_configs:
  - job_name: bloodraven
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
            - bloodraven
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
        action: keep
        regex: bloodraven
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        action: keep
        regex: metrics

Verify targets

kubectl get service -n bloodraven
kubectl get servicemonitor -n bloodraven
kubectl port-forward -n bloodraven deploy/bloodraven 8080:8080
curl http://localhost:8080/metrics | grep '^bloodraven_'

In Prometheus, check Status > Targets for the bloodraven job or ServiceMonitor-generated target.

Reader and source-convergence monitoring

bloodraven_replication_source_state{namespace,group,site,state} is a state-set gauge for every follower. The namespace and group labels keep identically named sites in different failover groups in separate series. It emits the bounded state values converged, pending, and blocked; exactly one is 1 for a follower and the others are 0. Active primaries have no active source state. Combine it with bloodraven_replication_running{site,thread} and bloodraven_replication_lag_seconds{site} when alerting on a reader.

Reader failures are deliberately isolated from the failover group's shared Ready and Degraded conditions. A reader can be unreachable, lagging, or source-blocked while the core candidate/DR topology remains Ready and not Degraded. Alert on reader sites directly rather than inferring reader health from group conditions. For example:

bloodraven_replication_source_state{namespace="warehouse",group="orders",site="reader",state!="converged"} == 1

bloodraven_replication_running{site="reader"} == 0
or bloodraven_replication_lag_seconds{site="reader"} > 30

Use a sustained for interval appropriate to the normal poll and recovery cadence. A blocked source commonly needs GTID investigation and possibly a cold reclone; a pending source may clear on a later bounded retry.

Alerts

Keep alert rules with your platform monitoring stack. Alert names should link to Alert To Runbook Map, and metric details live in Monitoring Reference.