Observability
Alert To Runbook Map
Keep this table synchronized with your PrometheusRule package. Alert annotations should link to the matching page or heading. Use the Observability Change Checklist when adding, removing, or changing alerts, alert annotations, runbook links, dashboard links, or actionable Events.
Alerts
| Alert | Primary runbook | First checks |
|---|---|---|
BloodravenOperatorDown | Operator unavailable | Deployment, leader election, logs |
BloodravenNoWritableSite | Emergency manual promotion or Total site loss | Active site, pod reachability, fencing |
BloodravenFailoverOccurred | Failover | DNS, app writes, old primary state |
BloodravenDivergentTransactions | Divergent old primary recovery | Events, GTIDs, old primary fenced |
BloodravenReplicationLagging | Replication lag high | lag metric, MySQL replica status |
BloodravenBackupStale | Failed backup | latest MysqlBackup, Job logs |
BloodravenBackupVerificationStale | Backup Verification | verification CRs, restore logs |
BloodravenPITRArchiveLagging | Backup And Restore | archiver metrics, object storage |
BloodravenDNSUpdateFailed | DNS failover stuck | DNSEndpoint, external-dns logs |
BloodravenSplitBrainDetected | Split-brain recovery | writable sites, app traffic, GTIDs |
BloodravenDragonflySiteDown | Dragonfly degraded | status.dragonfly.sites[], Dragonfly pods, active Service endpoints |
BloodravenDragonflyPromotionFailed | Dragonfly degraded | planned-failover Dragonfly status, Events, Redis client impact |
BloodravenDragonflySessionsLost | Dragonfly degraded | status.dragonfly.replTakeoverSupported, DragonflySessionsLost Events, image pin |
BloodravenDragonflyManagerPanic | Dragonfly degraded | operator logs, panic counter, current status.dragonfly |
BloodravenKeyringNotSealed | Data-at-rest encryption | status.encryptionAtRest.sites[].phase and .message, escrow Secret exists, sidecar /keyring/status |
BloodravenKeyringEscrowFailing | Data-at-rest encryption | bloodraven_keyring_escrow_pushes_total{outcome="failure"}, sidecar logs, operator /keyring/escrow rejections |
BloodravenUnencryptedTablespaces | Data-at-rest encryption | bloodraven_encryption_unencrypted_tablespaces, whether the group was adopted with partial coverage |
BloodravenLicenseUpdatesExpiring | Licensing | bloodraven_license_updates_expiry_timestamp_seconds. Renewal reminder only — the operator does not gate or change behavior when the timestamp is in the past. |
Kubernetes Events
| Event category | Expected operator action | Runbook |
|---|---|---|
| Planned failover requested/started/completed | Drain writes, promote target, update DNS | Planned Failover |
PlannedFailoverDeferred with DeploymentHold | Inspect blocking operation, instance, and retryAfter; prefer completion/release or expiry over interruption | Deployment coordination |
DeploymentLeaseRevoked | Stop deployment heartbeat and further DDL; check topologyGeneration, revocation reason, and post-DDL recovery requirements | Deployment coordination |
| Emergency failover started/completed | Promote best candidate, taint old site, update DNS | Failover |
| Reclone started/completed/failed | Rebuild replica from active primary | Divergent old primary recovery |
| Backup started/completed/failed | Create/update MysqlBackup Job and status | Failed backup |
| Restore started/completed/failed | Gate bootstrap or restore workflow | Failed restore |
| Verification started/completed/failed | Restore backup into ephemeral MySQL | Backup Verification |
| DNS update created/failed | Write DNSEndpoint for external-dns | DNS failover stuck |
| Split-brain or recovery pending | Fence losers or wait for manual recovery | Split-brain recovery |
| Dragonfly promotion/sync/upgrade events | Preserve or restore cache/session continuity while MySQL remains authoritative | Dragonfly degraded |
KeyringPhase / KeyringUnsealed | Informational — a site is moving through the keyring lifecycle | Data-at-rest encryption |
KeyringEscrowMissing / KeyringEscrowCorrupt | Urgent. A sealed site's only copy of its keys is gone or altered; do not restart that pod | Data-at-rest encryption |
KeyringNotReadOnly / KeyringDigestMismatch | Investigate the rendering — the site is not actually protected | Data-at-rest encryption |
KeyringRotationRefused / EncryptionAdoptionRefused | Follow the supported rotation or adoption procedure | Data-at-rest encryption |
KeyringPromotionSkipped / KeyringPromotionRefused | A rotating site is not a promotion target. If the group has no primary, finish the rotation (there is no cancel); then failover proceeds on the next poll | Data-at-rest encryption |
LicenseInvalid | None required for availability. Check the token string and kid. Operator behavior is unchanged. | Licensing |
LicenseVerified | Informational | Licensing |
Minimum alert annotations
Each alert should include:
summarywith the failing group and namespace.descriptionwith immediate user impact.runbook_urlpointing to this docs site.dashboard_urlpointing to the relevant Grafana dashboard.
If an alert intentionally has no runbook or dashboard link, the change must include a specific operational rationale in the pull request. Do not use a bare N/A.