Observability
Alert To Runbook Map
Keep this table synchronized with your PrometheusRule package. Alert annotations should link to the matching page or heading. Use the Observability Change Checklist when adding, removing, or changing alerts, alert annotations, runbook links, dashboard links, or actionable Events.
Alerts
| Alert | Primary runbook | First checks |
|---|---|---|
BloodravenOperatorDown | Operator unavailable | Deployment, leader election, logs |
BloodravenNoWritableSite | Emergency manual promotion or Total site loss | Active site, pod reachability, fencing |
BloodravenFailoverOccurred | Failover | DNS, app writes, old primary state |
BloodravenDivergentTransactions | Divergent old primary recovery or Divergent replica | Events (DataLossDetected / DivergentReplicaDetected), status.sites[].divergenceDetectedBy, GTIDs, old primary fenced |
BloodravenReplicationLagging | Replication lag high | lag metric, MySQL replica status |
BloodravenReplicationDown | Replication lag high; sites awaiting a post-restore reclone are excluded by design (Peer replication during a full-instance restore) | bloodraven_replication_running, bloodraven_restore_reclone_pending, MySQL replica status |
BloodravenRestoreReclonesHeld | Peer reclones after a full-instance restore | status.restoreReclones, status.restoreInPlace.message, the primary's data before choosing what to keep |
BloodravenBackupStale | Failed backup | latest MysqlBackup, Job logs |
BloodravenBackupArtifactCleanupFailing (increase(bloodraven_backup_artifact_cleanup_failures_total[6h]) > 0) | Artifact cleanup | ArtifactCleanup condition on Terminating MysqlBackups, cleanup Job termination message, S3 credentials or claim |
BloodravenBackupVerificationStale | Backup Verification | verification CRs, restore logs |
BloodravenPITRArchiveLagging | Backup And Restore | archiver metrics, object storage, PITRArchiveReady reason and status.pitr.archive (an archive identity refusal stops uploads) |
BloodravenDNSUpdateFailed | DNS failover stuck | DNSEndpoint, external-dns logs |
BloodravenSplitBrainDetected | Split-brain recovery | writable sites, app traffic, GTIDs |
BloodravenDragonflySiteDown | Dragonfly degraded | status.dragonfly.sites[], Dragonfly pods, active Service endpoints |
BloodravenDragonflyPromotionFailed | Dragonfly degraded | planned-failover Dragonfly status, Events, Redis client impact |
BloodravenDragonflySessionsLost | Dragonfly degraded | status.dragonfly.replTakeoverSupported, DragonflySessionsLost Events, image pin |
BloodravenDragonflyManagerPanic | Dragonfly degraded | operator logs, panic counter, current status.dragonfly |
BloodravenKeyringNotSealed | Data-at-rest encryption | status.encryptionAtRest.sites[].phase and .message, escrow Secret exists, sidecar /keyring/status |
BloodravenKeyringEscrowFailing | Data-at-rest encryption | bloodraven_keyring_escrow_pushes_total{outcome="failure"}, sidecar logs, operator /keyring/escrow rejections |
BloodravenUnencryptedTablespaces | Data-at-rest encryption | bloodraven_encryption_unencrypted_tablespaces, whether the group was adopted with partial coverage |
BloodravenLicenseUpdatesExpiring | Licensing | bloodraven_license_updates_expiry_timestamp_seconds. Renewal reminder only — the operator does not gate or change behavior when the timestamp is in the past. |
Kubernetes Events
| Event category | Expected operator action | Runbook |
|---|---|---|
| Planned failover requested/started/completed | Drain writes, promote target, update DNS | Planned Failover |
PlannedFailoverFailed with outcome SourceLeftFenced | Nothing was written to the target; the source stays read-only. The operator re-asserts an attributable fence on its own; some causes are manual | Planned failover left the source fenced |
PlannedFailoverPromotionConfirmationFailed | Names the target as the last failover target without a cooldown; emergency failover may promote the fenced source | Promoted but unconfirmed |
PlannedFailoverManualRecoveryRequired / PlannedFailoverHoldPersisted | Holds automatic failover for the group (AutomaticFailoverHeld=True) until a human acknowledges it. Page | Planned failover automation hold |
PlannedFailoverStalledManagerAbsent / PlannedFailoverStatusStalled | No topology manager, or status writes failing, while a planned failover holds its claim; the run ends after 5 or 2 minutes (10 seconds once its promotion is recorded and published) | Planned failover |
PlannedFailoverDeferred with DeploymentHold | Inspect blocking operation, instance, and admission.retryTime; prefer completion/release or expiry over interruption | Deployment coordination |
DeploymentLeaseRevoked | Stop deployment heartbeat and further DDL; check topologyGeneration, revocation reason, and post-DDL recovery requirements | Deployment coordination |
| Emergency failover started/completed | Promote best candidate, taint old site, update DNS | Failover |
| Reclone started/completed/failed | Rebuild replica from active primary | Divergent old primary recovery |
| Backup started/completed/failed | Create/update MysqlBackup Job and status | Failed backup |
ArtifactCleanupFailed / ArtifactCleanupBlocked / ArtifactCleanupNothingFound | Retry or block artifact cleanup of a deleted MysqlBackup; the CR stays Terminating | Artifact cleanup |
ArtifactCleanupStalledProducerStop | Deleting an abandoned backup waits for its producing Job's pods to stop; one has not after 10 min. Once its node is confirmed down, force-delete the pod | Artifact cleanup |
ArtifactCleanupDeferred / RetentionDeferred | Keep a backup a verification is reading; informational unless it persists after the run ended | Backup verification |
| Restore started/completed/failed | Gate bootstrap or restore workflow | Failed restore |
| Verification started/completed/failed | Restore backup into ephemeral MySQL | Backup Verification |
PITRArchiveNotReady with an Archive* reason (ArchiveLegacy, ArchiveForeign, ArchiveCopy, ArchiveDescriptorMissing, ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported, ArchiveStoreUnsupported) | A site refused the archive's identity and uploads nothing; usually point PITR at a new prefix and take a full backup | Archive identity |
PITRArchiveUnsafeHistoryDetected | A site set its permanent history-integrity marker after an out-of-band change; undo it, new prefix plus full backup. The marker removes every restore target that needs that site's bytes, including earlier ones | Unsafe history marker |
PITRMysqlConfRefused | The operator left a PITR-unsafe spec.mysqlConf option (filter, binlog_format, binlog_row_image) out of my.cnf; PITRMysqlConfAccepted=False names it. Remove the key, or disable PITR if the setting is required | Operator-owned MySQL settings |
PITRArchiveReissueRiskDetected | A site with divergent or quarantined archive rows was promoted; if it was recloned after a lossy failover, start a new prefix and take a full backup | Reissue risk after a reclone |
PITRArchiveEncryptionChangeRefused / PITRArchiveKeyChangeRefused | A passphrase or encryption change on a profile whose archive is in use was refused; pods keep the accepted key snapshot and archiving continues. Revert the edit, or rotate by moving to a new prefix | Rotation |
PITRArchiveKeyMaterialLost | A Bound binding's key snapshot is missing or wrong; MySQL keeps running, the archiver fails closed. Restore the Secret from a backup of it, or set a profile to the original passphrase so the operator re-creates a missing one. Never delete a snapshot that is present: it may hold the only copy of the key | Key snapshots |
PITRArchiveBindingRefused | A binding was refused (wrong key, foreign, copy, legacy, missing archive) or pre-binding Deployments (any site Deployment without a decodable render annotation) are deferred; the group holds its last accepted render, restores wait or fail | Archive encryption bindings |
PITRArchiveBindingLimitReached | 16 entries that stay (Bound or used by a target) exist; release unused ones by id | Releasing a binding |
PITRArchiveBindingReleaseRefused / PITRArchiveBindingReleaseRejected | A release is waiting for its binding to be unused, or its annotation was malformed and removed | Releasing a binding |
PITRArchiveKeySourceRemoved / PITRArchiveKeySnapshotRecreated / PITRArchiveKeySnapshotReadFailed / BackupScheduleRefused | Informational: the profile's Secret is gone (snapshot still serves), a snapshot was re-created, an acceptance annotation is malformed (never used, never deleted), or a schedule targets a read-only profile | Archive encryption bindings |
PITRDivergentHistoryExcluded | A restore or verification excluded transactions outside the target history; confirm the excluded set is the lost, errant or other-branch history you meant to drop | Target history and excludeGtids |
PITROnHistoryTransactionsExcluded | An excluded transaction on the target history precedes replayed ones; the result is not certified for it, so check the affected rows | On-history exclusions |
PITRReplayTruncated | allowPartial restored to the archive head instead of the requested target; confirm the earlier point meets the recovery objective | Partial outcomes |
PITR restore or verification refused (status.restore.reason or the Verified reason is a PITR* token) | Read the token and its message: identity, dump, frontier, divergent history, gap, order or replay | Restore refusals |
PITRArchiveRefused (MysqlStandbyCluster) | The DR standby refused the source archive; the dump is published without a PITR window | Restoring from a replicated archive |
| DNS update created/failed | Write DNSEndpoint for external-dns | DNS failover stuck |
| Split-brain or recovery pending | Fence losers or wait for manual recovery | Split-brain recovery |
| Dragonfly promotion/sync/upgrade events | Preserve or restore cache/session continuity while MySQL remains authoritative | Dragonfly degraded |
KeyringPhaseChanged / KeyringUnsealed | Informational — a site is moving through the keyring lifecycle | Data-at-rest encryption |
KeyringEscrowMissing / KeyringEscrowCorrupt | Urgent. A sealed site's only copy of its keys is gone or altered; do not restart that pod | Data-at-rest encryption |
KeyringNotReadOnly / KeyringDigestMismatch | Investigate the rendering — the site is not actually protected | Data-at-rest encryption |
KeyringEscrowTimedOut | A site has not escrowed its keyring within keyring.escrowTimeout and is Failed; it stays unsealed and the operator keeps retrying. Check the sidecar's /keyring/status and its logs | Data-at-rest encryption |
KeyringRotationRejected / EncryptionAdoptionRefused | Follow the supported rotation or adoption procedure | Data-at-rest encryption |
KeyringPromotionSkipped / KeyringPromotionRefused | A rotating site is not a promotion target. If the group has no primary, finish the rotation (there is no cancel); then failover proceeds on the next poll | Data-at-rest encryption |
LicenseInvalid | None required for availability. Check the token string and kid. Operator behavior is unchanged. | Licensing |
LicenseVerified | Informational | Licensing |
Minimum alert annotations
Each alert should include:
summarywith the failing group and namespace.descriptionwith immediate user impact.runbook_urlpointing to this docs site.dashboard_urlpointing to the relevant Grafana dashboard.
If an alert intentionally has no runbook or dashboard link, the change must include a specific operational rationale in the pull request. Do not use a bare N/A.