Operations
Failure-mode matrix
A single-table reference for "what happens when X breaks" — mapping common failure modes to the operator's detection signal, the action it takes, roughly when it acts, and what it will not do. Use this to cross-check an incident in progress against the operator's expected behavior, and to decide which alerts to carry.
Timing assumes the shipped defaults: spec.pollInterval=2s,
spec.failureThreshold=3, spec.failoverCooldown=5m, and sidecar
spec.sidecar.leaseTimeout=20s. The poll loop ticks every 2 seconds and probes
every reachable site on every poll. A site that has reached failureThreshold
and keeps failing has only its own probes backed off, exponentially to a
30-second cap, so any failure reaches the threshold in about 6 seconds, including
a second fault while another site is still down. The backoff does delay noticing
that a long-down site has come back: its next probe can be up to 30 seconds
away. See Failover: Debouncing.
Quick reference
| Failure | Observable signal | Operator action | Operator time-to-act | Operator limitations |
|---|---|---|---|---|
| Primary pod killed | MySQL poll fails on primary; pod NotReady. | Debounces to Unreachable → promotes replica via failover sequence (fence old primary, drain relay, promote, flip DNS). | First isolated fault: ≈ 6 s detection + up to 30 s relay drain, so at most ≈ 37 s total; the drain returns as soon as the backlog the replica already received is applied. The same holds while another site is already down: probe backoff is per site. | If replica is lagging, committed transactions that never reached it can be lost. In a healthy group, a primary that respawns inside the ≈ 6 s debounce window is not failed over; it comes back with the read_only=ON every mysqld boots with, and the operator re-asserts it on the next poll, so writes pause until then. |
| Replica pod killed | Poll fails on replica; replica becomes Unreachable and Degraded=True with reason SiteUnhealthy. | Marks replica Unreachable; narrows the -replicas Service selector to exclude it. No failover — the primary is still writable. | ≈ 6 s, including while another site is already down. | Read-only workloads hitting the -replicas Service lose endpoints until the replica returns. Backups fall back to the primary (controlled by maxLagSecondsForSource). |
| Primary node lost | Primary pod goes NotReady + no node heartbeat. | Same as primary pod killed — fails over to the replica. Kubernetes separately reschedules the pod onto a healthy node. | Up to ≈ 37 s, including while another site is already down. Pod reschedule is Kubernetes's job and depends on eviction policy (30 s – 5 min). | If the PVC is zone-bound (e.g. topology.kubernetes.io/zone-pinned EBS) and the lost node was the zone's only node, the pod cannot reschedule until capacity returns. The operator has already failed over; the zone-bound PVC problem is a Kubernetes concern. |
| Replica node lost | Poll fails on replica. | Marks replica Unreachable; narrows -replicas selector. No failover. | ≈ 6 s, including during a sustained outage elsewhere in the group. | Same as replica-pod-killed. Kubernetes reschedules the pod to another node with access to the PVC. |
| Primary PVC lost | Primary pod restarts into an empty MySQL or stuck bootstrap. GTID-empty detection fires on a successful poll. | Fails over (if the primary was active) → after recovery, auto-clones the empty site from the current primary via CLONE INSTANCE. | First-fault failover: up to ≈ 37 s. If the site was down long enough for its probes to back off, its first successful probe can be up to 30 s away. Clone duration depends on dataset size (minutes to hours). | Committed-but-unreplicated transactions on the lost PVC are unrecoverable. Auto-clone uses the operator Secret's replication credentials, which default to the operator account (see Failover → Prerequisites). |
| AZ / site partition (operator can still reach the surviving side) | All poll targets at the isolated side time out; one side's pods go NotReady. | Fails over to the reachable site. Sidecar on the partitioned primary self-fences at T + leaseTimeout = T + 20 s, closing the split-brain window. | Up to ≈ 37 s to promotion, including during a sustained outage elsewhere. Partitioned-side self-fencing remains driven by the 20 s sidecar lease. | If the partitioned side had unreplicated transactions, they are "stranded" until the partition heals — at which point the operator detects divergentGtid and blocks recovery until an admin runs the reclone flow. |
| Cross-site partition (link between MySQL pods dies; operator still reaches both) | Replica's IO thread stops receiving events; Seconds_Behind_Source climbs. Eventually Degraded=True ReplicationLagging. | None automatic. The primary is still writable, the replica is still read-only. The operator alerts but does not fail over. | Alert at maxLagSeconds (default 300 s). | This mode is indistinguishable, from the operator's point of view, from "replica fell behind because of I/O pressure". Human judgement decides whether to force a failover. |
| Operator pod crash | up{job="bloodraven"} == 0. Liveness probe fails. CR status stops updating. | Kubernetes restarts the pod. On return, leader election resumes, polling resumes. Until the primary's writable answer is confirmed (recoveryThreshold polls, ≈ 2 s more), Ready and Degraded read Unknown with reason Initializing, with no NoPrimary alert or Event. See Failover → Initializing after an operator start. | Pod restart ≈ 5–10 s. | No failover decisions happen during downtime. If the primary fails while the operator is down, writes are unavailable until the operator restart + a full failover sequence completes. Sidecars self-fence to preserve correctness. |
| Operator down + one site fails | Combined: operator liveness fails AND one MySQL site becomes Unreachable. | Sidecars self-fence at T + leaseTimeout; no promotion happens until the operator returns, then it runs the normal failover sequence. | Operator-down duration + up to ≈ 37 s. | Write availability is lost for the operator-down window plus the post-restart failover. Correctness is preserved (no split-brain), but availability takes a direct hit. RPO is unchanged vs. the operator-up case. |
| Both sites unreachable (TotalLoss) | All polls to both sites fail. Degraded=True with reason TotalLoss. | None. Emits TotalLossDetected Event and alerts. No automatic recovery is possible — nothing to fail over to. | ≈ 6 s after the last core site fails, whether or not the first one has been down for a while. | Requires human intervention. Once at least one site is back, the operator resumes; if both return, split-brain handling takes over. See Operations → Total loss recovery. |
Split brain (Writable/Writable) | Both sites poll as Writable simultaneously. | With no recorded failover history and empty spec.splitBrainPolicy.sitePriorities, alert via SplitBrainDetected and require human action. With priorities, fence every writable loser and promote the first listed writable candidate. After an operator-initiated failover, fence the returning stale primary regardless of priorities. | One poll interval, ≈ 2 s. A side returning from a long outage is first probed up to 30 s after it comes back, because its own probes had backed off. | No automatic data merge. If a fenced side accepted writes that the winner lacks, those transactions surface as divergentGtid and require replay or a reclone. See Failover → Split-brain resolution. |
| Old primary returns diverged | After a failover, the returning site has committed transactions the new primary never received. | Fences the returning site, records status.sites[].divergentGtid + count, sets RecoveryPending=True with reason DivergentTransactions, emits DataLossDetected Event. | The first successful observation can take up to 30 s, because the returning site's own probes back off while it is down. Connection drain and bounded recovery probes add time before divergence status is final. | Cannot auto-rejoin — divergent GTIDs would corrupt replication. Admin must replay the missing transactions or run the reclone flow; the operator enforces the confirmation interlock on the annotation. See Failover → Divergence detected. |
| Replica holds transactions the primary lacks (while replicating normally) | The replica GTID audit (follower read first, then the primary, about every 30 s) finds a non-empty follower − primary set, for example a standby demoted by a lossy promotion (issue #202) or a write made on a replica with binlogging on. | Records status.sites[].divergentGtid with divergenceDetectedBy=ReplicaAudit, sets Degraded=True with reason DivergentTransactions (Ready unchanged), emits DivergentReplicaDetected. Ordered updates and planned failovers do not promote the site; emergency failover ranks it last. | Up to one audit interval, ≈ 30 s, after the operator first sees a confirmed primary; immediately on the first poll after an operator start or primary change. | The site stays blocked until it is recloned or the primary is made to contain the set; thread health never clears it. An ordered update of the active site stalls, unfenced, until then. See Failover → Divergent replicas. |
| DNS provider down / slow | DNSEndpoint CR updates succeed, but application DNS lookups still return the old A-record. | Operator writes the new DNSEndpoint on failover and moves on — DNS propagation is external-dns's job, not the operator's. | CR update happens within the failover sequence (a few seconds). External-dns reconcile time and your DNS provider's TTL decide the app-visible delay. | The operator cannot accelerate DNS propagation. Applications should use short TTLs (spec.dns.ttlSeconds, default 60 s) and not cache MySQL connection addresses longer than the TTL. Monitor external-dns independently; a stuck external-dns is an outage for writes even after the operator has "finished". |
| S3 (or backup storage) unreachable | Backup Jobs fail; bloodraven_archiver_upload_failures climbs; bloodraven_archiver_backlog_files > 0; PITR lastUploadAt stops advancing. | Backup reconciler retries per spec.backup.retry. PITR archiver retries on every scan (archivePollInterval, default 60 s). | Retry backoffs are bounded by the retry spec; next archival attempt is one poll away. | No data-plane impact — MySQL keeps serving writes. PITR RPO silently drifts as unarchived binlogs accumulate. Alert on bloodraven_archiver_backlog_files > 0 sustained. See Monitoring. |
| PITR archive store cannot be initialized (unreadable passphrase or S3 credentials file, invalid endpoint, empty PVC mount path, PVC mount path that is not a mount point because the archive volume is missing from the pod, or not writable) | Sidecar logs failed to init archive store per attempt; /archiver/status reports enabled:true, storeReady:false, lastError: archive store init: …; bloodraven_archiver_upload_failures rises per attempt. Operator sets PITRArchiveReady=False reason ArchiveStoreInitFailed and emits one Warning PITRArchiveNotReady Event. | Sidecar keeps serving readiness, the startup safety net and self-fencing, and retries the store with jittered backoff from 5 s up to 5 min. Operator only reports; no failover or rollout gate. | Condition on the first poll that sees the failure (≤ 30 s). Recovery to True takes 2 clean poll rounds after the store comes up. An archiver error (ArchiverError) needs 2 consecutive failed archiver scans. | No data-plane impact: the pod stays Ready and in every Service. No binlog archiving on the affected site until it is fixed, so PITR RPO drifts there. An older sidecar image exits instead, crash-loops and drops the pod from Services; the operator shows that as SidecarUnreachable 5 min after an operator restart, since that sidecar has not answered yet (3 failed polls over ≥ 60 s once a sidecar has answered). The ordered update stops on a rolled site whose pod is not Ready, which catches that older sidecar's crash loop, but a current sidecar's pod stays Ready through an archive failure, so the update does not stop a bad PITR profile edit; watch PITRArchiveReady during the rollout. |
| Ordered update rolls a site whose new pod is not Ready (a sidecar that cannot start, an image that does not pull, an unschedulable pod) | The rolled pod stays NotReady while its MySQL may still answer through the -internal Service. Updating=True reason OrderedUpdateStalled (the update is still in progress), Warning Event OrderedUpdateStalled, log ordered update: stalled; site pod not Ready, each naming the site, pod and cause. A crash-looping container reads crash looping, last exit code … in every phase of the loop, so the message and Event do not change from retry to retry. | Stops the update: rolls no other site, does not hand off the active site, leaves the remaining drift queued, and retries every 30 s. A site that already runs the new template and is not Ready blocks the same way, including one an earlier attempt rolled. | The rolled pod gets up to 5 min to roll out and up to 5 min to be Ready continuously for 30 s, counted from its last transition to Ready and its newest container start; a site already on the template is checked by the same rule before every roll and handoff. | The stalled site is out of its Services and, with a broken sidecar, has no self-fencing until it is fixed; the active site keeps serving. The operator does not roll the site back. Fix the cause or the spec; a spec change that repairs the template rolls the broken site again. See Failover → Ordered updates. |
PITR archive identity refused (a legacy prefix, another group's or a copied descriptor, version 2 manifests without archive.json, an encryption or key that does not match the descriptor, a newer format, a damaged archive.json) | /archiver/status reports archive.state: Refused with the reason; lastError starts with the token (PITRArchiveLegacy: …). | Operator sets PITRArchiveReady=False with the reason (ArchiveLegacy, ArchiveForeign, ArchiveCopy, ArchiveDescriptorMissing, ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported) and status.pitr.archive.state: Refused; one Warning PITRArchiveNotReady. The refusing site uploads nothing. | First poll after the sidecar refuses (≤ 30 s). | No data-plane impact. No archiving on the refusing site until the profile points at an archive this group owns; PITR RPO drifts there. Most cases need a new prefix and a full backup; DescriptorMissing can also be fixed by restoring archive.json. ArchiveVersionUnsupported keeps the upgrade remedy only for a newer descriptor version or an unknown encryption mode; for a damaged archive.json (undecodable, trailing data, wrong kind, a malformed field, source names outside Kubernetes bounds, over 64 KiB) the message says archive.json is damaged …: restore the original object (bucket versioning, a copy) or use a new prefix. See Archive identity. |
| Archive store lacks conditional writes | The sidecar's atomic-create probe sees a second create succeed, or the header rejected. | PITRArchiveReady=False / ArchiveStoreUnsupported. A probe that only sees 409 conflicts is inconclusive and reports ArchiveStoreInitFailed, retrying next cycle. | First scan with no descriptor. | No data-plane impact. No archive is created. Use a store that enforces If-None-Match: *; see S3 backups. |
History-integrity violation (replication or binlog filter, binlog_format≠ROW, binlog_row_image≠FULL, commit order off), made out of band (SET GLOBAL, a hand-edited my.cnf) | Sidecar logs WARN archive history unsafe and persists historyIntegrity in its manifest. | PITRArchiveReady=False / ArchiveUnsafeHistory and an edge-triggered Warning PITRArchiveUnsafeHistoryDetected. | Store init or the next audit cycle. | No data-plane impact. The marked site stops archiving permanently for that prefix, and every restore target that needs its bytes is lost, including targets before the violation. Undo the change, then new prefix plus full backup; the new prefix restores only later targets. See Unsafe history marker. |
| Promotion of a site with divergent or quarantined archive rows | The active site changes to a site whose last /archiver/status reports divergentRows + quarantinedRows > 0, or a site is promoted after a NoPrimary interval on a group that has failed over before. | Warning PITRArchiveReissueRiskDetected. No other action. | On the promotion. | If the site was recloned after a lossy failover, it reissues the lost GTIDs and every restore from the archive refuses PITRArchiveHistoryReset. Start a new prefix and take a full backup. See Reissue risk after a reclone. |
PITR-unsafe option in spec.mysqlConf (while PITR is enabled) | The operator renders my.cnf for a site. | The option is left out of my.cnf (MySQL's safe default applies); PITRMysqlConfAccepted=False / PITRUnsafeSettingRefused lists <site>: <field> sets <key>[=<value>], and a Warning PITRMysqlConfRefused fires once per refused set. | Next reconcile. | No data-plane impact beyond the setting not applying; no marker is set. Remove the key, or disable PITR if it is required. See Operator-owned MySQL settings. |
| Backup stamp cannot be written or finished | The backup-stamp init container, dump.py or backup-encrypt-upload exits with a termination message starting backup stamp: ; logs write backup stamp, finish backup stamp, check backup stamp, read dumped server clock or check dumped server. | The backup fails like any other backup Job failure and is retried per spec.backup.retry. | Immediately. | No data-plane impact. No new backup until it is fixed. check dumped server means the Job reached a server whose server_id is not the source site's: check the Service the backup uses. |
Ordered update handoff: the standby's read_only=OFF returns an error (the statement may have applied) | The update ends ordered update failed; the error ends (source left fenced: the candidate may already accept writes). WARN ordered update: handoff promotion returned an error but the standby is writable; recording the handoff, or ordered update: could not read the standby's read_only after a failed handoff promotion. | Leaves the old active fenced and does not roll it. Reads the standby's read_only on a pooled connection, up to 3 tries. A writable standby is recorded as the promotion it is (the lastFailover record, the anti-flap cooldown and DNS) before the error is reported. Without that evidence nothing is recorded. | Immediate. | An unreadable standby is left to the next topology poll, which observes it. The handoff had already proven the standby holds the fenced set, so recording it loses nothing. |
Sidecar /archiver/status unreachable | Operator polls fail; archiver gauges for the site disappear. | PITRArchiveReady=False reason SidecarUnreachable after 3 consecutive failed polls spanning ≥ 60 s, or ≥ 5 min for a sidecar that has not answered since the operator started (new group, bootstrap, crash-looping older sidecar); one missed poll changes nothing. status.pitr keeps the last reported coverage and the message says it is last-known. | ≥ 60 s (≥ 5 min before the first answer). | Check the sidecar container (restarts, kubectl logs --previous) and the -internal Service. |
| Anti-flap cooldown blocks needed failover | Second failover condition arises within failoverCooldown of the previous one. | Operator logs failover blocked by anti-flap cooldown and does not fail over. | No automatic action until the cooldown expires (default 5 min). | The planned-failover API also honours this cooldown and will be rejected with reason: CooldownActive. If you genuinely need a second failover inside the cooldown window, perform a manual promotion — that path bypasses the cooldown explicitly. |
| Primary dies while a replica is mid-keyring-rotation | status.encryptionAtRest.sites[].unsealReason=Rotation on the would-be target. Degraded=True reason NoPrimary with a message naming UnsealReason=Rotation. Warning Event KeyringPromotionRefused (or KeyringPromotionSkipped if another sealed candidate was promoted). Counter bloodraven_keyring_promotions_blocked_total{outcome="refused|skipped"}. | Refuses to promote the rotating site. If another sealed PrimaryCandidate exists, that site is promoted instead. If the rotating site is the only remaining candidate, the group stays without a writable primary until the site returns to Sealed. Planned failover to that target is rejected with reason: KeyringRotation. An ordered update re-checks rotation when it picks the handoff standby and never hands the active site to a rotating one; with no other eligible standby it ends with ordered update failed before fencing anything. | Immediate on the failover decision (same debounce as any primary death). Healing after the replica seals is the next poll. | Deliberate availability cost: promoting a mid-rotation replica would make it the sole authoritative copy while its new master key is not yet sealed, so a pod loss in that window is unrecoverable. There is no cancel-rotation command. Finish the rotation (repair escrow/sidecar if it is stalled), then retry. See Rotating keys. |
| Planned failover target fails to catch up | Admin annotated bloodraven.shipstream.io/planned-failover=<site>; target's GTID_EXECUTED does not cover the fenced set (status.plannedFailover.fence.gtidSet) within maxLagWait. | Writes the rollback intent (phase: RollingBack, rollback.reason: LagTimeout, PlannedFailoverRollbackStarted Event), unfences the source on a pinned session, and stamps phase: Failed with result.outcome: RolledBack, result.reason: LagTimeout (PlannedFailoverFailed Event). The role=primary label is restored; no DNS flip occurred. If another site turned writable meanwhile, the source stays fenced (SourceLeftFenced). | maxLagWait (default 5 min) plus at most 30 s of rollback. | No data loss; cluster returns to pre-annotation state. Retry after replication has caught up, or use emergency failover if the source is truly gone. See Planned failover → Rollback. |
DisasterRecovery or PrimaryCandidate peer down when a planned failover is requested | status.plannedFailover.phase: Admitting with admission.waitReason: PeerUnreadable; PlannedFailoverAdmissionDelayed Event naming the site. | Waits without fencing anything; ends phase: Failed, result.outcome: NotStarted, result.reason: AdmissionTimeout. A peer lost after the fence stops the run at the pre-write gate instead: rolled back, or the source left fenced if another site is writable. | 2 min admission deadline. | Deliberate: a planned failover is optional and requires every promotable and DR site to answer read-only, because an unreachable site may still be writable. Bring the site back, then re-annotate (a site cannot be removed from spec.sites; see Replacing a site). Readers do not block it. Emergency failover is unaffected. |
| Planned failover cannot explain the target after a possible write | result.outcome: ManualRecoveryRequired; condition AutomaticFailoverHeld=True (reason PlannedFailoverManualRecoveryRequired); Warning Events PlannedFailoverManualRecoveryRequired, then PlannedFailoverHoldPersisted every 10 min. | Holds automation for the group: no emergency failover, split-brain resolution, re-assert, ordered update, rotation, recovery or DNS reconcile until acknowledged. Writes continue on the writable site. | Immediate. | Never picks a winner automatically: split-brain resolution could fence the only holder of acknowledged writes. Resolve by hand, then annotate bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>. See the hold runbook. |
| Status writes fail during a planned failover (RBAC 403, 422, failing webhook, or a write that conflicts on every attempt of a pass) | bloodraven.shipstream.io/planned-failover-stall-since annotation; then bloodraven.shipstream.io/planned-failover-terminal: {"schemaVersion":1,"runID":…,"outcome":…,"reason":…} (optionally with warnings), ERROR planned failover: status stalled; ended the run with a terminal marker, Warning Event PlannedFailoverStatusStalled. | Ends the run in memory without a MySQL rollback (a confirmed promotion is still recorded and published) and releases the claim. A source left fenced is named for the re-assert. Retries the terminal status record every 10 s. | 2 min from the durable stall-since time. | If metadata writes fail too, the claim stays held and only leader election bounds it. A fenced set over 64 KiB cannot use the marker release. |
| Topology manager absent during a planned failover | PlannedFailoverStalledManagerAbsent Warning Event after 30 s; bloodraven.shipstream.io/planned-failover-manager-absent-since annotation. | Waits (no other actor exists to race it), then ends without MySQL access: SourceLeftFenced or PromotedUnconfirmed, reason ManagerUnavailable. A promotion the operator had already confirmed ends Succeeded instead, the terminal write applying its failover record; DNS is left to the next manager. A fenced source is re-asserted once a manager starts, when the fence is attributable and every site is reachable. | 5 min from the durable absent-since time. | See Healing a fenced source for the manual cases. |
| Dragonfly master killed | spec.dragonfly.enabled=true; status.dragonfly.activeSite is unreachable, and one replica is reachable and not syncing/loading. | Promotes the replica with REPLTAKEOVER or fallback promotion, updates Dragonfly role/traffic labels, and leaves MySQL status.activeSite unchanged. | Next Dragonfly manager poll plus promotion budget. | Cache/session continuity is best-effort. If no healthy Dragonfly replica exists, the active Dragonfly Service may have no endpoints until recovery. MySQL availability is unaffected. |
| Dragonfly unavailable during MySQL emergency failover | MySQL primary is failed, and Dragonfly target/source cannot be reached or promoted. | Completes MySQL failover first. Dragonfly promotion is attempted best-effort and may record bloodraven_dragonfly_promotions_total{result="failed"} or {result="sessions_lost"} on the REPLICAOF NO ONE fallback. | MySQL path remains up to ≈ 37 s; Dragonfly attempt is bounded separately. | Sessions/cache may be lost or unavailable. Dragonfly never blocks emergency MySQL recovery. |
| Planned Dragonfly sync timeout | Planned failover enters WaitingForDragonflySync, but the target Dragonfly replica does not reach the source offset within maxSyncWait. | With onSyncTimeout=Proceed (default), promotes MySQL and stamps status.plannedFailover.dragonfly.sessions: Lost with dragonfly.reason: DragonflySyncTimeout. With onSyncTimeout=Fail, rolls back before MySQL promotion. | spec.dragonfly.plannedFailover.maxSyncWait (default 30 s). | Proceed preserves MySQL availability at the cost of cache/session continuity. Fail preserves the old active site and avoids claiming session preservation. |
| Replication credentials missing / wrong | Auto-clone fails; Bootstrapping=False with reason BootstrapFailed, and Degraded=True with reason BootstrapFailed until the site replicates or a new clone starts. | Operator reports the failure via the Bootstrapping and Degraded conditions and events; does not keep retrying indefinitely. The deployment API refuses leases while Degraded is BootstrapFailed. | Immediate (on the first bootstrap attempt). | Without MYSQL_REPLICATION_USER / MYSQL_REPLICATION_PASSWORD the operator account replicates; the first-boot init script creates a dedicated replication user. One that does not exist on the source (the datadir existed before the group, so the init script did not run) or whose password changed only in the Secret fails old-primary recovery and auto-clone. The rest of the operator (polling, failover, DNS) keeps working. An operator Secret with an empty username or password, or with only one of the two replication keys, is refused with a SecretMissingKey Event. |
Backup encryption failure modes
These rows apply only when a backup profile has spec.backup.profiles[].encryption
set. See Backup encryption for the full threat model.
| Failure | Detection path | Operator action | Typical time-to-act | Notes & limits |
|---|---|---|---|---|
| Passphrase Secret missing at restore time | Restore Job init container can't mount spec.initFromBackup.decryption.passphraseSecretKeyRef; kubectl describe pod shows CreateContainerConfigError. | Reconciler records DecryptionSecretMissing on the restore CR; stops creating retry Jobs after backoffLimit. | Seconds (Job pod fails to start). | Recovery: recreate the Secret with the original passphrase value, or delete & re-apply the restore CR. Ciphertext is unrecoverable without the exact passphrase used at backup time — treat the Secret as critical recovery material. |
| Passphrase rotated between backup and restore | A backupRef restore or verification is not affected: it decrypts with the backup's own dump-key snapshot (status.artifactStorage.passphraseSecretKeyRef). Only a decryption Secret, or a direct source.s3/source.pvc source, holding other bytes reaches decrypt-download, which exits non-zero with a chunk-auth failure from backupcrypto. | Job pod's logs show backupcrypto: chunk 0: cipher: message authentication failed; reconciler reports DecryptionFailed. | Immediate on first read. | AES-GCM catches mismatched keys cleanly — there's no silent garbage load. Recovery requires the original passphrase; there is no re-wrap path. A missing recorded snapshot fails PITRBackupDumpKeyMissing instead (restore the Secret from a backup of it, or set decryption). |
| PITR archive passphrase or encryption edited | PITRArchiveBindingReady=False / KeyChangeRefused or EncryptionChangeRefused, Warning PITRArchive…ChangeRefused. The admission rule already refuses selector and encryption edits while PITR is enabled. | None on pods: sidecars keep mounting the Bound binding's key snapshot, including after restarts. | Next reconcile (Secret watch). | No in-place rotation: revert, or move to a new prefix (Rotation). A shared dump + PITR profile then encrypts new dumps with the new bytes. |
| Archive key snapshot deleted or replaced | Snapshot NotFound, or its pitr-accepted keyCheck does not verify its bytes. | Re-creates it under the same name if the profile's live bytes reproduce it (Warning PITRArchiveKeySnapshotRecreated; not when another object of that name appeared first); otherwise Bound/KeyMaterialLost (Warning PITRArchiveKeyMaterialLost). Never deletes a snapshot. | Next reconcile. | MySQL keeps running: the pod mounts the snapshot optional: true, so a restarted sidecar's archiver fails closed (PITRArchiveReady not True) instead of the pod failing to start. Restore the Secret from a backup of it; for a snapshot that is present but fails the check, never delete it first (it may hold the only copy of the key): restore its labels, annotations or bytes from a backup of it. |
| Archive binding refused (wrong key, foreign, copy, legacy, missing) | pitr-archive-check Job result or a correlated sidecar refusal; PITRArchiveBindingReady=False/<reason>, Warning PITRArchiveBindingRefused. | Write target: the group keeps its last accepted render (the old archive keeps advancing). Read target: restore Pending or verification waiting on fixable reasons; Failed PITRArchiveBindingNotReady on ArchiveNotFound/DescriptorMissing/Legacy/VersionUnsupported. | Check Job: ≤ 120 s; retried when the input changes. | Pre-binding Deployments are not updated (nor their ConfigMaps) until a check verifies, while any site Deployment lacks a decodable render annotation (message names those). A check that cannot list the prefix is not a refusal: it stays Pending/Authenticating and is retried every minute. |
status.pitrArchiveBindings lost | Entries missing on the next reconcile. | Rebuilds every accepted encrypted binding from its snapshot's acceptance annotation after an uncached read confirms the snapshot (exists, not terminating) and verifying keyCheck; no Job, no roll. Plaintext bindings are re-proven by the sidecar report or a check Job. | Next reconcile. | More accepted snapshots than 16 slots → BindingLimitReached naming the rest. |
| Group deleted | — | Archive and dump key snapshots survive (no owner reference). | — | Release bindings before deleting, or delete by bloodraven.shipstream.io/failover-group-uid label afterwards. |
| Ciphertext truncated by S3 multipart abort | decrypt-download returns backupcrypto: ciphertext truncated before final chunk (ErrTruncated). | Job pod fails fast; reconciler reports BackupCiphertextTruncated. | Immediate. | ErrTruncated distinguishes "tail missing" from a generic auth failure. Recover by re-running the source backup and restoring from the replacement artifact. |
| Object has no BRV1 magic (tamper / downgrade) | decrypt-download returns sidecar: object missing BRV1 magic while encryption is required (ErrTamperedOrDowngrade). | Job pod fails fast; reconciler reports BackupDecryptionRejected. | Immediate. | Defends against an attacker with bucket write access overwriting ciphertext with attacker-chosen plaintext. Manual compatibility flags are BLOODRAVEN_PITR_ALLOW_PLAINTEXT_FALLBACK=1 for the sidecar and BLOODRAVEN_ALLOW_PLAINTEXT_FALLBACK=1 for decrypt-download. The operator renders neither flag; fallback weakens downgrade protection and is for time-bounded migration only. |
How to read this table for your on-call
Two common uses:
- "What's the operator doing right now?" Find the row matching the symptom, read the Operator action and time-to-act cells to set your clock for when the situation should resolve on its own. If the expected time has passed and nothing has changed, check the Operator limitations column.
- "Do I need to page a human?" Alert on the rows whose Operator action is "none automatic" (cross-site partition, total loss, split brain, DNS provider, S3). These are the cases where the operator has deliberately deferred to a human.
Related reading
- Failover → State machine — the per-site and cross-site truth tables that drive every row's Operator action cell.
- Failover → Failover sequence — the step-by-step sequence behind the first-isolated-fault "up to ≈ 37 s" bound.
- Operations — runbooks for the rows where the operator action is "none automatic".
- Monitoring — the metrics and events each row's Observable signal column refers to.
- Production hardening — which rows here the hardening checklist is specifically guarding against.