Runbooks

Every runbook starts with the same incident header. Fill it in before making changes.

If spec.dragonfly.enabled=true, include status.dragonfly.phase, active Dragonfly Service endpoints, and a Redis-compatible cache/session read/write check in the same incident record.

Planned failover

Use Planned Failover for controlled maintenance. Do not use during split-brain or when the target replica is unhealthy.

kubectl annotate mysqlfailovergroup orders -n orders \
  bloodraven.shipstream.io/planned-failover=pdx --overwrite
kubectl get mysqlfailovergroup orders -n orders -o wide

Every PrimaryCandidate and DisasterRecovery site must be up and answering: a planned failover waits (admission.waitReason: PeerUnreadable) and then ends NotStarted with AdmissionTimeout while one is down. Bring the site back first (a site cannot be removed from spec.sites; to drop one, see Replacing a site).

Wait for the run to finish, then check its outcome, not only the phase:

kubectl get mysqlfailovergroup orders -n orders \
  -o jsonpath='{.status.plannedFailover.phase}{" "}{.status.plannedFailover.result.outcome}{" "}{.status.plannedFailover.result.reason}{" "}{.status.plannedFailover.result.warnings}{"\n"}'

kubectl bloodraven promote orders pdx --wait does the same and also reports a run the operator ended with the bloodraven.shipstream.io/planned-failover-terminal annotation while status writes were failing (status may still read Promoting); kubectl bloodraven status orders prints Ended by terminal marker: <outcome>:<reason> for it. Treat the marker's outcome like result.outcome below.

Verify DNS and application writes before starting maintenance on the old active site.

If Dragonfly is enabled, also verify status.plannedFailover.dragonfly.sessions (Preserved or Lost) and confirm the active Dragonfly Service points at the new site.

kubectl get mysqlfailovergroup orders -n orders \
  -o jsonpath='{.status.plannedFailover.dragonfly}{"\n"}'
kubectl get endpoints orders-dragonfly -n orders

If the run did not succeed, result.outcome decides what to do:

Planned failover left the source fenced

Use when status.plannedFailover.result.outcome is SourceLeftFenced. Nothing was written to the target; the source is read-only.

1. Read the reason, the warnings and the message:

   kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.plannedFailover.result}{"\n"}{.status.plannedFailover.message}{"\n"}{.status.lastFailover.target}{"\n"}'
   

2. Check every site's state: SELECT @@GLOBAL.read_only, @@GLOBAL.super_read_only, @@GLOBAL.gtid_executed;.
3. If another site is writable, that site is the primary. Do not unfence the source; old-primary recovery repoints it as a replica.
4. If no site is writable, decide by the healing table:
   • SourceLost, or a peer unreachable: emergency failover decides. Restore connectivity.
   • An attributable fence (no FenceNotAttributable warning; reasons such as StatusStalled, ManagerUnavailable or a rollback that could not confirm the unfence), every site reachable, and status.lastFailover.target empty or the source: the topology manager re-asserts the source on its own within failoverCooldown (WARN re-asserting fenced promoted primary). Wait for it.
   • FenceNotAttributable (also on a run that sent its fence and then ended because authority moved), LegacyRunRefused, UnknownSite (the source or target was removed from spec.sites mid-run, so nothing was named), AuthorityChanged before the fence was applied, or status.lastFailover.target naming another site: manual. A FenceNotAttributable or LegacyRunRefused message ends with this advice (check which site is writable; if none is, unfence the site with the most complete GTID set by hand (confirm no other site is writable first); re-annotate only once a site is writable). Pick the site with the most complete GTID set, confirm no other site is writable, and unfence it (SET GLOBAL super_read_only=OFF; SET GLOBAL read_only=OFF;). Do not re-annotate a planned failover instead: with no writable site, admission waits (AuthorityUnsettled) and ends NotStarted{AdmissionTimeout} after 2 minutes, and writes stay refused all along. Request a planned failover only after one site is writable.
5. Verify status.activeSite, DNS and an application write.

While the run's own attributable fence has not healed (the source is not yet observed writable and no failover was recorded since the run started), a new planned-failover request is refused with a PlannedFailoverRejected Warning (... left source "<site>" fenced and the fence has not healed yet; ...) and status.plannedFailover keeps the fenced run: let the re-assert finish, or unfence by hand, then re-annotate.

Planned failover automation hold

Use when the AutomaticFailoverHeld condition is True (status.plannedFailoverHold is set), you see PlannedFailoverManualRecoveryRequired or PlannedFailoverHoldPersisted Events, or a bootstrap, reclone or keyring rotation is refused with automatic failover is held after planned failover <runID>; resolve the split brain, then annotate bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID> (a rotation adds ; retrying after the acknowledgement). A planned failover may have written to its target and ended ManualRecoveryRequired. Emergency failover, split-brain resolution, the re-assert, ordered updates, rotations, recovery and DNS reconciliation are held for the group until you acknowledge the hold. Dragonfly is held as well: a failed Dragonfly master is not replaced, replicas are not re-attached, and Dragonfly rollouts and snapshot upgrades do not promote, so a Dragonfly failure during the hold is an outage of the cache until you acknowledge. Writes continue on whichever site is writable, and backups, backup schedules, verification and keyring work keep running.

1. Read the held run and the sites it names:

   kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.plannedFailoverHold}{"\n"}{.status.plannedFailover.runID}{" "}{.status.plannedFailover.result}{"\n"}{.metadata.annotations.bloodraven\.shipstream\.io/planned-failover-terminal}{"\n"}'
   

   
   status.plannedFailoverHold names the run (runID), its reason, sourceSite, targetSite and the ambiguousSites: the target, the source, and every site the failing step found writable or could not read. A hold rebuilt from the terminal marker by a process that did not see the run lists every site in spec.sites. When status writes had stalled, a terminal-marker annotation {"schemaVersion":1,"runID":"<runID>","outcome":"ManualRecoveryRequired","reason":"<reason>"} (optionally with a warnings array) carries the same hold instead.
2. On every site read @@GLOBAL.read_only, @@GLOBAL.super_read_only, @@GLOBAL.gtid_executed and SHOW REPLICA STATUS. Compare the source and the target with GTID_SUBTRACT both ways.
3. Resolve the split brain by hand. Decide which site is the primary from the data, not from history: the site holding the transactions you must keep. Fence every other site (SET GLOBAL super_read_only=ON;). If both sites took writes, preserve the losing site's extra transactions (mysqlbinlog --include-gtids=<extra set>) before anything reclones it.
4. Make sure exactly one PrimaryCandidate is writable and DNS points at it. The topology manager does not move DNS while the hold stands.
5. Acknowledge the hold with the run's id. The plugin reads it from the group:

   kubectl bloodraven ack-planned-failover-hold orders -n orders
   # or, by hand:
   kubectl annotate mysqlfailovergroup orders -n orders \
     bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>
   

6. Verify: status.plannedFailoverHold disappears, the condition turns False with reason PlannedFailoverHoldAcknowledged, the annotation is removed by the operator, the Normal Event PlannedFailoverHoldAcknowledged is emitted, and the log shows planned failover: hold acknowledged then planned failover: claim released with why=ack. Automation then resumes from the state you left: old-primary recovery repoints the fenced site, or reports divergence for you to reclone.

An acknowledgement naming another run is ignored (PlannedFailoverHoldAcknowledgmentIgnored); correct the value. One that finds no standing hold, or the claim held for another run, is ignored the same way and removed; nothing is resumed. A True condition without status.plannedFailoverHold (a hand-edited status, or a CRD rolled back across the planned-failover claim release, which prunes the hold record but keeps the condition) names no run and cannot be acknowledged: after resolving the split brain, remove the condition by hand (kubectl patch mysqlfailovergroup orders -n orders --subresource=status --type=json -p '[{"op":"remove","path":"/status/conditions/<index>"}]'). The operator then releases the hold on its next pass, without a restart: the log shows planned failover: released an automation hold no durable record backs and planned failover: claim released with why=orphanHold. Until then such a refusal reads automatic failover is held by a True AutomaticFailoverHeld condition that names no run; resolve the split brain, then remove the condition. Do not acknowledge before step 4: the next poll would act on whatever state the sites are in.

Dragonfly degraded

Use when status.dragonfly.phase=Degraded, bloodraven_dragonfly_site_up==0, Dragonfly promotion failed, status.dragonfly.replTakeoverSupported=false, or you see DragonflySessionsLost / DragonflyReplTakeoverUnsupported. MySQL is still authoritative; this runbook protects cache/session continuity.

1. Confirm MySQL status.activeSite and application writes are healthy.
2. Inspect status.dragonfly.replTakeoverSupported and replTakeoverProbeMessage. A false value means the running image does not advertise REPLTAKEOVER — pin v1.38.0+ (or a private build that has the command) and roll the StatefulSets.
3. Inspect status.dragonfly.sites[] for sites whose role is Unreachable or StaleMaster, and for syncing or link-down sites.
4. Check the active Dragonfly Service endpoints and pod role/traffic labels.
5. Restart or reschedule only the affected Dragonfly pod if the stale site has not accepted writes.
6. If a stale master may have accepted writes, keep it out of the active Service and decide whether cache/session data can be discarded.

kubectl get mysqlfailovergroup orders -n orders \
  -o jsonpath='{.status.dragonfly}{"\n"}'
kubectl get pod -n orders -l app.kubernetes.io/name=dragonfly --show-labels
kubectl get endpoints orders-dragonfly -n orders

Emergency manual promotion

Use only when automation cannot complete and you have confirmed the old primary cannot accept writes.

1. Preserve Events and logs.
2. Confirm old primary is unreachable or fenced.
3. Confirm candidate has the best GTID/replication position available.
4. Promote using your approved MySQL operational procedure.
5. Update or unblock Bloodraven reconciliation only after state is consistent.

Verification:

kubectl get mysqlfailovergroup orders -n orders -o yaml
dig orders.az.example.com

Split-brain recovery

Use when more than one site can accept writes.

1. Stop application writes if possible.
2. Identify all writable sites.
3. Choose the authoritative site using business ownership and GTID evidence.
4. Fence losing sites by ensuring super_read_only=ON and blocking application traffic.
5. Preserve divergent data for manual reconciliation.
6. Reclone losing sites only after data owners approve.

Do not skip GTIDs to make replication look green.

Divergent old primary recovery

Use when the old primary contains transactions absent from the promoted site.

1. Export evidence from the old primary.
2. Ask the data owner whether divergent rows must be recovered manually.
3. Keep the old primary fenced.
4. Reclone from the active primary when approved.
5. Verify replication returns to healthy.

kubectl describe mysqlfailovergroup orders -n orders
kubectl get events -n orders --sort-by=.lastTimestamp | grep -i divergent

Divergent replica

Use when Degraded=True with reason DivergentTransactions names a site with divergenceDetectedBy=ReplicaAudit, or an ordered update or planned failover refuses a divergent target. The replica is usually replicating normally; its extra transactions are history the primary never had.

1. Read the set: kubectl get mysqlfailovergroup orders -n orders -o jsonpath='{.status.sites[?(@.name=="<site>")].divergentGtid}'.
2. Decode and keep the errant events on the replica: mysqlbinlog --include-gtids='<divergentGtid>' <binlog files> > errant.sql.
3. Ask the data owner whether the transactions must be kept.
4. Discard them with kubectl bloodraven reclone orders <site> -n orders, or keep them by replaying the decoded transactions on the primary. To accept the replica's divergence instead, commit an empty transaction per GTID on the primary (SET GTID_NEXT='<uuid>:<n>'; BEGIN; COMMIT;, then SET GTID_NEXT='AUTOMATIC';). That makes the GTID sets agree without moving rows: the replica keeps rows the primary lacks, becomes promotable again once the audit clears, and a later promotion makes those rows authoritative with no DivergentPromotion record. Use it only when the transactions' effects are absent, harmless, or already reconciled.
5. Within about 30s the audit logs divergence cleared: replica holds no transactions the primary lacks, Degraded clears, and a stalled ordered update resumes on its own.

See Divergent replicas.

PVC loss recovery

Use when a MySQL data PVC is deleted or corrupted.

1. Confirm whether the affected site is active.
2. If active, stop application writes and follow emergency recovery policy.
3. If standby, reclone from the active primary.
4. If both sites are lost, restore from backup.

See Backup And Restore for restore workflows.

Total site loss

Use when all nodes/storage/network for a site are gone.

1. Confirm surviving site is writable and healthy.
2. Confirm DNS points to the surviving site.
3. Disable app scheduling to the lost site.
4. Rebuild site infrastructure.
5. Rejoin via clone or restore.

Expected impact is bounded by failover detection, relay-log drain, DNS TTL, and application reconnect behavior.

Keyring not sealed

Use when bloodraven_keyring_phase{phase="sealed"} is 0 for a site, or status.conditions[type=EncryptionAtRestReady] is False.

A site that is not sealed is running with a writable keyring. Its data is still encrypted, but the operator cannot yet prove the key is durably escrowed.

1. Find out what it is waiting for — message carries the reason:

   kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'
   

2. Unsealed with unsealReason: Clone or Bootstrap is normal and transient. Wait for the clone to finish.
3. Unsealed with "waiting for the sidecar to escrow" — check the sidecar can reach the operator:

   kubectl exec -n orders deploy/mysql-orders-iad -c sidecar -- \
     wget -qO- localhost:8080/keyring/status
   

   
   lastError on that payload names the failure. Common causes: the operator's auxiliary Service is unreachable, or the escrow token Secret was deleted (the operator re-mints it on the next reconcile, but the pod needs to roll to pick it up).
4. Escrowed that never becomes Sealed — the pod has not rolled onto the sealed rendering. Check for a stuck ordered update (status.updatePhase).
5. Failed — read the message and follow Keyring escrow lost if it names a missing or corrupt Secret.

Full background: Data-at-rest encryption.

Keyring escrow lost

Use on a KeyringEscrowMissing or KeyringEscrowCorrupt Event.

1. Confirm the pod is still running:

   kubectl get pod -n orders -l bloodraven.shipstream.io/site=iad
   

2. If it is running, recreate the exact missing Secret from the keyring still mounted in the running pod. Do not force an unseal or rollout: that would destroy the only surviving in-memory copy before it was durable again.

   set -o pipefail
   # The site's MySQL pod, selected as its Deployment selects it (Dragonfly
   # and backup pods share the site label). Expect exactly one name.
   POD=$(kubectl get pod -n orders \
     -l app.kubernetes.io/name=mysql,app.kubernetes.io/instance=orders,bloodraven.shipstream.io/site=iad \
     --field-selector=status.phase=Running -o jsonpath='{.items[*].metadata.name}')
   [[ "$POD" != *" "* && -n "$POD" ]] || { echo "expected one MySQL pod, got: '$POD'" >&2; exit 1; }
   PHASE=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].phase}|{.status.encryptionAtRest.sites[?(@.name=="iad")].unsealReason}')
   # Only a sealed site whose escrow was lost: a rotation, bootstrap or
   # clone in progress owns its Secret.
   [[ "$PHASE" == "Failed|" || "$PHASE" == "Sealed|" ]] || { echo "site is $PHASE; not an escrow loss" >&2; exit 1; }
   SECRET=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringSecret}')
   VERSION=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringVersion}')
   DIGEST=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringDigest}')
   KEYRING_DIR=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.spec.encryptionAtRest.keyring.dataFileDir}')
   KEYRING_DIR=${KEYRING_DIR:-/run/mysql-keyring}
   FG_UID=$(kubectl get mysqlfailovergroup orders -n orders -o jsonpath='{.metadata.uid}')
   
   if kubectl get secret "$SECRET" -n orders -o name >/dev/null 2>&1; then
     echo "Secret $SECRET exists (KeyringEscrowCorrupt?): not touching it; see below" >&2
     exit 1
   fi
   KEYRING_FILE=$(mktemp)
   kubectl exec -n orders "$POD" -c mysql -- cat "$KEYRING_DIR/keyring" >"$KEYRING_FILE" || exit 1
   ACTUAL_DIGEST="sha256:$(sha256sum "$KEYRING_FILE" | cut -d' ' -f1)"  # macOS: shasum -a 256
   if [[ "$ACTUAL_DIGEST" != "$DIGEST" ]]; then
     echo "the running keyring does not match status; nothing was written" >&2
     exit 1
   fi
   kubectl create secret generic "$SECRET" -n orders --from-file=keyring="$KEYRING_FILE" || exit 1
   rm -f "$KEYRING_FILE"
   kubectl label secret "$SECRET" -n orders \
     app.kubernetes.io/name=mysql-keyring \
     app.kubernetes.io/managed-by=bloodraven \
     bloodraven.shipstream.io/failover-group=orders bloodraven.shipstream.io/site=iad \
     bloodraven.shipstream.io/keyring-version="$VERSION"
   kubectl annotate secret "$SECRET" -n orders bloodraven.shipstream.io/keyring-digest="$DIGEST"
   kubectl patch secret "$SECRET" -n orders --type=merge -p="{
     \"metadata\":{\"ownerReferences\":[{
       \"apiVersion\":\"shipstream.io/v1alpha1\",\"kind\":\"MysqlFailoverGroup\",
       \"name\":\"orders\",\"uid\":\"${FG_UID}\",\"controller\":true,\"blockOwnerDeletion\":true
     }]},\"immutable\":true
   }"
   

   
   Do not set the phase back to Sealed by hand. The operator does it once the Secret hashes to the recorded digest and MySQL confirms it is running that keyring (Event KeyringRecovered); a hand-written status could overwrite a rotation started meanwhile. Verify that the site returns to Sealed without restarting the pod.
   The doctor helper .agents/skills/bloodraven-doctor/scripts/keyring-escrow-recover.sh -n orders orders iad does the same and also stops if the site's state changes while it runs.
   The block checks the running keyring against the recorded digest before it writes anything and never deletes a Secret. On KeyringEscrowCorrupt the Secret still exists, so it refuses. Replacing it is a deliberate, separate step: back the existing Secret up (kubectl get secret "$SECRET" -n orders -o yaml > escrow-backup.yaml), delete exactly that Secret, and run the block again; it re-verifies the running keyring first.
3. If the pod is already gone, that site's data is unrecoverable. Delete the unreadable PVC, then re-clone it from a healthy peer. The confirmation token is mandatory because this permanently discards the target site's local data:

   kubectl delete pvc mysql-orders-iad-data -n orders
   kubectl bloodraven reclone orders iad --cold -n orders
   # or by hand, with the group's uid:
   kubectl annotate mysqlfailovergroup orders -n orders \
     bloodraven.shipstream.io/reclone-site="iad:confirm=orders:uid=$(kubectl get mfg orders -n orders -o jsonpath='{.metadata.uid}')"
   

4. If every site lost its escrow, restore from backup. Backup artifacts are logical dumps protected by a separate key, so they do not depend on any site's keyring.

Network partition diagnosis

Use Network Partitions. Prioritize preventing two writable sites.

kubectl get mysqlfailovergroup orders -n orders -o yaml
kubectl get endpoints,networkpolicy -n orders
kubectl get events -n orders --sort-by=.lastTimestamp

Failed backup

1. Keep the last successful backup.
2. Inspect MysqlBackup status and Job logs.
3. Fix credentials, storage, or MySQL source health.
4. Trigger a manual backup.
5. Verify the new backup.

kubectl get mysqlbackup -n orders
kubectl describe mysqlbackup -n orders <backup-name>

Failed restore

1. Confirm this is a recovery environment or approved destructive action.
2. Inspect restore Job logs.
3. Validate source path and decryption Secret.
4. Recreate the recovery failover group if bootstrap partially initialized MySQL.

Operator unavailable

Use Operator Availability. Sidecar self-fencing is part of the safety model; do not disable it during diagnosis.

kubectl rollout status deployment/bloodraven -n bloodraven
kubectl logs -n bloodraven deploy/bloodraven
kubectl get lease -A | grep bloodraven

DNS failover stuck

1. Confirm status.activeSite changed.
2. Inspect DNSEndpoint and external-dns logs.
3. Confirm provider record and TTL.
4. If applications still write old site, fence the old site and restart affected clients.

kubectl get dnsendpoint -A | grep orders
kubectl logs -n external-dns deploy/external-dns
dig orders.az.example.com