Runbooks

Every runbook starts with the same incident header. Fill it in before making changes.

If spec.dragonfly.enabled=true, include status.dragonfly.phase, active Dragonfly Service endpoints, and a Redis-compatible cache/session read/write check in the same incident record.

Planned failover

Use Planned Failover for controlled maintenance. Do not use during split-brain or when the target replica is unhealthy.

kubectl annotate mysqlfailovergroup orders -n orders \
  bloodraven.shipstream.io/planned-failover=pdx --overwrite
kubectl get mysqlfailovergroup orders -n orders -o wide

Verify DNS and application writes before starting maintenance on the old active site.

If Dragonfly is enabled, also verify status.plannedFailover.dragonfly.sessionsPreserved and confirm the active Dragonfly Service points at the new site.

kubectl get mysqlfailovergroup orders -n orders \
  -o jsonpath='{.status.plannedFailover.dragonfly}{"\n"}'
kubectl get endpoints orders-dragonfly -n orders

Dragonfly degraded

Use when status.dragonfly.phase=Degraded, bloodraven_dragonfly_site_up==0, or Dragonfly promotion failed. MySQL is still authoritative; this runbook protects cache/session continuity.

1. Confirm MySQL status.activeSite and application writes are healthy.
2. Inspect status.dragonfly.sites[] for unreachable, stale-master, syncing, or link-down sites.
3. Check the active Dragonfly Service endpoints and pod role/traffic labels.
4. Restart or reschedule only the affected Dragonfly pod if the stale site has not accepted writes.
5. If a stale master may have accepted writes, keep it out of the active Service and decide whether cache/session data can be discarded.

kubectl get mysqlfailovergroup orders -n orders \
  -o jsonpath='{.status.dragonfly}{"\n"}'
kubectl get pod -n orders -l app.kubernetes.io/name=dragonfly --show-labels
kubectl get endpoints orders-dragonfly -n orders

Emergency manual promotion

Use only when automation cannot complete and you have confirmed the old primary cannot accept writes.

1. Preserve Events and logs.
2. Confirm old primary is unreachable or fenced.
3. Confirm candidate has the best GTID/replication position available.
4. Promote using your approved MySQL operational procedure.
5. Update or unblock Bloodraven reconciliation only after state is consistent.

Verification:

kubectl get mysqlfailovergroup orders -n orders -o yaml
dig orders.az.example.com

Split-brain recovery

Use when more than one site can accept writes.

1. Stop application writes if possible.
2. Identify all writable sites.
3. Choose the authoritative site using business ownership and GTID evidence.
4. Fence losing sites by ensuring super_read_only=ON and blocking application traffic.
5. Preserve divergent data for manual reconciliation.
6. Reclone losing sites only after data owners approve.

Do not skip GTIDs to make replication look green.

Divergent old primary recovery

Use when the old primary contains transactions absent from the promoted site.

1. Export evidence from the old primary.
2. Ask the data owner whether divergent rows must be recovered manually.
3. Keep the old primary fenced.
4. Reclone from the active primary when approved.
5. Verify replication returns to healthy.

kubectl describe mysqlfailovergroup orders -n orders
kubectl get events -n orders --sort-by=.lastTimestamp | grep -i divergent

PVC loss recovery

Use when a MySQL data PVC is deleted or corrupted.

1. Confirm whether the affected site is active.
2. If active, stop application writes and follow emergency recovery policy.
3. If standby, reclone from the active primary.
4. If both sites are lost, restore from backup.

See Backup And Restore for restore workflows.

Total site loss

Use when all nodes/storage/network for a site are gone.

1. Confirm surviving site is writable and healthy.
2. Confirm DNS points to the surviving site.
3. Disable app scheduling to the lost site.
4. Rebuild site infrastructure.
5. Rejoin via clone or restore.

Expected impact is bounded by failover detection, relay-log drain, DNS TTL, and application reconnect behavior.

Keyring not sealed

Use when bloodraven_keyring_phase{phase="sealed"} is 0 for a site, or status.conditions[type=EncryptionAtRestReady] is False.

A site that is not sealed is running with a writable keyring. Its data is still encrypted, but the operator cannot yet prove the key is durably escrowed.

1. Find out what it is waiting for — message carries the reason:

   kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'
   

2. Unsealed with unsealReason: Clone or Bootstrap is normal and transient. Wait for the clone to finish.
3. Unsealed with "waiting for the sidecar to escrow" — check the sidecar can reach the operator:

   kubectl exec -n orders deploy/mysql-orders-iad -c sidecar -- \
     wget -qO- localhost:8080/keyring/status
   

   
   lastError on that payload names the failure. Common causes: the operator's auxiliary Service is unreachable, or the escrow token Secret was deleted (the operator re-mints it on the next reconcile, but the pod needs to roll to pick it up).
4. Escrowed that never becomes Sealed — the pod has not rolled onto the sealed rendering. Check for a stuck ordered update (status.updatePhase).
5. Failed — read the message and follow Keyring escrow lost if it names a missing or corrupt Secret.

Full background: Data-at-rest encryption.

Keyring escrow lost

Use on a KeyringEscrowMissing or KeyringEscrowCorrupt Event.

1. Confirm the pod is still running:

   kubectl get pod -n orders -l shipstream.io/site=iad
   

2. If it is running, recreate the exact missing Secret from the keyring still mounted in the running pod. Do not force an unseal or rollout: that would destroy the only surviving in-memory copy before it was durable again.

   set -o pipefail
   POD=$(kubectl get pod -n orders \
     -l shipstream.io/failover-group=orders,shipstream.io/site=iad \
     -o jsonpath='{.items[0].metadata.name}')
   SECRET=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringSecret}')
   VERSION=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringVersion}')
   DIGEST=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringDigest}')
   KEYRING_DIR=$(kubectl get mysqlfailovergroup orders -n orders \
     -o jsonpath='{.spec.encryptionAtRest.keyring.dataFileDir}')
   KEYRING_DIR=${KEYRING_DIR:-/run/mysql-keyring}
   FG_UID=$(kubectl get mysqlfailovergroup orders -n orders -o jsonpath='{.metadata.uid}')
   
   kubectl exec -n orders "$POD" -c mysql -- cat "$KEYRING_DIR/keyring" | \
     kubectl create secret generic "$SECRET" -n orders --from-file=keyring=/dev/stdin
   ACTUAL_DIGEST=$(kubectl get secret "$SECRET" -n orders -o jsonpath='{.data.keyring}' | \
     base64 -d | sha256sum | cut -d' ' -f1)
   if [[ "sha256:${ACTUAL_DIGEST}" != "$DIGEST" ]]; then
     kubectl delete secret "$SECRET" -n orders
     echo "restored keyring digest does not match status; refusing to continue" >&2
     exit 1
   fi
   kubectl label secret "$SECRET" -n orders \
     app.kubernetes.io/name=mysql-keyring \
     app.kubernetes.io/managed-by=bloodraven \
     shipstream.io/failover-group=orders shipstream.io/site=iad \
     shipstream.io/keyring-version="$VERSION"
   kubectl annotate secret "$SECRET" -n orders shipstream.io/keyring-digest="$DIGEST"
   kubectl patch secret "$SECRET" -n orders --type=merge -p="{
     \"metadata\":{\"ownerReferences\":[{
       \"apiVersion\":\"shipstream.io/v1alpha1\",\"kind\":\"MysqlFailoverGroup\",
       \"name\":\"orders\",\"uid\":\"${FG_UID}\",\"controller\":true,\"blockOwnerDeletion\":true
     }]},\"immutable\":true
   }"
   
   SITE_INDEX=$(kubectl get mysqlfailovergroup orders -n orders -o json | \
     jq -er '.status.encryptionAtRest.sites | map(.name) | index("iad")')
   kubectl patch mysqlfailovergroup orders -n orders --subresource=status \
     --type=json -p="[
       {\"op\":\"test\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/name\",\"value\":\"iad\"},
       {\"op\":\"replace\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/phase\",\"value\":\"Sealed\"}
     ]"
   

   
   Verify that the site returns to Sealed without restarting the pod.
3. If the pod is already gone, that site's data is unrecoverable. Delete the unreadable PVC, then re-clone it from a healthy peer. The confirmation token is mandatory because this permanently discards the target site's local data:

   kubectl delete pvc mysql-orders-iad-data -n orders
   kubectl annotate mysqlfailovergroup orders -n orders \
     bloodraven.shipstream.io/reclone-site='iad:confirm=orders'
   

4. If every site lost its escrow, restore from backup. Backup artifacts are logical dumps protected by a separate key, so they do not depend on any site's keyring.

Network partition diagnosis

Use Network Partitions. Prioritize preventing two writable sites.

kubectl get mysqlfailovergroup orders -n orders -o yaml
kubectl get endpoints,networkpolicy -n orders
kubectl get events -n orders --sort-by=.lastTimestamp

Failed backup

1. Keep the last successful backup.
2. Inspect MysqlBackup status and Job logs.
3. Fix credentials, storage, or MySQL source health.
4. Trigger a manual backup.
5. Verify the new backup.

kubectl get mysqlbackup -n orders
kubectl describe mysqlbackup -n orders <backup-name>

Failed restore

1. Confirm this is a recovery environment or approved destructive action.
2. Inspect restore Job logs.
3. Validate source path and decryption Secret.
4. Recreate the recovery failover group if bootstrap partially initialized MySQL.

Operator unavailable

Use Operator Availability. Sidecar self-fencing is part of the safety model; do not disable it during diagnosis.

kubectl rollout status deployment/bloodraven -n bloodraven
kubectl logs -n bloodraven deploy/bloodraven
kubectl get lease -A | grep bloodraven

DNS failover stuck

1. Confirm status.activeSite changed.
2. Inspect DNSEndpoint and external-dns logs.
3. Confirm provider record and TTL.
4. If applications still write old site, fence the old site and restart affected clients.

kubectl get dnsendpoint -A | grep orders
kubectl logs -n external-dns deploy/external-dns
dig orders.az.example.com