Network partitions

This runbook expands the network rows in the failure-mode matrix. It focuses on what the operator can observe, which action it takes, which metrics/events move, and what an operator should do after connectivity returns.

Timing assumes the defaults: pollInterval=2s, failureThreshold=3, sidecar.leaseTimeout=20s, and a 30-second relay-log drain timeout. pollInterval is the base cadence. Continued failures after a site is marked unreachable back the group loop off exponentially to a 30-second cap. A first isolated partition still debounces in about 6 seconds; a second fault during a sustained partition can take up to about 90 seconds to reach the three-poll threshold. Sidecar lease timing is independent of this operator poll backoff.

Effective self-fencing timeout

For production with a 60-second DNS TTL, keep max(configured leaseTimeout, 3s, 3 * effective peerCheckInterval) < 60s. The sidecar first clamps peerCheckInterval to at least 1s, then applies this lease minimum. Defaults are a 20s lease and 5s peer interval, so the effective timeout is 20s. Setting a 20s peer interval with a 20s configured lease produces an effective 60s timeout and violates the strictly-less-than requirement. Check the sidecar starting log's effective leaseTimeout and peerCheckInterval, not only the CR values.

This is an expiry threshold, not a hard wall-clock fence-completion bound. The next monitor tick, reachability probe timeouts, scheduling delay, and the MySQL super_read_only write/confirmation take additional time; connection eviction is best-effort and separately bounded. Leave margin and verify actual fencing completion under production-like faults. DNS caches and established connections also need not obey the record TTL. A fresh conflicting operator-authoritative topology, learned directly or through /peer/active-site, triggers fencing without waiting for lease expiry. Without that mismatch, lease fencing requires both the operator and every peer to be unreachable. Deployment API holds are unrelated to this sidecar lease and never postpone self-fencing.

First principles

• Bloodraven promotes only when the current primary is unreachable and a promotable primary-candidate replica is reachable.
• A replica-side network problem does not trigger failover while the primary remains writable.
• The sidecar self-fences a writable MySQL when it cannot reach the operator and cannot reach any peer for longer than leaseTimeout.
• Cross-site replication lag is an alert, not a failover trigger, when the primary is otherwise healthy.
• Once a partition heals, GTID comparison decides whether the returning site can rejoin automatically or must be recloned.

Scenario A: operator cannot reach site A, site B reachable

Example: the active primary is iad; the operator and pdx are on the surviving side of a site partition; all operator polls to iad time out.

Recovery after heal:

1. Watch status.sites[?(@.name=="iad")].recoveryState.
2. If there is no divergence, the operator reconfigures iad as a replica automatically.
3. If divergentGtid is set, review lost transactions and trigger the reclone flow in Operations.

Scenario B: replica site isolated, primary reachable

Example: active primary iad remains reachable by the operator; replica pdx cannot receive replication traffic or cannot be polled.

Recovery after heal:

1. Confirm replication restarts and lag falls below spec.replication.maxLagSeconds.
2. If replication does not restart, inspect MySQL replica status and operator logs for old primary recovery failed or replication errors.
3. If the replica has been manually written to and now diverges, reclone it from the current primary.

Scenario C: MySQL-to-MySQL link broken, operator reaches both

Example: the operator can poll both iad and pdx, but pdx cannot pull binlog events from iad.

This is the scenario most likely to page a human without automatic action. Bloodraven intentionally refuses to guess that lag means the primary should be abandoned.

Scenario D: asymmetric peer reachability

Example: operator reaches both sites, iad can reach pdx, but pdx cannot reach iad; or only one sidecar can reach its peer.

Asymmetric partitions are exactly why GTID reconciliation after heal matters. If either side accepted writes that the final primary did not receive, the returning site is fenced and marked divergent.

MySQL replication is replica-initiated, so the two orientations are not equivalent. A cut from replica to primary stops IO fetch; a cut from primary to replica does not. Sidecar peer checks are likewise client-initiated: a writable primary self-fences on the lease rule only when it cannot reach the operator and cannot reach any peer.

Scenario E: both sites unreachable to the operator

Testing partitions

Use pod-level NetworkPolicy or CNI-native fault injection. Host-level iptables rules in a k3d node are not reliable for Kubernetes Service traffic because kube-proxy DNAT and pod networking happen in different paths.

Playground examples live in playground/chaos-scenarios.md and use NetworkPolicy-based partitions. Always run ./playground/chaos.sh recover or delete the NetworkPolicy manually after a test.

Coverage of the five shapes:

Shape D is the lease-rule case: one reachable peer suppresses self-fence even when the operator is silent. The reverse orientation (primary cannot reach the replica) is what lets the lease expire. Both orientations are hand-built DST trials; the campaign also schedules partitionOneWay.

On-call checklist

1. Identify the current active site: kubectl get mysqlfailovergroup orders -o jsonpath='{.status.activeSite}'.
2. Print per-site state: kubectl get mysqlfailovergroup orders -o jsonpath='{range .status.sites[*]}{.name}: {.state} lag={.secondsBehindSource} recovery={.recoveryState}{"\n"}{end}'.
3. Check whether bloodraven_failovers_total increased. If not, the operator likely chose to alert rather than promote.
4. If a site returns with divergentGtid, do not manually attach it as a replica. Follow the reclone flow.
5. After recovery, verify replication lag is below your RPO threshold and that DNS/external-dns has converged if clients use external DNS.