# Bloodraven ![intro infographic](https://bloodraven.dev/img/infographics/intro.webp) Bloodraven is a Kubernetes operator for MySQL async replication failover groups across sites. It automates failover detection, promotion, DNS steering, application workload migration, and optional Dragonfly cache/session sidekicks that follow the active MySQL site so a site-level outage can recover without human intervention. ## Who this is for | Reader | Use these docs to | | --------------------- | ------------------------------------------------------------------------------------ | | New user | Try Bloodraven locally and create a first failover group. | | Platform operator | Install the operator, define production guardrails, and run go-live checks. | | Application developer | Connect safely and handle failover reconnect behavior. | | On-call responder | Map alerts to runbooks and verify recovery under pressure. | | Backup owner | Configure, verify, and restore backups without reading the full CRD reference first. | ## What Bloodraven does not do - It does not provide synchronous replication or zero RPO after sudden primary loss. - It does not replace external-dns, cert-manager, Prometheus, Grafana, or your object store. - It does not make application connection pools failover-aware automatically. - It does not reconcile divergent writes for you after split-brain. - It does not make PVC-local backups durable after cluster or storage loss. - It does not treat Dragonfly as durable application storage; managed Dragonfly is for cache/session continuity. ## User journeys | Journey | Path | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | New user | [Getting Started](https://bloodraven.dev/docs/get-started/getting-started) → [Playground](https://bloodraven.dev/docs/get-started/playground) → [App Integration](https://bloodraven.dev/docs/configuration/app-integration) | | Platform operator | [Production Install](https://bloodraven.dev/docs/get-started/install-production) → [Production Hardening](https://bloodraven.dev/docs/configuration/production-hardening) → [Monitoring](https://bloodraven.dev/docs/observability/monitoring) | | On-call | [Operations Overview](https://bloodraven.dev/docs/operations/operations-overview) → [Failure Mode Matrix](https://bloodraven.dev/docs/operations/failure-mode-matrix) → [Runbooks](https://bloodraven.dev/docs/operations/runbooks) | | Backup owner | [Backup Overview](https://bloodraven.dev/docs/backup-and-restore/backup-overview) → [S3](https://bloodraven.dev/docs/backup-and-restore/backup-s3) or [PVC](https://bloodraven.dev/docs/backup-and-restore/backup-pvc) → [Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) → [Restore](https://bloodraven.dev/docs/backup-and-restore/backup-restore) | ## Standout features - **Automatic MySQL site failover**: If the active MySQL site dies, Bloodraven promotes another site, moves traffic, updates DNS, and helps the old primary rejoin safely. - **Split-brain protection**: If two sites might both accept writes, the operator and sidecars fence unsafe MySQL nodes so the cluster does not keep writing in two places. - **Graceful planned switchover**: An admin can move the primary site with one command; Bloodraven waits for the replica to catch up first, so planned moves can have zero data loss. - **Backup, restore, PITR, and verification**: Bloodraven can create backups, archive binlogs for point-in-time recovery, encrypt artifacts, restore from them, and test backups by loading them into a throwaway MySQL. - **Dragonfly cache/session failover**: Bloodraven can manage Dragonfly alongside MySQL, move the active cache/session endpoint during failover, and try to preserve sessions during planned moves. - **Chaos-tested in CI**: 30+ automated chaos scenarios — primary kills, network partitions, split-brain, self-fencing, data wipes, backup/PITR verification — run against real Kubernetes clusters nightly, and a smoke subset gates every release before artifacts are published. The same [playground](https://bloodraven.dev/docs/get-started/playground) runs locally so you can test these failure modes yourself before you trust them in production. ::tip **Using an AI agent?** This documentation is available as `llms.txt` and `llms-full.txt` at the site root for consumption by LLM-based tools. Give your agent context with a prompt like: ```text Read https://bloodraven.dev/llms-full.txt and help me configure a MysqlFailoverGroup for two sites with async replication and automatic DNS failover. ``` :: ::note **When to use Bloodraven** Bloodraven targets the **two-site, accept-non-zero-RPO** deployment. If you need synchronous writes and zero RPO on primary loss, read [Why not Group Replication?](https://bloodraven.dev/docs/architecture/why-not-group-replication) first to make sure the tradeoffs match your use case. :: ## Components | Component | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `bloodraven` | The operator binary. Runs as a Deployment, watches `MysqlFailoverGroup` CRs, reconciles MySQL state. | | `bloodraven-sidecar` | Runs alongside each MySQL container. Provides health probes and self-fencing when the operator is unreachable. | | Managed Dragonfly | Optional per-site Dragonfly pods created when `spec.dragonfly.enabled=true`. Applications use the active `dragonfly` Service for Redis-compatible cache/session traffic. | ## Custom resource Bloodraven introduces a single CRD: - **`MysqlFailoverGroup`** (`shipstream.io/v1alpha1`) -- Declares MySQL instances across named sites, their storage, networking, DNS, transport layer security (TLS), failover tuning, and optional Dragonfly co-management. ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: credentials: operatorSecret: mysql-operator-creds appSecret: mysql-app-creds dns: hostname: orders.az.example.com ttl: 60 sites: - name: iad zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.1.1 storage: size: 50Gi storageClassName: gp3 - name: pdx zone: us-west-2a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.0.2.1 storage: size: 50Gi storageClassName: gp3 ``` ## Next steps - [Architecture](https://bloodraven.dev/docs/architecture/architecture) -- How the operator, sidecars, MySQL instances, and optional Dragonfly sidekicks interact. - [Getting Started](https://bloodraven.dev/docs/get-started/getting-started) -- Install the operator and create your first failover group. - [Production Install](https://bloodraven.dev/docs/get-started/install-production) -- Production dependencies, Helm values, CRD ownership, and verification. - [Playground](https://bloodraven.dev/docs/get-started/playground) -- Try Bloodraven locally on k3d, kind, or minikube with a live dashboard and chaos tools. - [Backup Overview](https://bloodraven.dev/docs/backup-and-restore/backup-overview) -- Choose S3 or PVC, configure schedules, and verify recoverability. - [Runbooks](https://bloodraven.dev/docs/operations/runbooks) -- Incident response entry points. - [CRD Reference](https://bloodraven.dev/docs/configuration/crd-reference) -- Complete spec and status field reference. # License token contract This is the contract the Bloodraven operator verifies offline. Tokens are minted on demand at [/license](https://bloodraven.dev/license) from a Polar order. The operator never calls Polar; it only checks this JWT. The token is a compact JWS. Polar's native license keys are opaque UUID4 strings validated against Polar's API and cannot be verified offline, so Bloodraven issues this JWT instead. ## Header | Field | Type | Required | Value | | ----- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `alg` | string | yes | Exactly `EdDSA`. Anything else, including `none`, `Ed25519`, `HS256`, and `ES256`, is rejected before the signature is checked. | | `kid` | string | yes | Non-empty key id. Must match an entry in the operator's append-only trust store. | | `typ` | string | no | If present, must be `JWT`. | ## Claims | Claim | Type | Required | Meaning | | -------------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `iss` | string | yes | Exactly `https://license.shipstream.io/bloodraven`. | | `sub` | string | yes | Customer / subject id. Non-empty. | | `org` | string | yes | Organization display name. Used as the Prometheus `organization` label. | | `edition` | string | yes | `production` or `organization`. | | `issuedFor` | string | yes | Polar order id, for support correlation. | | `iat` | number | yes | Unix seconds. Integer, ≥ 0. The operator allows 7 days of clock skew into the future. | | `nbf` | number | no | Unix seconds. Same 7-day future leeway if present. | | `updatesUntil` | number | yes | Unix seconds. End of the paid update period. **Not** license expiry. A past value is the supported perpetual state. | | `exp` | — | **omit** | Do not include this claim. The product is a perpetual license plus 12 months of updates. If `exp` is the update-period end, every conformant JWT library will reject the token after 12 months and the operator will be unable to read `org` or `edition`. The operator ignores `exp` if a buggy signer includes it. | Do not put the update-period end in `exp`. Put it in `updatesUntil`. Size limits the operator enforces (larger values are `malformed`): | Limit | Value | | --------------- | ---------- | | Compact token | 8192 bytes | | Decoded header | 1024 bytes | | Decoded payload | 4096 bytes | ## Signature Ed25519 over `base64url(header) + "." + base64url(payload)` (unpadded base64url). The signature segment is the raw 64-byte signature, also unpadded base64url. The operator never fetches keys, JWKS, or a revocation list. The public key for each `kid` is compiled into the binary. Keys are append-only: retired kids stay in the store so old tokens keep verifying. ## Renewal Fetch the token again at [/license](https://bloodraven.dev/license) with the Polar order ID. If Polar attached a subscription to that order, `updatesUntil` comes from the current period end. If the renewal is a separate one-time order, use that order ID. Replace `spec.license` or the operator `license` value with the new string. The operator just verifies it. A first purchase with no subscription mints `updatesUntil` from the order's `created_at` plus 12 months (or the commercial term in force). ## Worked example This token is signed with a **test-only** key. The production operator will report it as `unknown kid`. Do not ship this key. Header: ```json {"alg":"EdDSA","kid":"test-only-1","typ":"JWT"} ``` Payload: ```json { "edition": "organization", "iat": 1755216000, "iss": "https://license.shipstream.io/bloodraven", "issuedFor": "ord_example", "org": "Acme Corp", "sub": "cus_example", "updatesUntil": 1786752000 } ``` `iat` is 2025-08-15T00:00:00Z. `updatesUntil` is 2026-08-15T00:00:00Z. There is no `exp`. Compact token: ```text eyJhbGciOiJFZERTQSIsImtpZCI6InRlc3Qtb25seS0xIiwidHlwIjoiSldUIn0.eyJlZGl0aW9uIjoib3JnYW5pemF0aW9uIiwiaWF0IjoxNzU1MjE2MDAwLCJpc3MiOiJodHRwczovL2xpY2Vuc2Uuc2hpcHN0cmVhbS5pby9ibG9vZHJhdmVuIiwiaXNzdWVkRm9yIjoib3JkX2V4YW1wbGUiLCJvcmciOiJBY21lIENvcnAiLCJzdWIiOiJjdXNfZXhhbXBsZSIsInVwZGF0ZXNVbnRpbCI6MTc4Njc1MjAwMH0.MlHGkwxyk325K5RWI_rIYCLkFBzmRWD6jTa2OQ2zp9rHgjW6Fy1gT_V93T2WHzQebLnJhkH8eKDe-YwQyLFkDg ``` Test-only public key (32 bytes, hex), **not** in the production trust store: ```text 4cb5abf6ad79fbf5abbccafcc269d85cd2651ed4b885b5869f241aedf0a5ba29 ``` ## Inserting the production public key Generate the keypair on the signer host. Never commit the private key. ```bash openssl genpkey -algorithm ED25519 -out license-ed25519.pem openssl pkey -in license-ed25519.pem -pubout -outform DER | tail -c 32 | xxd -p -c 32 ``` Add the 64 hex characters to `internal/license/keys.go` as `"br-YYYY-N": ""`. A build with no entries compiles and treats every token as unknown kid (Community behavior, `valid="false"`). It does not panic. # Licensing Bloodraven is **source-available**, not open source. The full source is public, you can read it, build it, modify it, and contribute to it. Running it in production at a commercial company requires a one-time license. We are deliberate about that wording. Bloodraven is licensed under the [Business Source License 1.1](https://github.com/ShipStream/bloodraven/blob/main/LICENSE){rel=""nofollow""}, which is not an [OSI-approved](https://opensource.org/licenses){rel=""nofollow""} open source license, so we do not call it one. ## Do I need to pay? Not if you are an individual, a student, a researcher, an educator, a non-profit, or a company under $1M annual revenue. Not for dev, test, staging, CI, or evaluation — those are free at any scale, forever, whoever you are. A company over $1M annual revenue running Bloodraven in production needs a license. [**Pricing**](https://bloodraven.dev/pricing) carries the full eligibility table, the price of every edition, renewal prices, and the checkout links. It is the only page on this site that states a price, so there is nothing to reconcile. Licensing is counted in failover groups. A failover group is one `MySQLFailoverGroup` resource, however many sites, replicas, or read-only instances it contains. Two sites is still one failover group. If you are not sure which side of the line you are on, email **** and ask. The answer is free and we will put it in writing. ## What you are actually buying Bloodraven is not a support contract with software attached. It is the reverse. What is included, for everyone, free: - Every page of [these docs](https://bloodraven.dev/docs/), including the [runbooks](https://bloodraven.dev/docs/operations/runbooks), the [failure mode matrix](https://bloodraven.dev/docs/operations/failure-mode-matrix), and the [known limitations](https://bloodraven.dev/docs/get-started/known-limitations) we publish up front. - The [playground](https://bloodraven.dev/docs/get-started/playground), which reproduces the full two-site topology and every chaos scenario locally. - Published results from the deterministic simulation and chaos suites. - Bug reports on GitHub, from licensees and non-licensees alike. What a license buys is the right to run Bloodraven in production, perpetually, plus 12 months of updates. **Your license never expires.** If you stop renewing updates you keep the perpetual right to run every version published while your update period was active. What is **not** included at any price except [Priority Triage](https://bloodraven.dev/pricing#priority-triage): technical support, an SLA, on-call coverage, emergency incident response, installation help, consulting, or guaranteed feature delivery. See [commercial terms §6](https://github.com/ShipStream/bloodraven/blob/main/LICENSE-COMMERCIAL.md#6-what-is-not-included){rel=""nofollow""}. We would rather write documentation once than answer the same question fifty times, and we price accordingly. If you need a vendor to call at 3am, buy a managed database service instead — that is a legitimate choice and Bloodraven is not trying to replace it. ## The license key Licenses are sold through [Polar](https://polar.sh){rel=""nofollow""}, which is the merchant of record. [Pricing](https://bloodraven.dev/pricing#how-buying-works) covers checkout, invoicing, VAT and refunds. Fetch a license key at [/license](https://bloodraven.dev/license) with the Polar order ID from your receipt and the email you used at checkout. You can fetch it again if you lose it. **You do not need it to run Bloodraven.** There is no activation, no license server, no feature gating, and no phone-home. The software is fully functional without a key and always will be. Your receipt is the license, and compliance is on the honor system. The key exists only so you have something concrete to file: an artifact for your own procurement records, expense report, and internal software inventory. ## Recording the license in the cluster The token is an Ed25519-signed JWT. It is a public assertion, not a credential. You can paste it into the cluster so the operator can verify it **offline** — no network calls, no phone-home — and log your organization, edition, and update-period end, plus expose them as Prometheus metric labels. This is for your own compliance auditing. **It never gates functionality or changes operator behavior.** An invalid token, a missing token, and a token whose update period has ended all leave failover, reconcile, and MySQL behavior unchanged. Offline verification shipped in 1.1.0. An Organization license usually lives on the operator and covers every group: ```yaml # Helm values license: "eyJhbGciOiJFZERTQSIsImtpZCI6ImJyLTEiLCJ0eXAiOiJKV1QifQ...." ``` That sets `BLOODRAVEN_LICENSE` (or you can pass `--license` on the operator command line). A Production license for one group goes on the group and overrides the operator default: ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: license: "eyJhbGciOiJFZERTQSIsImtpZCI6ImJyLTEiLCJ0eXAiOiJKV1QifQ...." # ... ``` Resolution per group: `spec.license` if set, else the operator default, else Community. If `spec.license` is set and fails verification, the operator does **not** fall through to the operator default — it reports the field as invalid and continues as Community. A license whose update period has ended is still valid. That is the documented perpetual state. The operator logs it at Info, keeps `valid="true"`, and keeps running every version published while the update period was active. Metrics: ```text bloodraven_license_info{namespace="orders",group="orders",organization="Acme Corp",edition="organization",valid="true"} 1 bloodraven_license_updates_expiry_timestamp_seconds{namespace="orders",group="orders",organization="Acme Corp",edition="organization"} 1786752000 ``` With no license configured the info series is still emitted, with `edition="community"` and `valid="true"`, so dashboards do not break on the free tier. Alert on a paid update period ending in the next 30 days with: ```promql (bloodraven_license_updates_expiry_timestamp_seconds - time()) / 86400 < 30 ``` That alert is a renewal reminder. It is not an outage. The operator does not change behavior when the timestamp is in the past. The signer contract — header, claims, `updatesUntil`, and a worked example — is on [License token contract](https://bloodraven.dev/docs/license-token). Your receipt remains the license even if you never paste the token anywhere. ## Why source-available and not Apache 2.0 Two reasons, and we would rather state them plainly than let you guess. **We want you to be able to audit this.** Bloodraven automatically promotes production databases and fences nodes it believes are unsafe. You should not run software like that on trust. Every promotion decision, every fencing path, and every split-brain guard is in the repository for you to read before you deploy it. **We want it to still exist in five years.** Bloodraven is maintained by ShipStream, which runs it in production for its own warehouse management system. Licensing revenue is what justifies the maintenance time. A free-with-paid-support model would push us toward selling support, which we do not want to sell and you probably do not want to buy. ## What happens if ShipStream stops maintaining Bloodraven Every version converts to the **Apache License 2.0 two years after it is published**. This is written into the [license](https://github.com/ShipStream/bloodraven/blob/main/LICENSE){rel=""nofollow""} as the Change Date — it is automatic and we cannot revoke it. In practice: the version you are running today becomes fully open source, forever, in two years, no matter what happens to us. If ShipStream disappears, the project does not become unusable and you are not locked out of code you depend on. Two years is shorter than the four-year maximum the license allows, and shorter than what most BSL projects choose. That is intentional. ## Contributing Pull requests are welcome from everyone, licensed or not. First-time contributors sign a [CLA](https://github.com/ShipStream/bloodraven/blob/main/CLA.md){rel=""nofollow""}, which is a license grant rather than a copyright assignment — you keep the copyright in your own work. See [CONTRIBUTING.md](https://github.com/ShipStream/bloodraven/blob/main/CONTRIBUTING.md){rel=""nofollow""}. ## Common questions **Where do I buy it?**[Pricing](https://bloodraven.dev/pricing) — prices, the eligibility table, and the checkout links for every edition and renewal. **Can I fork it?** Yes. The license permits copying, modification, and derivative works. What it does not permit is offering Bloodraven, or a substantially similar derivative, to third parties as a hosted or managed service. **Can I run it for my own SaaS product's databases?** Yes, with a license. That is ordinary internal production use. What you cannot do is resell Bloodraven itself, or offer managed MySQL to third parties with it. **Do staging and DR sites need separate licenses?** Staging, dev, test, CI, and evaluation are free at any scale. A standby or DR *site within a production failover group* is already covered by that group's license — a two-site production group is one license, not two. **We are over $1M revenue but only use it for internal tooling.** That is still production use if it serves a live system. It is one failover group, so one Production license — see [pricing](https://bloodraven.dev/pricing#editions). **What if we grow past $1M mid-year?** Buy a license when you cross the threshold. We are not going to audit you. **Do you offer volume, multi-year, or PO billing?** Yes. Email ****. **Can we get a signed agreement, a W-9, or a security questionnaire completed?** Yes, for Organization purchases. Email **** before buying. **Is there a trial?** The free tier is the trial and it has no time limit. Run it in staging as long as you want. There is also a 30-day refund on any purchase. --- Questions this page does not answer: **** # Known limitations ![known limitations infographic](https://bloodraven.dev/img/infographics/known-limitations.webp) This page is the short version of Bloodraven's current boundaries. Read it before writing production manifests so the failure and recovery model matches your expectations. ## API maturity - The CRD API version is `shipstream.io/v1alpha1`. - Fields may still change before `v1beta1` / `v1`. - There is not yet a published CRD conversion-webhook or migration contract. Track this under the CRD version-migration wishlist item. ## Replication and RPO - Bloodraven uses asynchronous MySQL replication. Emergency failover can lose transactions that committed on the old primary but had not reached the promoted replica. - The operator records promotion and divergent GTID sets so data loss is observable, but it does not merge divergent data automatically. - Planned failover is the zero-RPO path: it fences the source, waits for the target's `GTID_EXECUTED` to cover the fenced source GTID, then promotes. - If you require synchronous commit semantics or quorum-based zero RPO on primary loss, Bloodraven is the wrong tool; see [Why not Group Replication?](https://bloodraven.dev/docs/architecture/why-not-group-replication). - Read-only sites are full MySQL replicas. Filtered or partial replication, replication proxying, and query routing through an operator-managed proxy are not implemented. - Clone and reclone copy from the active primary at full speed. Bloodraven does not throttle clone traffic, so adding or rebuilding a large reader can consume primary network, storage, and CPU capacity. Schedule the operation and monitor the donor accordingly. ## Operator availability - Leader election is enabled by the chart, but the default deployment still runs one replica. Run more than one operator replica only after validating the deployment model in your cluster. - Sidecars preserve safety while the operator is unavailable, but new failover decisions wait for an operator to run. - If the primary fails while the operator is down, writes remain unavailable until the operator returns and completes failover. - Anti-flap state (`lastFailover`, `lastFailoverTarget`) is persisted twice on every promotion: in `status.lastFailover` / `status.lastFailoverTarget`, and out of band in the failover group's own annotations (`bloodraven.shipstream.io/last-failover`, `bloodraven.shipstream.io/last-failover-target`). The two travel different API paths — the status subresource has its own RBAC rule and admission chain — so an outage on one does not take the other with it. A restarting operator rehydrates from whichever copy is newer, and both writes retry every poll until accepted. Losing the cooldown across a restart therefore requires **both** paths to be rejecting writes at once; while that holds, a subsequent site failure inside the intended cooldown window can promote earlier than configured (deterministic-simulation finding, `CooldownViolated(restart+stateLost)` class). GTID gates still bound the damage — any divergence this causes is fenced and reported. Alert on `mysqlfailovergroups` and `mysqlfailovergroups/status` write failures (the operator logs and retries both every poll) to close the window. A restart that had to fall back to the annotations logs `restored lastFailover from out-of-band annotations` at WARN, which is a direct signal that this group's status writes were failing. This guarantee starts with operator builds that understand those annotations. Rolling back to an older build makes the annotation copy invisible again; do not rely on cooldown continuity across such a rollback while status writes are failing. ## Placement and shared nodes - Taints and node discovery are scoped per failover group with `spec.sites[].taintNodeSelector`, so one physical node can advertise membership in multiple failover groups at the same site. - Application workloads on shared nodes must tolerate other groups' readonly taints but not their own group's taint. ## Backups and restore - Backup and PITR support is present, including backup verification, but restore-duration metrics and restore-performance guidance are still missing. - In-place restore exists for destructive rollback of a live group. Use it carefully: full-instance restore fences writes and reclones the peer after loading the dump. - PVC loss can be recovered by recloning from the current primary, but committed transactions that only existed on the lost PVC are gone. See [Operations](https://bloodraven.dev/docs/operations/operations#pvc-loss-recovery-runbook). - Total cluster loss (all nodes and PVCs destroyed) requires recovering into a separate Kubernetes cluster from the source backup archive. See [Multi-cluster DR](https://bloodraven.dev/docs/architecture/multi-cluster-dr) for the end-to-end runbook using `spec.initFromBackup` with optional PITR replay. - `read-only` sites cannot be selected for scheduled, automatic, or source-overridden backups. Healthy `dr-only` followers remain eligible. ## Dragonfly co-management - Managed Dragonfly is optional and intended for cache/session continuity, not durable application state. - Dragonfly pods use ephemeral storage unless you configure `spec.dragonfly.snapshot` for planned snapshot-restore maintenance. Bloodraven does not schedule Dragonfly backups as durable data backups. - Emergency MySQL failover never blocks on Dragonfly. If Dragonfly sync or promotion fails, sessions/cache may be discarded while MySQL recovery completes. - `spec.tls` applies to MySQL, not Dragonfly. Protect Dragonfly with NetworkPolicy, Dragonfly auth, and any external TLS/service-mesh controls your environment requires. ## Network partitions - The [network-partition runbook](https://bloodraven.dev/docs/operations/network-partitions) covers shapes A–E, including asymmetric peer reachability (shape D). Shape D is injected in the deterministic simulator (`partitionOneWay`); there is still no live-cluster chaos scenario for C or D. - Playground partition tests must use pod-level NetworkPolicy or an equivalent mechanism; host-level iptables rules do not reliably block Kubernetes Service traffic in k3d. ## Observability and tooling - Grafana dashboards and metrics are shipped, but PrometheusRule alert examples are not yet packaged as first-class chart artifacts. - The `kubectl bloodraven` plugin wraps supported day-2 operations, but it remains a thin API client; the operator performs the authoritative safety checks. ## Data-at-rest encryption - `spec.encryptionAtRest` uses MySQL Community's `component_keyring_file`. It is **not** Oracle "MySQL Enterprise TDE", and Oracle states that file-based keyrings are not intended as regulatory-compliance solutions. Do not present it as compliance-grade TDE. - The live keyring is a Kubernetes Secret, so **etcd becomes part of your key custody**. API-server encryption at rest for Secrets (ideally KMS-backed) and tight Secret RBAC are prerequisites, not recommendations. Bloodraven cannot verify either. - Memory-backed volumes can reach swap on kernels older than 6.3, where `tmpfs noswap` is unsupported. Disable or encrypt swap on worker nodes. - Enabling encryption on a group that already holds data is refused by default: MySQL only encrypts what is written afterwards, so existing tablespaces stay plaintext. The supported adoption path is replica-first (bootstrap encrypted, load, planned-failover, wipe the old PVCs); an annotation exists to accept partial coverage knowingly. - Master-key rotation is refused on the active primary. Rotate replicas, planned-failover, then rotate the ex-primary. - Backup, restore, and backup-verification still stage plaintext in pod `emptyDir` volumes. Backup *artifacts* are separately protected by `spec.backup.profiles[].encryption`; the transient staging inside the Job's own pod is not covered by live MySQL encryption. - The `mysqldDir` and `pluginDir` defaults match the official `mysql:9.x` images. Other images need those fields set explicitly, and a wrong value shows up as InnoDB refusing to start. ## External dependencies - DNS steering depends on external-dns consuming `DNSEndpoint` objects. Bloodraven updates the CR; DNS provider propagation time and TTLs are outside the operator's control. - Production installs need real topology-aware persistent storage. Local path / hostPath storage is acceptable for playground use only. - The unauthenticated auxiliary and sidecar HTTP surfaces assume a trusted pod network. Use NetworkPolicy before exposing those Services broadly. # How to create your first MysqlFailoverGroup ![getting started infographic](https://bloodraven.dev/img/infographics/getting-started.webp) Use this guide to install Bloodraven and create a two-site `MysqlFailoverGroup` in a Kubernetes cluster. It is the shortest successful path for a first non-production group. For a complete local demo with a dashboard, counter app, and chaos scenarios, use the [Playground](https://bloodraven.dev/docs/get-started/playground). For production dependency ownership, image pinning, monitoring, and NetworkPolicy, use [Production Install](https://bloodraven.dev/docs/get-started/install-production). ## Prerequisites - Kubernetes 1.27 or newer. - Helm 3. - Two sets of nodes, each labeled for a different site. - A working StorageClass for MySQL persistent volumes. - [external-dns](https://github.com/kubernetes-sigs/external-dns){rel=""nofollow""} configured for your DNS provider. - The `externaldns.k8s.io/v1alpha1` `DNSEndpoint` custom resource definition (CRD) installed in the cluster. - Optional: cert-manager, if you want Bloodraven to request certificates through `spec.tls.issuerRef`. ## 1. Install the operator Add the Helm repository and install the operator: ```bash helm repo add bloodraven https://raw.githubusercontent.com/shipstream/bloodraven/gh-pages helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven \ --create-namespace ``` Verify that the operator is running: ```bash kubectl rollout status deployment/bloodraven -n bloodraven kubectl get crd mysqlfailovergroups.shipstream.io ``` You should see the `bloodraven` Deployment complete its rollout and the `MysqlFailoverGroup` CRD registered. ## 2. Create MySQL credentials Create separate Secrets for operator and application credentials. Bloodraven uses the operator credentials to manage MySQL users and replication, and it uses the application credentials for app-facing access. ```bash kubectl create secret generic mysql-operator-creds \ --namespace default \ --from-literal=username=bloodraven \ --from-literal=password=OPERATOR_PASSWORD \ --from-literal=MYSQL_ROOT_PASSWORD=ROOT_PASSWORD kubectl create secret generic mysql-app-creds \ --namespace default \ --from-literal=username=app \ --from-literal=password=APP_PASSWORD ``` Replace `OPERATOR_PASSWORD`, `ROOT_PASSWORD`, and `APP_PASSWORD` with real values before running the commands. ::note **Legacy DSN secret** Bloodraven still accepts `spec.secretName` for older manifests that use one MySQL data source name (DSN) Secret. New installs should use `spec.credentials.operatorSecret` and `spec.credentials.appSecret`. :: ## 3. Label nodes by site Label at least one node for each site. The labels must match the selectors you put in `spec.sites[].taintNodeSelector`. ```bash kubectl label node NODE_IAD_1 \ shipstream.io/failover-group.orders=true \ shipstream.io/site.orders=iad kubectl label node NODE_PDX_1 \ shipstream.io/failover-group.orders=true \ shipstream.io/site.orders=pdx ``` Replace `NODE_IAD_1` and `NODE_PDX_1` with real node names. See [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract) for the full labeling and scheduling contract. ## 4. Create the failover group Create `orders-fg.yaml`: ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: default spec: image: mysql:9.7 sidecarImage: ghcr.io/shipstream/bloodraven-sidecar:1.1.0 credentials: operatorSecret: mysql-operator-creds appSecret: mysql-app-creds dns: hostname: orders.az.example.com ttl: 60 sites: - name: iad zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.1.1 storage: storageClassName: fast-ssd size: 100Gi - name: pdx zone: us-west-2a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.0.2.1 storage: storageClassName: fast-ssd size: 100Gi pollInterval: 2s failureThreshold: 3 recoveryThreshold: 2 failoverCooldown: 5m updateStrategy: OrderedUpdate replication: maxLagSeconds: 300 cloneTimeout: 3600 ``` Update these values before applying the manifest: | Field | Replace with | | --------------------------------------- | ------------------------------------------------------- | | `spec.dns.hostname` | The hostname that should point at the active primary. | | `spec.sites[].lbIP` | The load balancer IP for each site. | | `spec.sites[].storage.storageClassName` | A StorageClass that can create MySQL PVCs in that site. | | `spec.sites[].taintNodeSelector` | Labels that select only nodes in that site. | | `spec.image` and `spec.sidecarImage` | Pinned image tags for production. | Apply the manifest: ```bash kubectl apply -f orders-fg.yaml ``` ## 5. Verify the group Wait for the MySQL pods: ```bash kubectl get pods -n default -l shipstream.io/failover-group=orders ``` Check the failover group status: ```bash kubectl get mysqlfailovergroup orders -n default -o yaml ``` The status should show one active site, one writable site, one read-only site, and a `Ready=True` condition: ```yaml status: activeSite: iad sites: - name: iad state: writable replicating: false - name: pdx state: read-only replicating: true secondsBehindSource: 0 conditions: - type: Ready status: "True" - type: Degraded status: "False" ``` Check the operator status API: In one terminal, start a port forward: ```bash kubectl port-forward -n bloodraven deploy/bloodraven 8082:8082 ``` In another terminal, call the status endpoint: ```bash curl http://localhost:8082/status ``` Smoke-test MySQL through the primary Service: ```bash kubectl run mysql-client -n default --rm -it --restart=Never \ --image=mysql:9.7 -- \ mysql -h mysql-orders-primary.default.svc.cluster.local \ -u app -pAPP_PASSWORD \ -e 'SELECT @@hostname, @@read_only, @@super_read_only;' ``` Verify DNS after the first active site is selected: ```bash kubectl get dnsendpoint -A | grep orders.az.example.com dig orders.az.example.com ``` ## Troubleshooting first runs If the group does not become ready, check these common causes first: | Symptom | Likely cause | Next step | | ----------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | MySQL pod remains Pending | Node labels or StorageClass do not match the site. | Check [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract). | | Operator logs mention missing Secret keys | The credential Secret is missing a required key. | Recreate the Secret from [Credentials and TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls). | | DNS does not change | external-dns is not watching the namespace or `DNSEndpoint` CRD. | Check [Troubleshooting](https://bloodraven.dev/docs/operations/troubleshooting#dns-not-changing-after-failover). | | Replica does not catch up | Replication credentials, network access, or clone bootstrap failed. | Check [Troubleshooting](https://bloodraven.dev/docs/operations/troubleshooting#replication-stopped). | ## Next steps - [App Integration](https://bloodraven.dev/docs/configuration/app-integration) - Connect applications to the primary and replica Services. - [GitOps Guidance](https://bloodraven.dev/docs/operations/gitops) - Configure Argo CD or Flux ownership, sync order, and health checks. - [Failover](https://bloodraven.dev/docs/operations/failover) - Understand detection, promotion, DNS steering, and cooldown behavior. - [Monitoring](https://bloodraven.dev/docs/observability/monitoring) - Set up Prometheus metrics and alerting. - [Production Install](https://bloodraven.dev/docs/get-started/install-production) - Move from a first group to a production-ready install. # Playground ![playground infographic](https://bloodraven.dev/img/infographics/playground.webp) The playground deploys a fully working Bloodraven setup on your local machine so you can watch failovers happen in real time. It runs inside any multi-node Kubernetes cluster and includes a live dashboard, a counter app that proves data survives failovers, and a simulated DNS pipeline. No cloud account, no DNS provider, no production infrastructure required. The playground is the recommended learning path before production. It lets you see failover, DNS steering, taints, backup, restore, and dashboard state without touching real infrastructure. ![Bloodraven Playground showing the dashboard and counter app](https://bloodraven.dev/img/playground.png) ## What you get - **Three-site MySQL cluster** with promotable IAD/PDX sites and a dedicated non-promotable reader - **Real-time dashboard** showing site health, replication state, DNS records, and an event log - **Counter app** that writes to MySQL through the primary service, proving state persists across failovers - **Simulated external-dns pipeline** — the operator creates `DNSEndpoint` CRs, external-dns watches them and pushes records to an in-memory webhook provider, and the dashboard displays the results - **Chaos tools** for triggering failovers by killing pods, cordoning nodes, or simulating network partitions ## Prerequisites You need three tools installed: - **docker** or **podman** — for building container images. Docker is preferred because k3d's podman support is experimental and the image-load path is faster on docker. Set `BLOODRAVEN_CONTAINER_RUNTIME=podman` to force podman if both are installed. - **kubectl** — for talking to your cluster - **helm** — for deploying the operator And a local Kubernetes cluster with at least 3 worker nodes. The third worker is dedicated to the `read-only` reader so storage-loss testing is deterministic. We recommend [k3d](https://k3d.io){rel=""nofollow""} because it's fast and lightweight, but kind and minikube work too. ### Create a cluster with k3d (recommended) ```bash # Install k3d if you haven't already curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash # Create a cluster with 3 worker nodes k3d cluster create bloodraven --agents 3 ``` That's it. You now have a 4-node cluster (1 server + 3 agents) ready to go. Using kind or minikube instead **kind:** ```bash cat <: as the confirmation token. # Example: divergentGtid 61553741-443a-11f1-... → reclone-site=pdx:61553741 kubectl -n bloodraven-playground annotate mysqlfailovergroup playground \ bloodraven.shipstream.io/reclone-site=pdx:61553741 --overwrite ``` The reclone takes \~60s. Expected outcome: `state=read-only`, `replicating=true`, `recoveryState` cleared. After that the planned failover demo below will work. ## Access the apps Use `kubectl port-forward` to access the dashboard and counter app: ```bash # Dashboard (real-time cluster visualization) kubectl -n bloodraven-playground port-forward svc/dashboard 8091:8091 # Counter app (write-through-failover demo) kubectl -n bloodraven-playground port-forward svc/counter-app 8090:8090 ``` Then open {rel=""nofollow""} for the dashboard and {rel=""nofollow""} for the counter app. ::tip **Remote access** Add `--address 0.0.0.0` to `port-forward` if you want to reach the apps from another machine (e.g. over Tailscale). :: ## Try a failover The dashboard toolbar has buttons that copy `kubectl` commands to your clipboard. You can also run them directly: ```bash # Kill the IAD MySQL pod — the operator will detect the outage and fail over to PDX kubectl delete pod -n bloodraven-playground -l shipstream.io/site=iad # Or cordon the IAD node to simulate a full site outage kubectl cordon $(kubectl get nodes -l topology.kubernetes.io/zone=zone-iad -o name) ``` Watch the dashboard — you'll see the site state change, the health banner update, DNS records flip, and the counter app reconnect to the new primary. To restore the cordoned node: ```bash kubectl uncordon $(kubectl get nodes -o name | tr '\n' ' ') ``` ### Chaos script For more advanced scenarios, use the chaos script: ```bash ./playground/chaos.sh kill-site iad # Kill MySQL+Dragonfly pods at a site ./playground/chaos.sh kill-site pdx ./playground/chaos.sh cordon iad # Cordon a site's node ./playground/chaos.sh network-partition iad # Simulate a network partition ./playground/chaos.sh kill-dragonfly iad # Kill only the Dragonfly pod at a site ./playground/chaos.sh dragonfly-status # Print Dragonfly roles, traffic labels, and active endpoints ./playground/chaos.sh recover # Undo all chaos ``` ### Automated chaos suite Beyond the interactive scripts, 30+ scripted chaos scenarios (primary kills, operator crashes mid-failover, network partitions, self-fencing, GTID divergence, data wipes, rolling updates, Dragonfly failover, backup and PITR verification) are automated by the `playground-chaos` runner. Each scenario states a hypothesis, injects real faults into the cluster, asserts on operator behavior, and captures full forensics on failure: ```bash make chaos-list # List every scenario with its hypothesis make chaos-run SCENARIO=06-self-fence-isolated-primary make chaos-run SCENARIO=40-reader-data-loss-reclone make chaos-run-all-profile PROFILE=smoke # Core failover subset (~5 minutes) make chaos-run-all # The full suite ``` Scenarios 40-44 cover the dedicated read-only reader site end to end: - **40** deterministically interprets reader node/storage loss as Deployment scale-down plus PVC replacement on the dedicated reader worker. It continuously proves reader loss does not unset group `Ready`, verifies the reader client endpoint is shed during clone/catch-up, and confirms direct-source replication and endpoint return after auto-clone. - **41** answers reader `SELECT`s continuously through an unplanned failover (staleness allowed, availability required) and asserts the reader repoints directly to the new primary with no chained or blocked intermediate state. - **42** (smoke profile) stalls only the reader with `SOURCE_DELAY` and proves unbounded reader lag has zero group-level effect: no failover, no cooldown consumed, just endpoint shedding plus alertable status and metrics. - **43** makes the reader anomalously writable with an errant row: the operator fences it without debounce, a planned failover targeting it is rejected with the role error, and the errant GTID blocks source convergence at the containment gate instead of silently repointing. - **44** manually repoints the reader at the standby and proves direct-source convergence is a poll-loop invariant — any wrong-source state heals, with the documented convergence log events and no failover. This same suite is Bloodraven's E2E test bed in CI: the full suite runs nightly against a real Kubernetes cluster, and the smoke profile gates every release — a tag cannot publish images or charts unless core failover works on a live cluster. The current status is on the [E2E workflow badge](https://github.com/ShipStream/bloodraven/actions/workflows/e2e.yml){rel=""nofollow""}. ## Dragonfly co-management The playground `MysqlFailoverGroup` enables `spec.dragonfly`, so Bloodraven also creates one Dragonfly StatefulSet per site (`playground-dragonfly-iad`, `playground-dragonfly-pdx`, and `playground-dragonfly-reader`), one Dragonfly PodDisruptionBudget per site, and a single app-facing `playground-dragonfly` Service whose endpoints follow the active site. The Service selector AND-gates `shipstream.io/dragonfly-role=master` AND `shipstream.io/dragonfly-traffic=enabled`: removing the traffic label sheds an endpoint atomically, which is how planned failover avoids a window where both the old and new master would match the selector during `REPLTAKEOVER`. Disable Dragonfly by removing the `spec.dragonfly` block. Bloodraven supports Dragonfly `v1.38.0+` for managed deployments. The playground pins `docker.dragonflydb.io/dragonflydb/dragonfly:v1.38.0`. The operator does not parse the image tag: on first successful connect it probes the running command table for `REPLTAKEOVER` and reports the result on `status.dragonfly.replTakeoverSupported`. An older image that lacks the command still forms a cluster; emergency promotion then falls back to `REPLICAOF NO ONE` and increments `bloodraven_dragonfly_promotions_total{result="sessions_lost"}`. Bloodraven treats Dragonfly as cache/session state, not durable application data. The operator provisions ephemeral storage, does not manage Dragonfly backups or snapshot schedules, and does not support `DFLY LOAD` against a live master with attached replicas because loaded data bypasses the replication journal. Topology changes can force full replica resyncs and briefly increase master latency, so schedule planned failovers and Dragonfly image rollouts the same way you would schedule cache-impacting maintenance. The baseline playground does not enable `spec.dragonfly.snapshot`. That keeps normal Dragonfly pods independent of the optional RustFS/S3 path; Dragonfly v1.38 exits during startup when an S3 snapshot directory is configured but the bucket or credentials are unavailable. The D6a snapshot-restore upgrade scenario provisions the RustFS bucket on demand, temporarily enables the snapshot config, and validates that Dragonfly pods restart with the S3 snapshot directory before it requests the upgrade. The operator owns the safety-critical Dragonfly flags it emits (`--port`, `--admin_port`, `--requirepass`, `--break_replication_on_master_restart`, and related sizing knobs). Extra `spec.dragonfly.args` are an escape hatch for site-specific tuning; do not use them to enable tiered storage, ACL files, Lua compatibility relaxations, TLS replication, snapshot scheduling, or load/import workflows unless the operational tradeoff is documented for that deployment. Verify the cache subsystem before exercising failover: ```bash # Both per-site StatefulSets reach Ready kubectl -n bloodraven-playground get statefulset -l app.kubernetes.io/name=dragonfly # Operator's view of which site is master, plus pod role/traffic labels # and the active Service endpoints ./playground/chaos.sh dragonfly-status ``` For ad-hoc Redis-protocol queries (replication state, GET/SET against a key), launch a one-shot pod with `redis-cli`: ```bash kubectl -n bloodraven-playground run redis-cli --rm -it --restart=Never \ --image=redis:7-alpine -- redis-cli -h playground-dragonfly INFO replication ``` The counter app (`http://localhost:8090` after port-forward) writes to **both** MySQL and Dragonfly on every increment. The MySQL counter is durable; the Dragonfly counter (shown as "Cache (Dragonfly)" below the main number) is the session/cache continuity signal. After a planned failover with `sessionsPreserved=true`, the Dragonfly counter survives. After an emergency failover, it usually resets to 0 because the new master may have been an unsynced replica or a freshly-promoted empty pod. Exercise a planned failover that includes Dragonfly session preservation: ```bash # Click "+ Increment" a few times in the counter UI so both counters are non-zero. # Trigger a planned failover to pdx kubectl -n bloodraven-playground annotate mysqlfailovergroup playground \ bloodraven.shipstream.io/planned-failover=pdx # Watch the planned-failover status walk WaitingForLag → WaitingForDragonflySync # → PromotingDragonfly → Promoting → Resuming → Succeeded. kubectl -n bloodraven-playground get mysqlfailovergroup playground \ -o jsonpath='{.status.plannedFailover.phase}{"\n"}' # After Succeeded, sessionsPreserved should be true on the success path. kubectl -n bloodraven-playground get mysqlfailovergroup playground \ -o jsonpath='{.status.plannedFailover.dragonfly}{"\n"}' # Reload the counter UI — both the MySQL and the Dragonfly counter should # be unchanged. The active Service now resolves to the pdx Dragonfly pod. ``` Emergency Dragonfly behavior is best-effort. Killing the active MySQL pod while Dragonfly is healthy still completes the MySQL emergency failover; Dragonfly is promoted via REPLTAKEOVER (sessions preserved when reachable) or `REPLICAOF NO ONE` (sessions lost), and never blocks MySQL recovery past a 10-second budget. The D6a snapshot-restore upgrade path can be exercised directly: ```bash make chaos-run SCENARIO=29-dragonfly-snapshot-upgrade ``` The ordinary Dragonfly image rollout path is covered separately and patches the image to a cached digest reference so it does not depend on pulling a new external tag: ```bash make chaos-run SCENARIO=27-dragonfly-rolling-image-update ``` ## Exercise backup verification The playground's `minio` backup profile ships with scheduled verification enabled (`*/30 * * * *`). The operator materializes a CronJob that fires `bloodraven trigger-verification` on the schedule, which creates a `MysqlBackupVerification` CR against the latest Succeeded `MysqlBackup` and copies the profile's `verification` block (`sanityCheck`, etc.) onto the CR. To drive the feature by hand without waiting for the schedule: ```bash # Create a bare verification and wait for it to reach a terminal phase. # This skips profile-level inheritance (no sanityCheck / pointInTime); # see below for the scheduled-contract path. ./playground/verify-backup.sh run minio # List all verifications with their phase ./playground/verify-backup.sh status # Tail the Job pod log for the most recent verification ./playground/verify-backup.sh logs # Remove completed runs ./playground/verify-backup.sh cleanup # Succeeded only ./playground/verify-backup.sh cleanup --failed # Failed only # Inspect the scheduled CronJob the operator materialized ./playground/verify-backup.sh schedule-list ``` To run the **full scheduled contract** on demand — the CR built exactly the way the CronJob would build it, including the profile's `sanityCheck` — fire the operator's CronJob as a one-off Job: ```bash kubectl -n bloodraven-playground create job verify-now \ --from=cronjob/mysql-playground-verify-minio ``` A successful run provisions an ephemeral PVC + mysqld, loads the dump, runs the sanity query (if inherited from the profile), and cleans up. On failure the Pod and PVC are retained so you can `kubectl exec` in and inspect why the load failed. ::note The playground does not configure PITR binlog archival, so `spec.pointInTime` on a verification CR will be rejected by the reconciler until `spec.backup.pitr.enabled=true` (and a binlog archiver) is wired up on the failover group. :: ## Dashboard features The dashboard connects to the operator via WebSocket and polls DNS state every 3 seconds. - **Health banner** — shows overall cluster health at a glance: Healthy (green), Degraded (amber, e.g. replica not replicating), or critical states like Split Brain or No Primary (red, pulsing) - **Site cards** — shows each site's state (writable, read-only, unreachable), GTID position, replication status, and lag - **DNS records** — shows what external-dns has published, with a fallback to reading DNSEndpoint CRs directly from the Kubernetes API - **Event log** — real-time stream of state transitions, failovers, and DNS changes ## Rebuilding after code changes When you change Go code in the operator, sidecar, or any playground app, use the rebuild script to update your running cluster: ```bash # Rebuild everything ./playground/rebuild.sh # Rebuild only specific components ./playground/rebuild.sh dashboard # just the dashboard ./playground/rebuild.sh counter # just the counter app ./playground/rebuild.sh operator sidecar # operator + sidecar ``` The script builds the container images (using docker if available, otherwise podman; set `BLOODRAVEN_CONTAINER_RUNTIME=podman` to force podman), loads them into your cluster, restarts the affected deployments, and waits for rollout to complete. Valid component names: `operator`, `sidecar`, `counter`, `dashboard`, `dns-webhook`. ## Useful commands ```bash # Check the failover group status kubectl -n bloodraven-playground get mysqlfailovergroups # List all pods kubectl -n bloodraven-playground get pods # Check DNS endpoint CRs kubectl -n bloodraven-playground get dnsendpoints # Follow operator logs kubectl -n bloodraven-playground logs -l app.kubernetes.io/name=bloodraven -f ``` ## Teardown To remove everything and start fresh: ```bash ./playground/teardown.sh ``` This deletes the namespace, Helm release, CRDs, and node labels. Your k3d/kind/minikube cluster itself is left intact so you can re-run `setup.sh` without recreating it. # Production Install ![install production infographic](https://bloodraven.dev/img/infographics/install-production.webp) This guide installs Bloodraven for production. For a short first run, use [Getting Started](https://bloodraven.dev/docs/get-started/getting-started) or the [Playground](https://bloodraven.dev/docs/get-started/playground) first. ## Prerequisites - Kubernetes 1.27 or newer. - Helm 3. - Nodes labelled by site as described in [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract). - A tested StorageClass for MySQL PVCs. - external-dns and the `externaldns.k8s.io/v1alpha1` `DNSEndpoint` CRD. - cert-manager when using `spec.tls.issuerRef`. - Prometheus and Grafana if you want the shipped monitoring path. - Object storage or PVC storage for backups. - Dragonfly `v1.38.0+` image access if you enable `spec.dragonfly` for cache/session co-management. After the first probe, `status.dragonfly.replTakeoverSupported` reports whether the running image advertises `REPLTAKEOVER`. ## Namespace layout | Namespace | Contains | | ------------ | ------------------------------------------------------------------ | | `bloodraven` | Operator Deployment, RBAC, metrics Service | | `orders` | `MysqlFailoverGroup`, MySQL pods, app-facing Services, backup Jobs | | `monitoring` | Prometheus, Grafana, alert rules, dashboard ConfigMaps | ## Install CRDs GitOps users should choose one owner for CRDs. Do not let both Helm and a separate CRD application race to own the same files. ```bash helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven \ --create-namespace ``` Helm installs CRDs from the chart's `crds/` directory on first install. If CRDs are managed separately: ```bash kubectl apply -f https://raw.githubusercontent.com/ShipStream/bloodraven/main/config/crd/bases/shipstream.io_mysqlfailovergroups.yaml kubectl apply -f https://raw.githubusercontent.com/ShipStream/bloodraven/main/config/crd/bases/shipstream.io_mysqlbackups.yaml kubectl apply -f https://raw.githubusercontent.com/ShipStream/bloodraven/main/config/crd/bases/shipstream.io_mysqlbackupverifications.yaml ``` Then install the operator chart after the platform CRD app has applied the CRDs. Helm does not upgrade CRDs in `crds/`; apply CRD updates explicitly during upgrades. ## Helm values ```yaml image: repository: ghcr.io/shipstream/bloodraven tag: "1.1.0" pullPolicy: IfNotPresent replicaCount: 1 # Optional Organization license JWT. Public assertion, not a secret. # license: "eyJhbGciOiJFZERTQSIsImtpZCI6ImJyLTEiLCJ0eXAiOiJKV1QifQ...." leaderElection: enabled: true metrics: service: enabled: true serviceMonitor: enabled: true interval: 30s scrapeTimeout: 10s labels: release: kube-prometheus-stack grafanaDashboards: enabled: true namespace: monitoring folder: Bloodraven auxiliary: service: enabled: false wsAllowedOrigins: "https://dashboard.example.com" ``` Apply: ```bash helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven \ --create-namespace \ --values bloodraven-values.yaml ``` ## NetworkPolicy recommendations Allow only these paths: | Source | Destination | Port | Why | | ------------- | ---------------------------- | -------------------------------- | ----------------------------------------------------------------------------------- | | Operator | Kubernetes API | 443 | Reconcile CRDs, Services, Events, Jobs | | Operator | MySQL pods | 3306 | Health, promotion, user management | | Operator | Dragonfly Services | 6379, admin port 9999 by default | Optional Dragonfly observation, replication wiring, promotion, and snapshot upgrade | | MySQL sidecar | Kubernetes API | 443 | Lease and fencing coordination | | Prometheus | Operator metrics Service | 8080 | Metrics scraping | | Applications | Primary and replica Services | 3306 | Data plane | | Applications | Active Dragonfly Service | 6379 | Optional Redis-compatible cache/session data plane | | Backup Jobs | S3 endpoint | 443 | Backup artifacts | ## Post-install verification ```bash kubectl rollout status deployment/bloodraven -n bloodraven kubectl get crd mysqlfailovergroups.shipstream.io mysqlbackups.shipstream.io mysqlbackupverifications.shipstream.io kubectl get service -n bloodraven kubectl auth can-i list mysqlfailovergroups.shipstream.io --as=system:serviceaccount:bloodraven:bloodraven ``` ## Before production traffic - Complete [Production Hardening](https://bloodraven.dev/docs/configuration/production-hardening). - Create per-role credentials with [Credentials and TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls). - Configure [Prometheus](https://bloodraven.dev/docs/observability/monitoring-prometheus) and [Grafana](https://bloodraven.dev/docs/observability/monitoring-grafana). - Configure [S3 backups](https://bloodraven.dev/docs/backup-and-restore/backup-s3) or [PVC backups](https://bloodraven.dev/docs/backup-and-restore/backup-pvc). - If using Dragonfly, configure application Redis clients to use `-dragonfly` and validate `status.dragonfly.phase=Ready`. - Run a planned failover in a staging environment. # Production install examples ![production install examples infographic](https://bloodraven.dev/img/infographics/production-install-examples.webp) These examples are starting points for production installs. They are not a substitute for environment-specific review, but they capture the manifests operators otherwise have to assemble by hand. ::tip **Guided install** Use [Production Install](https://bloodraven.dev/docs/get-started/install-production) for the ordered production installation path. This page keeps larger snippets that are referenced from that guide. :: ## When to use these examples | Example | Use when | | --------------------- | ----------------------------------------------------------------------- | | Helm values overlay | Platform team installs the operator with monitoring and private images. | | NetworkPolicy | Cluster enforces namespace or pod network boundaries. | | Per-role credentials | New production failover group. | | Full combined example | Reviewing all moving parts in one manifest set. | ## Helm values overlay ```yaml # values-production.yaml replicaCount: 2 image: repository: registry.example.com/bloodraven tag: "1.1.0" pullPolicy: IfNotPresent imagePullSecrets: - name: private-registry leaderElection: enabled: true resources: requests: cpu: 250m memory: 256Mi limits: cpu: 1 memory: 512Mi metrics: service: enabled: true serviceMonitor: enabled: true interval: 15s labels: release: prometheus grafanaDashboards: enabled: true namespace: monitoring auxiliary: service: enabled: true type: ClusterIP wsAllowedOrigins: https://dashboard.example.com wsMaxClients: 100 ``` Install with: ```bash helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven --create-namespace \ -f values-production.yaml ``` If Argo CD owns CRDs separately, commit the CRDs under `charts/bloodraven/crds/` or your platform CRD app and install the operator chart after that CRD app syncs. Helm installs chart CRDs on first install but does not upgrade them automatically. ## NetworkPolicy The auxiliary HTTP and sidecar HTTP endpoints are intentionally internal and unauthenticated. Restrict them to the operator, dashboard, and Prometheus namespaces that need access. The sidecar also needs to reach the operator auxiliary endpoint on `:8082` for `/active-site` checks. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: bloodraven-operator-ingress namespace: bloodraven spec: podSelector: matchLabels: app.kubernetes.io/name: bloodraven policyTypes: [Ingress] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring ports: - protocol: TCP port: 8080 - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: bloodraven-dashboard - namespaceSelector: matchLabels: kubernetes.io/metadata.name: tenant-db podSelector: matchLabels: app.kubernetes.io/name: mysql app.kubernetes.io/managed-by: bloodraven shipstream.io/failover-group: orders ports: - protocol: TCP port: 8082 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: orders-mysql-ingress namespace: tenant-db spec: podSelector: matchLabels: app.kubernetes.io/name: mysql app.kubernetes.io/managed-by: bloodraven shipstream.io/failover-group: orders policyTypes: [Ingress] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: bloodraven ports: - protocol: TCP port: 3306 - protocol: TCP port: 8080 - from: - podSelector: matchLabels: app.kubernetes.io/name: mysql app.kubernetes.io/managed-by: bloodraven shipstream.io/failover-group: orders ports: - protocol: TCP port: 8080 - from: - podSelector: {} ports: - protocol: TCP port: 3306 ``` Adjust selectors to match your chart labels and namespace layout. If your CNI defaults to deny egress, also permit the operator to reach MySQL `:3306`, sidecar `:8080`, the Kubernetes API, and DNS, and permit sidecars to reach the operator auxiliary Service on `:8082`. ## PrometheusRule ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: bloodraven-alerts namespace: monitoring labels: release: prometheus spec: groups: - name: bloodraven.failover rules: - alert: BloodravenOperatorDown expr: up{job="bloodraven"} == 0 for: 2m labels: severity: critical annotations: summary: Bloodraven operator is down description: No failover decisions run while the operator is unavailable. - alert: BloodravenNoWritableSite expr: max by (namespace, group) (bloodraven_site_state{state="writable"}) == 0 for: 30s labels: severity: critical annotations: summary: No Bloodraven site is writable - alert: BloodravenFailoverOccurred expr: increase(bloodraven_failovers_total[5m]) > 0 labels: severity: warning annotations: summary: Bloodraven failover promoted {{ $labels.target_site }} - alert: BloodravenDivergentTransactions expr: bloodraven_divergent_transactions > 0 labels: severity: critical annotations: summary: Divergent GTIDs detected on {{ $labels.site }} description: Review data loss and reclone the diverged site before rejoining it. - alert: BloodravenReplicationLagging expr: bloodraven_replication_lag_seconds{role=~"primary-candidate|dr-only"} > 300 for: 2m labels: severity: warning annotations: summary: Bloodraven replication lag is high on {{ $labels.group }}/{{ $labels.site }} ({{ $labels.role }}) - name: bloodraven.backup rules: - alert: BloodravenBackupStale expr: time() - bloodraven_backup_last_success_timestamp_seconds > 86400 for: 15m labels: severity: warning annotations: summary: Bloodraven backup is stale for {{ $labels.group }}/{{ $labels.profile }} - alert: BloodravenBackupVerificationStale expr: time() - bloodraven_backup_verified_timestamp_seconds > 172800 for: 15m labels: severity: warning annotations: summary: Bloodraven backup verification is stale - alert: BloodravenPITRArchiveLagging expr: bloodraven_archiver_backlog_files > 0 for: 10m labels: severity: warning annotations: summary: Bloodraven PITR archiver backlog exists on {{ $labels.site }} - alert: BloodravenPITRUploadFailures expr: increase(bloodraven_archiver_upload_failures[15m]) > 0 labels: severity: warning annotations: summary: Bloodraven PITR upload failures on {{ $labels.site }} ``` Tune thresholds to your RPO/RTO. For example, lower the backup freshness threshold if your backup schedule runs hourly. ## Cloudflare external-dns Bloodraven writes `DNSEndpoint` CRs. Cloudflare-specific credentials and zone configuration live in your external-dns deployment, not in Bloodraven. Example external-dns arguments: ```yaml args: - --source=crd - --crd-source-apiversion=externaldns.k8s.io/v1alpha1 - --crd-source-kind=DNSEndpoint - --provider=cloudflare - --cloudflare-proxied=false - --domain-filter=az.example.com - --txt-owner-id=bloodraven-prod - --policy=sync ``` Use DNS-only records (`--cloudflare-proxied=false`) for MySQL. Cloudflare proxying is HTTP-oriented and is not appropriate for raw MySQL traffic. Store the Cloudflare token in a Secret consumed by external-dns. The token should be scoped to the specific zone Bloodraven manages. ## k3s storage guidance - Do not use the default local-path provisioner for production MySQL. - Use a topology-aware network storage provider that can bind volumes to the intended site and survive node replacement. - Prefer `volumeBindingMode: WaitForFirstConsumer` so Kubernetes waits for scheduling constraints before binding a PVC. - Use `reclaimPolicy: Retain` for MySQL data PVCs. - Test PVC deletion and reclone in a staging cluster before onboarding a tenant. Example StorageClass shape: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: mysql-retained provisioner: example.csi.driver volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true reclaimPolicy: Retain parameters: type: ssd ``` ## Example failover group hardening ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: tenant-db spec: image: container-registry.oracle.com/mysql/community-server:9.7 sidecarImage: ghcr.io/shipstream/bloodraven-sidecar:1.1.0 updateStrategy: OrderedUpdate failoverCooldown: 5m replication: maxLagSeconds: 60 credentials: operatorSecret: orders-mysql-operator appSecret: orders-mysql-app readOnlySecret: orders-mysql-readonly monitorSecret: orders-mysql-monitor backupSecret: orders-mysql-backup dns: hostname: orders.az.example.com ttl: 30 backup: pitr: enabled: true profileName: minio maxBinlogSize: 64M profiles: - name: minio storage: type: s3 s3: bucket: shipstream-backups prefix: orders endpoint: https://s3.example.com region: us-east-1 credentialsSecret: orders-backup-s3 encryption: passphraseSecret: name: orders-backup-passphrase key: passphrase verification: enabled: true schedule: "17 */6 * * *" sanityCheck: query: "SELECT COUNT(*) FROM information_schema.tables" expect: minRows: 1 sites: - name: iad zone: iad-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.1.10 storage: storageClassName: mysql-retained size: 500Gi - name: pdx zone: pdx-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.0.2.10 storage: storageClassName: mysql-retained size: 500Gi ``` Pair this with the [production hardening checklist](https://bloodraven.dev/docs/configuration/production-hardening) and [known limitations](https://bloodraven.dev/docs/get-started/known-limitations) before deploying real tenant data. # Configuration Overview ![configuration infographic](https://bloodraven.dev/img/infographics/configuration.webp) Use this section when you already know what Bloodraven does and need to choose the right configuration surface. ## Pick the right page | Need | Start here | | ---------------------------------------- | -------------------------------------------------------------------------------------- | | Smallest working manifest | [Getting Started](https://bloodraven.dev/docs/get-started/getting-started) | | Production installation path | [Production Install](https://bloodraven.dev/docs/get-started/install-production) | | Every CRD field and default | [CRD Reference](https://bloodraven.dev/docs/configuration/crd-reference) | | Per-role users and MySQL TLS | [Credentials and TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls) | | Node labels, zones, and taints | [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract) | | Application connection behavior | [App Integration](https://bloodraven.dev/docs/configuration/app-integration) | | Security boundaries and network surfaces | [Security Model](https://bloodraven.dev/docs/configuration/security-model) | | Go-live requirements | [Production Hardening](https://bloodraven.dev/docs/configuration/production-hardening) | ## Configuration model Bloodraven has three layers: | Layer | Resource | Owner | | ------------------ | ------------------------------------------------------------- | ----------------------------------------- | | Operator install | Helm release, CRDs, RBAC, metrics Service | Platform team | | Failover group | `MysqlFailoverGroup` | Database/platform team | | Application access | MySQL Services, optional Dragonfly Service, Secrets, DNS name | Application team with platform guardrails | ## MySQL configuration precedence Bloodraven renders a separate deterministic MySQL ConfigMap for every site. The effective option order is: 1. Bloodraven built-in defaults. 2. Normalized `spec.mysqlConf` group overrides. 3. Normalized `spec.sites[].mysqlConf` overrides for that site. 4. Bloodraven-owned replication safety settings, reasserted last. Option names normalize `_` to `-` before every merge. For example, `max_connections` and `max-connections` address the same effective key; do not define both spellings in one map and expect two settings. A site spelling overrides the equivalent normalized group key. Final keys are sorted, so semantically equivalent maps render and hash identically. The operator always owns `gtid-mode`, `enforce-gtid-consistency`, `log-bin`, `log-replica-updates`, `skip-replica-start`, and clone plugin loading. User values cannot weaken these replication invariants. Each Deployment mounts `mysql---config`. A rendered-config hash, not the raw maps, drives drift: changing or deleting a site override rolls that site while leaving unaffected sites alone. Ordered updates restart every drifted non-active follower sequentially after checking read-only state, direct source, and replication threads. If the active site also drifted, a healthy promotable standby takes over before the old active is restarted. A reader-only or other non-active-only change never causes failover, and neither readers nor `dr-only` sites are chosen as handoff targets. Upgrades from the legacy group ConfigMap are level-driven. Bloodraven creates all per-site maps first, rolls live Deployment references one site at a time, and deletes the legacy map only after every Deployment references its expected site map. Reconciliation can resume safely after an operator restart at any point in this migration. ## Service inheritance and isolation `spec.serviceTemplate` defaults the client-facing site Services and the `-primary` and `-replicas` Services. A site's `spec.sites[].serviceTemplate` overrides only that site's client Service: - Non-zero site `type`, `externalTrafficPolicy`, and `nodePort` values override inherited values. - Group and site annotations merge, with site keys winning. Site annotations never leak to siblings, `-primary`, `-replicas`, or internal Services. - `externalTrafficPolicy` is legal only when the effective Service type is `NodePort` or `LoadBalancer`. The group value also applies to primary and replicas Services when their type supports it. - `nodePort` is site-only, legal only for `NodePort` or `LoadBalancer`, and applies only to the named MySQL port. Kubernetes admission enforces the cluster's configured allocation range. Omitting a site NodePort preserves its compatible allocated MySQL NodePort while the Service remains external. Switching between `NodePort` and `LoadBalancer` preserves that compatible allocation; switching to `ClusterIP` clears external-only fields. Bloodraven preserves Kubernetes-assigned Service identity and annotations it does not own. For every site, Bloodraven also creates `mysql---internal`. It is always `ClusterIP`, ignores all user Service templates, publishes not-ready addresses, and exposes MySQL plus the sidecar for operator, replication, clone, backup, and peer traffic. The client-facing Service exposes MySQL only. When Go clients dial the internal Service with `spec.tls`, Bloodraven retains the client-facing site hostname as the TLS `ServerName`, so existing certificates do not need another DNS name. Operators that enable strict hostname verification inside MySQL-native clone or replication must account for the internal Service hostname separately. ## Recommended production baseline - Use `spec.credentials` instead of legacy `spec.secretName`. - Use distinct Secrets for operator, app, read-only, monitor, and backup users. - Set `spec.tls` and require client TLS in application connection strings. - Pin operator, sidecar, MySQL, and backup images. - If you enable `spec.dragonfly`, pin `spec.dragonfly.image` to Dragonfly `v1.38.0+` and route applications to the Bloodraven-created active Dragonfly Service. Confirm `status.dragonfly.replTakeoverSupported=true` after the first probe — Bloodraven reports the capability; it does not reject an older tag. - Configure backups before production traffic starts. - Enable Prometheus scraping, Grafana dashboards, and alerts. - Link alerts to [Runbooks](https://bloodraven.dev/docs/operations/runbooks). ## Resource defaults Bloodraven now sets a default `resources.requests` on every container it creates, so the operator's workloads can be scheduled on clusters that enforce a `ResourceQuota` or a `LimitRange` requiring requests. Where the user already has a CR override field, that field continues to win; only previously-unset containers picked up a default. | Workload | Container | Resource source | | ---------------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------- | | MySQL Deployment (per site) | `mysql` | `spec.sites[].resources` (user-supplied) | | MySQL Deployment (per site) | `sidecar` | `spec.sidecarResources` (user-supplied) | | Dragonfly StatefulSet | `dragonfly` | `spec.dragonfly.resources` (user-supplied) | | Backup Job | `mysqlsh` / `mysqlsh-dump` / `backup-encrypt-upload` | `spec.backup.resources` | | Cleanup Job | `mysqlsh` | `spec.backup.resources` | | Verification Job | `mysqlsh` | `spec.verification.resources` | | Verification Job init | `decrypt-download` | `spec.backup.resources` when set, else `100m` CPU / `128Mi` memory (requests only) | | Restore Job (bootstrap + in-place) | `mysqlsh` | `spec.backup.resources` when set, else `100m` CPU / `128Mi` memory (requests only) | | Restore Job init | `pitr-download`, `decrypt-download` | `spec.backup.resources` when set, else `100m` CPU / `128Mi` memory (requests only) | | Operator (Helm) | `operator` | `.Values.resources` in `charts/bloodraven/values.yaml` | The `100m`/`128Mi` init-container default matches the operator's own request in the Helm chart. It is intentionally conservative: the AES-GCM decrypt stream and the PITR archive walker tolerate small requests when the container can burst on the host. Override by setting `spec.backup.resources` on the failover group; the same value flows to the init containers. The cleanup Job reuses `spec.backup.resources` because it runs the same `cleanup.py` payload sized by the same backup profile parameters. If a future workload needs a separate budget, we add a dedicated field then. ## Security context defaults Two new opt-in fields let cluster operators apply Kubernetes Restricted PSS to the MySQL Deployments and Dragonfly StatefulSets without forcing a uid/gid migration on existing clusters: - `spec.podSecurityContext` and `spec.containerSecurityContext` — applied to the MySQL Deployment pod and to its `mysql` and `sidecar` containers. - `spec.dragonfly.podSecurityContext` and `spec.dragonfly.containerSecurityContext` — applied to the Dragonfly StatefulSet. When these fields are nil (the default), Bloodraven leaves the workload's PodSpec untouched; existing CRs render identical pods to before. When they are set, the operator applies the value verbatim — it does **not** merge with internal hardened defaults. The user owns the uid, gid, fsGroup, seccomp profile, and capability set, because the right values depend on which MySQL and Dragonfly images you run. See [Production Hardening → Opt-in Restricted PSS for MySQL and Dragonfly](https://bloodraven.dev/docs/configuration/production-hardening#opt-in-restricted-pss-for-mysql-and-dragonfly) for worked examples and the upgrade procedure for existing clusters whose `/var/lib/mysql` PVCs were created without an `fsGroup`. ## Related pages - [Production Install](https://bloodraven.dev/docs/get-started/install-production) - [Credentials and TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls) - [CRD Reference](https://bloodraven.dev/docs/configuration/crd-reference) - [Production Hardening](https://bloodraven.dev/docs/configuration/production-hardening) # Production hardening checklist ![production hardening infographic](https://bloodraven.dev/img/infographics/production-hardening.webp) Work through this before labeling a Bloodraven deployment "production". Each bullet links to the page with the setting's detail. Items are grouped; within a group, order doesn't matter. Requirement levels: | Level | Meaning | | -------------------- | -------------------------------------------------------------- | | Required | Must be satisfied before production traffic. | | Recommended | Strong default unless your platform has an equivalent control. | | Environment-specific | Required only when that feature or environment applies. | Owner hints: | Owner | Typical responsibilities | | -------- | -------------------------------------------------------------------- | | Platform | Operator install, CRDs, RBAC, node labels, NetworkPolicy, monitoring | | Database | MySQL sizing, backup, restore, PITR, failover drills | | App team | Connection pools, DNS caching, write/read endpoint use, app alerts | | Security | Secrets, TLS, image provenance, network boundaries | ## Image supply chain - Pin `spec.image`, `spec.sidecarImage`, and `spec.backup.image` to **immutable tags** (or digests). No `:latest`, no MySQL `:9`, no `:edge`. A silently-drifting MySQL image breaks dump/restore and sometimes replication. See the [Upgrade and version-skew policy](https://bloodraven.dev/docs/operations/upgrade-policy#supported-mysql-versions) for which MySQL tags are supported. - **Mirror the operator, sidecar, and backup images to a private registry.** A DockerHub rate-limit or Oracle registry outage should never be able to stop a pod from starting mid-incident. - Verify images are pulled using `imagePullSecrets` scoped to the namespace; service-account-wide pulls leak credentials across workloads. - Enable `imagePullPolicy: IfNotPresent` once tags are immutable. - If `spec.dragonfly.enabled=true`, pin `spec.dragonfly.image` to Dragonfly `v1.38.0+`, mirror it with the other runtime images, and do not use `:latest`. Confirm `status.dragonfly.replTakeoverSupported=true` after the first probe — Bloodraven reports the capability and does not reject an older tag. ## Credentials - Use **`spec.credentials`** (per-role Secrets), not the legacy `spec.secretName` DSN mode. Per-role credentials are required for the backup-verification and PITR paths to hold least privilege. See [CRD Reference → CredentialsSpec](https://bloodraven.dev/docs/configuration/crd-reference#credentialsspec). - Populate `operatorSecret`, `appSecret`, `readOnlySecret`, `monitorSecret`, and `backupSecret` — five distinct users. - Store Secrets with a provider that supports rotation (External Secrets Operator, sealed-secrets, Vault). Rotate regularly. - Supply `MYSQL_REPLICATION_USER` / `MYSQL_REPLICATION_PASSWORD` explicitly. Without them, automatic old-primary recovery is skipped — see [Failover → Prerequisites](https://bloodraven.dev/docs/operations/failover#prerequisites). ## TLS - Set `spec.tls` with a cert-manager issuer. Bloodraven forces `require-secure-transport` when TLS is configured, so all client traffic is encrypted in-cluster. - Use a separate issuer for MySQL TLS from the one your mesh / ingress uses, so rotating the MySQL cert doesn't churn unrelated workloads. ## Failover tuning - Set `spec.failoverCooldown` to **≥ 5 m** (the default). Lower values increase the risk of cascade failovers during unstable infrastructure; higher values extend MTTR after a flapping primary comes back. - Set `spec.replication.maxLagSeconds` based on your RPO tolerance. This is an alerting threshold (`Degraded=True` with reason `ReplicationLagging`), not a promotion gate — treat it as a page-worthy signal, not a soft warning. - Use **`spec.updateStrategy: OrderedUpdate`** (the default). The alternative, `Recreate`, restarts both MySQL pods simultaneously on any spec change and creates a write outage. Ordered updates roll the replica first, fail over, then roll the former primary — zero write downtime for typical spec changes. - If `spec.dragonfly.enabled=true`, choose `spec.dragonfly.plannedFailover.onSyncTimeout` deliberately. The default `proceed` protects MySQL availability and may discard cache/session continuity; `fail` preserves the old active site when Dragonfly session preservation cannot be proven. ## Storage - Use a **network-attached, topology-aware StorageClass** for the MySQL data PVCs (EBS + `WaitForFirstConsumer`, Persistent Disk, Ceph RBD, etc.). Never `hostPath` or `emptyDir`. - Confirm the StorageClass's `reclaimPolicy` is `Retain` for the MySQL data PVCs. `Delete` + a routine `kubectl delete pvc` turns a pod-restart case into a full reclone. - Confirm the StorageClass supports `allowVolumeExpansion: true` if you anticipate growing storage. Resizing without expansion support means a data migration. ## PITR / backups - Enable PITR (`spec.backup.pitr.enabled: true`) with a `maxBinlogSize` sized for your RPO target. Default `100M` balances rotation overhead against unarchived-tail size; smaller if your write rate is high. - Pick a backup profile storage that is **in a different fault domain** from the MySQL PVCs (different region / different provider). Backups stored on the same disk as the DB are not backups. - Configure at least one recurring schedule under `spec.backup.schedules[]` and pin its `timeZone`. See [Backup and restore → TimeZone](https://bloodraven.dev/docs/backup-and-restore/backup-restore#timezone). - Set a **retention policy** (`retentionPolicy.count`, `maxAgeDays`, `minKeep`, `maxFailedKeep`) on every profile. An unbounded retention is a bill, not a backup strategy. - Run backup-verification externally (restore into a throwaway namespace periodically, assert a sanity query works). Unverified backups are Schrödinger backups. - **Encrypt backups at rest.** Set `spec.backup.profiles[].encryption.passphraseSecret` on every profile that writes to shared storage. Client-side envelope encryption (AES-256-GCM) keeps the passphrase in a Kubernetes-controlled Secret, so a compromised S3 credential does not expose backup contents. See [Backup encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption). Back the passphrase Secret up out-of-band — losing it makes the backups unrecoverable. ## Placement - **Label nodes** with a fault-domain zone topology label (e.g. `topology.kubernetes.io/zone=zone-iad`). Bloodraven's per-site `spec.sites[].zone` targets nodes via `nodeAffinity` — without correct node labels the sites will co-locate. - Set per-site `resources.requests` generously enough that MySQL has breathing room; `resources.limits` optional for memory only if you're also using `innodb_buffer_pool_size` tuning. - Honour the [placement contract](https://bloodraven.dev/docs/configuration/placement-contract): apps must respect the `shipstream.io/db-readonly-` node taint that Bloodraven applies to losing sites after a failover. ## Observability - Scrape `bloodraven` on `:8080/metrics`and alert on: - `bloodraven_state_transitions_total` spikes - `bloodraven_failovers_total` increase (at least a warning) - `bloodraven_replication_lag_seconds` sustained > your threshold - `bloodraven_divergent_transactions > 0` (critical — data loss) - `up{job="bloodraven"} == 0` for > 2 × `pollInterval` (operator is down; see [Monitoring](https://bloodraven.dev/docs/observability/monitoring#available-metrics) for the full list and recommended rules) - `bloodraven_dragonfly_site_up == 0`, `bloodraven_dragonfly_promotions_total{result="failed"}`, and `bloodraven_dragonfly_promotions_total{result="sessions_lost"}` when `spec.dragonfly.enabled=true` - Ship a `PrometheusRule` resource alongside your install with the above expressions baked in. The Helm chart installs a `PodMonitor` but not alerting rules; curate your own. - Forward Kubernetes Events for `MysqlFailoverGroup` and `MysqlBackup` CRs somewhere humans read. See [Monitoring → Kubernetes Events](https://bloodraven.dev/docs/observability/monitoring#kubernetes-events). - Scrape your sidecars too: PITR archiver status is surfaced via Prometheus gauges the operator exports (labeled by site). - Publish / log dashboards — Grafana, otherwise the first post-incident question becomes "was this visible?". ## Control-plane availability - Keep the operator's pod up. Use a tight liveness probe and `restartPolicy: Always`. Size the `PodDisruptionBudget` so the operator isn't evicted during routine node drains — the shipped chart does **not** ship a PDB for the operator; add one. - Leader election is on by default and stays on. Running multiple operator replicas is fine but doesn't speed up MTTR; the design tradeoffs are in the README's "Non-HA control plane" note. ## Network - Configure NetworkPolicy so the operator Pod can reach both MySQL Services and the sidecar HTTP endpoint on `:8080`. A silently-blocking CNI is the #1 source of spurious `unreachable` states in the state machine. - If `spec.dragonfly.enabled=true`, allow the operator to reach the per-site Dragonfly client and admin ports (`6379` and `9999` by default), and allow applications to reach only the active `-dragonfly` Service. - Make sure `external-dns` (or whatever consumes the `DNSEndpoint` CR) is running. Failover updates the CR, but the actual DNS record only changes when the external-dns pod reconciles. Monitor its lag. - If you're using TLS with a private CA, mount the CA into the sidecar and operator pods so their MySQL connections trust it. ## Opt-in Restricted PSS for MySQL and Dragonfly The MySQL Deployments and Dragonfly StatefulSets render with **no** pod-level or container-level `securityContext` by default, preserving backward compat with existing PVCs whose files were created under the image's default uid. Existing clusters keep their current PodSpec unchanged on upgrade. When you are ready to put the MySQL and Dragonfly pods under the [Kubernetes Restricted Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted){rel=""nofollow""}, fill in the new opt-in fields on the `MysqlFailoverGroup` CR. The operator applies the value **verbatim** to the workload — it does not merge it with hardened defaults, because the right uid/gid depend on the image you run. The examples below target the upstream `mysql:9.7` and Dragonfly images, both of which run as uid 999. ### MySQL — `spec.podSecurityContext` / `spec.containerSecurityContext` ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: podSecurityContext: runAsNonRoot: true runAsUser: 999 runAsGroup: 999 fsGroup: 999 fsGroupChangePolicy: OnRootMismatch seccompProfile: type: RuntimeDefault containerSecurityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: false capabilities: drop: - ALL seccompProfile: type: RuntimeDefault # ...rest of spec unchanged ``` The container's `readOnlyRootFilesystem` is `false` because the upstream `mysql` image writes temp files outside the data PVC (`/tmp`, `/var/run/mysqld`, `/etc/mysql/conf.d` for entrypoint scratch). Leaving the root filesystem writable here is intentional; the PVC is what matters for durability. If your image supports a read-only root with explicit emptyDir mounts for those paths, set `readOnlyRootFilesystem: true` and add the mounts via your own pod template overlay. The `fsGroupChangePolicy: OnRootMismatch` setting tells Kubernetes to skip the recursive `chown` on the data volume when the top-level directory is already owned by `fsGroup`. On a fresh PVC the chown runs once; on a PVC that was already migrated it is a no-op. See [Upgrading existing clusters](https://bloodraven.dev/#upgrading-existing-clusters) below for the migration path on a PVC that was created without an `fsGroup`. `spec.containerSecurityContext` is applied to both the `mysql` and the `sidecar` containers. The two containers share a uid/gid because they share the data volume. ### Dragonfly — `spec.dragonfly.podSecurityContext` / `spec.dragonfly.containerSecurityContext` ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: dragonfly: enabled: true podSecurityContext: runAsNonRoot: true runAsUser: 999 runAsGroup: 999 fsGroup: 999 fsGroupChangePolicy: OnRootMismatch seccompProfile: type: RuntimeDefault containerSecurityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL seccompProfile: type: RuntimeDefault # ...rest of dragonfly spec unchanged ``` Dragonfly **does** support a read-only root filesystem in this configuration: its only writable path is the data volume where it persists its RDB snapshot. Set `readOnlyRootFilesystem: true`. ### Upgrading existing clusters If you already run Bloodraven in production and your MySQL data PVCs were created **before** these fields existed, the files on disk are owned by whatever uid the image originally ran as (uid 999 for upstream `mysql:9.7`, something else for a fork). Naively applying `fsGroup: 999` to a PVC created by a pod that ran as uid 0 will trigger a recursive chown on every pod start that can take many minutes on a large data volume — Kubernetes holds the pod in `ContainerCreating` for the duration, which on a primary is a write outage. Two safe migration paths: **Option A — let Kubernetes handle it once with `OnRootMismatch`.** If you are willing to take one extended pod-start window per pod, set `fsGroupChangePolicy: OnRootMismatch` (as in the example above) and apply the spec change. Kubernetes will run the recursive chown the first time each pod restarts, then notice the root directory is already correct and skip on subsequent starts. Roll the StatefulSet one pod at a time during a maintenance window; do **not** restart both sites at once. **Option B — chown the data offline first.** For very large volumes (1 TB+) or operations teams that prefer to control the chown, run an offline Job against each MySQL PVC **after** scaling the StatefulSet down to zero, then apply the spec change. Example: ```yaml apiVersion: batch/v1 kind: Job metadata: name: orders-iad-chown namespace: orders spec: template: spec: restartPolicy: Never containers: - name: chown image: registry.k8s.io/build-image/debian-base:bookworm-v1.0.2 command: - sh - -c - chown -R 999:999 /var/lib/mysql volumeMounts: - name: data mountPath: /var/lib/mysql securityContext: runAsUser: 0 volumes: - name: data persistentVolumeClaim: # Adjust the claim name to the MysqlFailoverGroup's per-site PVC. claimName: data-orders-iad-0 ``` Apply the Job per site, wait for it to complete, then apply the `MysqlFailoverGroup` spec change and scale the StatefulSet back up. Repeat on the second site after the first one is healthy. ### Service account tokens on backup, restore, cleanup, and verification Jobs Backup, restore, restore-in-place, cleanup, and verification execution Jobs now render with `automountServiceAccountToken: false`. None of these jobs call the Kubernetes API at runtime — they execute `mysqlsh`, `cleanup.py`, or `verify.sh` and communicate with MySQL over TCP — so the SA token has no purpose and is one less credential to leak from a compromised pod. Schedule-trigger CronJob pods (the `MysqlBackup` / `MysqlBackupVerification` creator pods, not the execution Jobs they create) keep the SA token because they POST a backup or verification CR through the in-cluster API and need their `ServiceAccount`'s create permission. ## Disaster readiness - Rehearse failover on the playground (`./playground/chaos.sh kill-site iad`) and confirm your application behaves. - Rehearse PVC loss end-to-end at least once per quarter — delete a PVC, watch the operator auto-clone, confirm data is intact. - Write down the non-obvious bits that only you know (donor secrets, runbook phone numbers, which engineer holds which credential). A production system that only one person can recover is a liability. - Periodically check that the PITR archive has current manifests and that `status.pitr.newestArchivedTime` is close to "now". A stale archive is a silent RPO regression. ## Change management - Validate CRD spec changes with `kubectl apply --dry-run=server` before rolling them to production. The CRD has several opinionated defaults that can silently re-align your config on `apply`. - When rotating the sidecar image independently of the MySQL image (`spec.sidecarImage`), the `OrderedUpdate` path handles it automatically. Don't `kubectl delete pod` to force the rollout — let the operator drive it. - Never force-promote a site by patching CR status directly. The operator reads status as *truth* from the data plane; tampering with it leads to split-brain. Use the manual-promotion procedure in [Operations → Manual promotion](https://bloodraven.dev/docs/operations/operations#manual-promotion-fallback-operator-unreachable) if you truly need to override. ## Go-live gate Copy this checklist into the production release ticket: - CRDs installed and owned by the correct deployment system. - Operator image pinned. - Sidecar image pinned. - MySQL image pinned. - Backup image pinned. - Dragonfly image pinned if enabled. - Node labels correct for every site. - StorageClass tested for scheduling, expansion, and reclaim behavior. - external-dns tested with the production DNS provider. - TLS enabled for MySQL. - Per-role credentials configured. - Backups scheduled. - Restore verified. - PITR enabled if required by RPO. - Prometheus scraping enabled. - Grafana dashboards installed. - Minimum alerts installed and linked to runbooks. - NetworkPolicy installed. - Failover tested. - Application connection behavior tested. - Dragonfly active Service and Redis-client reconnect behavior tested if enabled. - On-call runbooks reviewed by responders. # CRD Reference ![crd reference infographic](https://bloodraven.dev/img/infographics/crd-reference.webp) Complete reference for Bloodraven custom resources (`shipstream.io/v1alpha1`). ## CRDs at a glance | Kind | Short name | Scope | Description | | ------------------------- | ---------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `MysqlFailoverGroup` | `mfg` | Namespaced | Core resource. Declares a multi-site MySQL HA cluster with automatic failover. | | `MysqlBackup` | `mb` | Namespaced | One backup run. Created ad-hoc or by a schedule CronJob. Tracks phase, location, GTID, and binlog coordinates. | | `MysqlBackupVerification` | `mbv` | Namespaced | Periodic restore-of-latest-backup into a throwaway mysqld to prove a backup loads cleanly. See [Backup verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification). | | `MysqlStandbyCluster` | `msc` | Namespaced | **Phase 1 of WISHLIST #7.** Passive DR relationship descriptor: lives on the DR cluster, continuously monitors the source archive bucket, and stamps readiness conditions. See [Multi-cluster DR](https://bloodraven.dev/docs/architecture/multi-cluster-dr). | --- ## `MysqlFailoverGroup` Complete reference for the `MysqlFailoverGroup` custom resource (`shipstream.io/v1alpha1`). ## Common fields quick reference | Field | Use most often for | Guide | | ---------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `spec.sites` | Site topology, placement, storage, LB IPs | [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract) | | `spec.credentials` | Per-role MySQL users | [Credentials And TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls) | | `spec.tls` | MySQL TLS Secret and issuer | [Credentials And TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls) | | `spec.dns` | external-dns hostname and TTL | [Production Install](https://bloodraven.dev/docs/get-started/install-production) | | `spec.backup` | Backup profiles, schedules, PITR | [Backup Overview](https://bloodraven.dev/docs/backup-and-restore/backup-overview) | | `spec.initFromBackup` | Bootstrap restore into a new group | [Backup And Restore](https://bloodraven.dev/docs/backup-and-restore/backup-restore) | | `spec.plannedFailover` | Defaults for planned failover API | [Planned Failover](https://bloodraven.dev/docs/operations/planned-failover) | | `spec.dragonfly` | Optional Dragonfly cache/session co-management | [App Integration](https://bloodraven.dev/docs/configuration/app-integration#connection-endpoints) | ::caution **Destructive and high-risk fields** Review runbooks before changing `spec.restoreInPlace`, restore-related `confirm` tokens, reclone controls, and any field that changes `spec.sites`, `spec.credentials`, `spec.tls`, or MySQL images. Restore and reclone operations can replace data; image and credential changes can trigger restarts or failed reconnects. :: ## Spec ### Top-level fields | Field | Type | Required | Default | Description | | ------------------------------- | -------------------- | -------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | string | No | `mysql:9.7` | MySQL container image | | `sidecarImage` | string | No | `ghcr.io/shipstream/bloodraven-sidecar:1.1.0` | Bloodraven sidecar container image | | `sites` | []SiteSpec | Yes | -- | Site definitions. At least two sites with `role: primary-candidate`; additional `dr-only` and `read-only` followers may be appended. MinItems=2, MaxItems=16. | | `secretName` | string | No | -- | **(Legacy)** Name of the Secret containing MySQL credentials (key: `dsn`). Mutually exclusive with `credentials`. | | `credentials` | CredentialsSpec | No | -- | Per-role MySQL credential management. The operator creates MySQL users with role-appropriate privileges and rotates passwords when secrets change. Mutually exclusive with `secretName`. | | `dns` | DNSSpec | Yes | -- | DNS configuration for traffic steering via external-dns | | `tls` | TLSSpec | No | -- | TLS configuration (cert-manager integration) | | `pollInterval` | duration | No | `2s` | Base interval for polling each MySQL instance. After a site reaches `failureThreshold` and continues failing, the shared group cadence backs off exponentially to a 30-second cap. | | `failureThreshold` | int | No | `3` | Consecutive failed polls before a site is marked unreachable | | `recoveryThreshold` | int | No | `2` | Consecutive successful polls before a site is marked writable | | `failoverCooldown` | duration | No | `5m` | Minimum time between automatic failovers | | `connectionDrainTimeout` | duration | No | `30s` | Maximum operator-side retry window for evicting stale application sessions from a fenced former primary after promotion | | `splitBrainPolicy` | SplitBrainPolicySpec | No | -- | Opt-in automated resolution when multiple sites are writable and the operator has no prior failover history. See [SplitBrainPolicySpec](https://bloodraven.dev/#splitbrainpolicyspec). | | `sidecar` | SidecarSpec | No | -- | Sidecar-container knobs (lease timeout, peer check interval, Bloodraven aux endpoint). | | `sidecarResources` | ResourceRequirements | No | -- | Compute resources (requests/limits) for the sidecar container | | `terminationGracePeriodSeconds` | int | No | `60` | Grace period for MySQL container shutdown. | | `mysqlConf` | map [string] string | No | -- | MySQL configuration overrides merged into the generated `my.cnf`. Option keys may use underscores or hyphens; Bloodraven normalizes underscores to hyphens before merging. Bloodraven defaults `log-bin-trust-function-creators=1` so replicated function DDL is accepted consistently at every site. Changes trigger an ordered rolling restart when `updateStrategy: OrderedUpdate` is set. | | `replication` | ReplicationSpec | No | -- | Replication health settings | | `updateStrategy` | string | No | `OrderedUpdate` | Update strategy (`OrderedUpdate` or `Recreate`). | | `cloneTimeout` | int | No | `3600` | Timeout in seconds for CLONE INSTANCE (session `net_*_timeout` and `clone_ddl_timeout`). A change is applied on the next reconcile and used by the next clone/reclone; the topology manager does not need to restart. | | `podLabels` | map [string] string | No | -- | Additional labels applied to every MySQL pod. Operator labels take precedence on conflict. | | `podAnnotations` | map [string] string | No | -- | Additional annotations applied to every MySQL pod. Operator annotations take precedence on conflict. | | `serviceTemplate` | ServiceTemplate | No | -- | Defaults the client-facing site, primary, and replicas Services. Internal per-site Services ignore this template and remain ClusterIP. | | `extraContainers` | []Container | No | -- | Additional containers injected into every MySQL pod (e.g. exporters) | | `extraInitContainers` | []Container | No | -- | Additional init containers injected after the operator's built-in init container | | `backup` | BackupSpec | No | -- | Backup configuration (profiles, schedules, retry, PITR, encryption, security contexts). See [BackupSpec](https://bloodraven.dev/#backupspec). | | `initFromBackup` | InitFromBackupSpec | No | -- | One-shot restore-on-first-boot before normal bootstrap completes. See [InitFromBackupSpec](https://bloodraven.dev/#initfrombackupspec-specinitfrombackup). | | `restoreInPlace` | RestoreInPlaceSpec | No | -- | Re-triggerable destructive restore against the active primary. Bumping `confirm` to a newer RFC 3339 timestamp re-arms. See [RestoreInPlaceSpec](https://bloodraven.dev/#restoreinplacespec-specrestoreinplace). | | `plannedFailover` | PlannedFailoverSpec | No | -- | Cluster-wide defaults for the graceful planned-failover API (triggered via `bloodraven.shipstream.io/planned-failover=`). | | `dragonfly` | DragonflySpec | No | -- | Optional per-site Dragonfly cache/session co-management. When enabled, the operator creates one Dragonfly StatefulSet per site, an active Redis-compatible Service, replication wiring, promotion status, and optional snapshot-restore upgrade support. See [DragonflySpec](https://bloodraven.dev/#dragonflyspec-specdragonfly). | | `encryptionAtRest` | EncryptionAtRestSpec | No | -- | MySQL InnoDB data-at-rest encryption via the GPL `component_keyring_file` keyring component, with the live keyring projected from a per-site Secret so no key material lands on the data PVC or a node disk. Requires `spec.tls`. See [EncryptionAtRestSpec](https://bloodraven.dev/#encryptionatrestspec-specencryptionatrest) and [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest). | | `license` | string | No | -- | Optional Ed25519-signed JWT recording this group's license. Public assertion, not a credential. Verified offline; never gates functionality. Overrides the operator-level license. See [Licensing](https://bloodraven.dev/docs/licensing). | ### SiteSpec Each entry in `spec.sites` defines one site in the failover group. | Field | Type | Required | Default | Description | | ------------------- | -------------------- | ----------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Yes | -- | Unique site identifier (e.g. `iad`, `pdx`). Used in Service names and labels. MaxLength=253. | | `role` | string | No | `primary-candidate` | `primary-candidate` can be promoted; `dr-only` and `read-only` are never promoted. A reader provides a health-gated site-specific read pool. At least two `primary-candidate` sites are required. | | `zone` | string | Yes | -- | Availability zone or region for pod scheduling | | `taintNodeSelector` | map [string] string | Conditional | -- | Required and non-empty for `primary-candidate` and `dr-only`; optional for `read-only`. Selects nodes that receive this group's `db-readonly` taint. Accepted but ignored on readers, which never cause taint operations. | | `lbIP` | string | Conditional | -- | Required and non-empty for `primary-candidate` and `dr-only`; optional for `read-only`. Used for active-primary DNS steering. Accepted but ignored on readers, which are never DNS targets. | | `storage` | StorageSpec | Yes | -- | Persistent storage configuration for this site | | `resources` | ResourceRequirements | No | -- | Compute resources (requests/limits) for the MySQL container at this site | | `mysqlConf` | map [string] string | No | -- | Site-specific MySQL options applied after normalized group overrides. Changing this map rolls only affected sites unless other drift is pending. | | `serviceTemplate` | SiteServiceTemplate | No | -- | Scalar and annotation overrides for this site's client-facing MySQL Service. Does not affect sibling, primary, replicas, or internal Services. | Reader manifests may retain legacy `lbIP` and `taintNodeSelector` values for compatibility, but Bloodraven ignores them. Changing a reader to `primary-candidate` or `dr-only` is rejected until both fields are present. A reader does not satisfy the two-candidate minimum and cannot appear in `splitBrainPolicy.sitePriorities`. ### Admission combinations | Combination | Admitted? | | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | | Two `primary-candidate` sites, each with non-empty `lbIP` and `taintNodeSelector` | Yes | | Two candidates plus a reader with both placement fields omitted | Yes | | Reader with either or both legacy placement fields supplied | Yes; supplied values are ignored | | Omitted/default role, explicit candidate, or `dr-only` with either placement field omitted | No | | Fewer than two candidates, or a reader/DR site in `sitePriorities` | No | | `externalTrafficPolicy` on an effective `ClusterIP` Service | No | | Site `nodePort` on an effective `ClusterIP` Service | No | | ETP on `NodePort`/`LoadBalancer`, or site NodePort in 1-65535 on those types | Yes; Kubernetes then applies its cluster-specific NodePort range | ### StorageSpec | Field | Type | Required | Description | | ------------------ | ------ | -------- | ----------------------------------- | | `storageClassName` | string | Yes | Kubernetes StorageClass name | | `size` | string | Yes | Storage request size (e.g. `100Gi`) | ### DNSSpec | Field | Type | Required | Description | | ---------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `hostname` | string | Yes | The DNS hostname managed by Bloodraven (e.g. `orders.az.example.com`). The operator creates/updates a `DNSEndpoint` CR with an A record pointing this hostname to the active site's `lbIP`. A change is applied on the next DNS reconcile (the topology poll, default 2s) and replaces `dnsName` on the existing `DNSEndpoint`; the operator does not need to restart. Apps should CNAME their DNS to this hostname. | | `ttl` | int | No | TTL in seconds for the DNS A record (default: `60`). A change is applied on the next DNS reconcile, same as `hostname`. | ### TLSSpec | Field | Type | Required | Description | | ------------ | --------- | -------- | -------------------------------- | | `issuerRef` | IssuerRef | Yes | cert-manager issuer reference | | `secretName` | string | Yes | Name of the TLS Secret to create | ### IssuerRef | Field | Type | Required | Description | | ------ | ------ | -------- | ---------------------------- | | `name` | string | Yes | Issuer or ClusterIssuer name | | `kind` | string | Yes | `Issuer` or `ClusterIssuer` | ### ReplicationSpec | Field | Type | Required | Description | | ----------------------- | ---- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `maxLagSeconds` | int | No | Maximum acceptable replication lag in seconds before the replica is considered unhealthy | | `readOnlyMaxLagSeconds` | int | No | Maximum lag for a reader's client endpoint. Omitted inherits `maxLagSeconds` (including its 300-second effective fallback); explicit `0` requires zero reported lag. | ### SplitBrainPolicySpec Opt-in configuration for how the operator should behave when more than one site is simultaneously writable and the operator cannot infer a winner from prior failover history. See [Split-brain resolution](https://bloodraven.dev/docs/operations/failover#split-brain-resolution) for a full description of semantics and tradeoffs. | Field | Type | Required | Description | | ---------------- | -------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sitePriorities` | []string | No | Ordered list of `primary-candidate` site names. The operator walks the list and promotes the first entry that is currently writable; every other writable site is fenced. Empty or unset falls back to manual resolution (alert only). MaxItems=16. Entries must reference sites with role `primary-candidate`. | ### ServiceTemplate | Field | Type | Required | Description | | ----------------------- | ------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | No | Kubernetes Service type: `ClusterIP` (default), `LoadBalancer`, or `NodePort` | | `externalTrafficPolicy` | string | No | `Cluster` or `Local`. Valid only when the effective Service type is `NodePort` or `LoadBalancer`. Applied to site, primary, and replicas Services when supported. | | `annotations` | map [string] string | No | Default annotations for client-facing site, primary, and replicas Services. Not applied to internal Services. | ### SiteServiceTemplate (`spec.sites[].serviceTemplate`) | Field | Type | Required | Description | | ----------------------- | ------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | No | Site Service override: `ClusterIP`, `LoadBalancer`, or `NodePort`. Omitted inherits the group type. | | `externalTrafficPolicy` | string | No | Site override: `Cluster` or `Local`. Legal only when the effective site type is `NodePort` or `LoadBalancer`. | | `nodePort` | int | No | Requested NodePort for this site's named `mysql` port, range 1-65535. Legal only for effective type `NodePort` or `LoadBalancer`; cluster admission enforces its configured allocation range. | | `annotations` | map [string] string | No | Merged over group annotations for this site only; site keys win. | The operator preserves Kubernetes-assigned Service identity and compatible allocated NodePorts across reconciles and external-type transitions. Moving a site Service to `ClusterIP` clears external-only fields. The sidecar port is never exposed through a client Service or assigned a NodePort. ### CredentialsSpec Exactly one of `secretName` or `credentials` must be set. When `credentials` is used, the operator manages MySQL users on the primary — creating them during first boot via an init script and updating passwords via `ALTER USER` when the referenced Secrets change. | Field | Type | Required | Description | | ---------------- | ------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `operatorSecret` | string | Yes | Secret for operator and sidecar connections. Required keys: `username`, `password`, `MYSQL_ROOT_PASSWORD`. Optional: `MYSQL_REPLICATION_USER`, `MYSQL_REPLICATION_PASSWORD` (default to operator credentials). Grants: `ALL PRIVILEGES WITH GRANT OPTION`. | | `appSecret` | string | No | Secret for application read-write connections. Keys: `username`, `password`. Grants: `ALL PRIVILEGES` (no `GRANT OPTION`, no `SUPER`). | | `readOnlySecret` | string | No | Secret for application read-only connections. Keys: `username`, `password`. Grants: `SELECT`, `SHOW VIEW`, `SHOW DATABASES`, `PROCESS`. | | `monitorSecret` | string | No | Secret for Prometheus exporter connections. Keys: `username`, `password`. Grants: `PROCESS`, `REPLICATION CLIENT`, `SELECT` on `performance_schema`. | | `backupSecret` | string | No | Secret for backup/restore connections. Keys: `username`, `password`. Grants: `SELECT`, `LOCK TABLES`, `SHOW VIEW`, `EVENT`, `TRIGGER`, `RELOAD`, `BACKUP_ADMIN`, `REPLICATION CLIENT`. | ### BackupSpec Top-level backup configuration, embedded as `spec.backup`. Omitting the field disables backups for this failover group. | Field | Type | Required | Default | Description | | -------------------------- | ---------------------- | -------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | string | No | `container-registry.oracle.com/mysql/community-server:9.7` | Image containing `mysqlsh` used for dump / load. | | `imagePullSecrets` | []LocalObjectReference | No | -- | Pull secrets for the backup image. | | `profiles` | []BackupProfile | No | -- | Reusable named backup configurations referenced by schedules and `MysqlBackup` CRs. | | `schedules` | []BackupSchedule | No | -- | Cron-driven recurring backups; each references a profile by name. Each entry becomes a Kubernetes CronJob owned by this failover group. | | `maxLagSecondsForSource` | int | No | `300` | Threshold above which the backup reconciler falls back from replica-first source selection to the primary. | | `resources` | ResourceRequirements | No | -- | Resources for the backup Job's mysqlsh container. | | `activeDeadlineSeconds` | int | No | `7200` | Wall-clock cap on a single backup Job. | | `backoffLimit` | int | No | `2` | Job-level backoffLimit applied to backup Jobs. | | `retry` | BackupRetrySpec | No | -- | Operator-level retries for Failed scheduled MysqlBackup CRs. Independent of Job `backoffLimit`. | | `pitr` | PITRSpec | No | -- | Continuous binlog archival for point-in-time recovery. | | `podSecurityContext` | PodSecurityContext | No | -- | Overrides default hardened pod security context merged onto the defaults. | | `containerSecurityContext` | SecurityContext | No | -- | Overrides default hardened container security context. | | `stagingVolumeSizeLimit` | Quantity | No | -- | Cap on the plaintext-staging `emptyDir` used by backup / restore / verify Jobs. Unset falls through to the node's ephemeral-storage limit; set on shared clusters to keep a large dump from triggering `DiskPressure` (AUDIT H6). | ### BackupProfile (`spec.backup.profiles[]`) | Field | Type | Required | Default | Description | | ----------------- | -------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Yes | -- | Profile identifier, `[a-z0-9]([-a-z0-9]*[a-z0-9])?`. | | `storage` | BackupStorage | Yes | -- | Tagged union: `type: S3` + `s3` block, or `type: PVC` + `pvc` block. | | `dump` | DumpOptions | No | -- | `util.dumpInstance()` tuning (threads, chunk size, compression, schema filters, consistent locking, OCIMDS checks). | | `retention` | int | No | `7` | Legacy shorthand: max number of successful MysqlBackup CRs to keep. Ignored when `retentionPolicy` is set. | | `retentionPolicy` | RetentionPolicy | No | -- | Structured retention: `count`, `maxAgeDays`, `minKeep` floor, `maxFailedKeep` cap. Replaces the `retention` shorthand. | | `verification` | VerificationSpec | No | -- | Opt-in periodic restore-of-this-profile's-latest-backup into a throwaway MySQL instance; renders a CronJob when `enabled: true`. | | `encryption` | BackupEncryptionSpec | No | -- | Enables client-side envelope encryption (AES-256-GCM + HKDF-SHA256) for every dump file and, when this profile is also used for PITR, every archived binlog. Passphrase is owned by the operator — rotating or deleting the Secret renders existing ciphertexts unrecoverable. See [Backup encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption). | ### BackupEncryptionSpec | Field | Type | Required | Default | Description | | ------------------ | ------------------- | -------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `algorithm` | string | No | `AES-256-GCM` | Encryption algorithm. Only `AES-256-GCM` is currently supported. | | `passphraseSecret` | PassphraseSecretRef | Yes | -- | Reference to a Secret in the same namespace holding the passphrase. See [PassphraseSecretRef](https://bloodraven.dev/#passphrasesecretref). | ### PassphraseSecretRef Reused by `BackupEncryptionSpec`, `BackupDecryptionSpec`, `spec.initFromBackup.decryption`, and `spec.restoreInPlace.decryption`. | Field | Type | Required | Default | Description | | ------ | ------ | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Yes | -- | Name of the Secret. | | `key` | string | No | `passphrase` | Key within the Secret that holds the passphrase bytes. Leading and trailing whitespace is stripped when the passphrase is read. | ### BackupDecryptionSpec Used on `spec.initFromBackup.decryption` and `spec.restoreInPlace.decryption` to supply the passphrase that decrypts an encrypted backup. When the source is a same-group `mysqlBackupRef` and the field is omitted, the operator falls back to the profile's own `passphraseSecret`. | Field | Type | Required | Description | | ------------------ | ------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `passphraseSecret` | PassphraseSecretRef | Yes | Secret holding the passphrase used at backup time. Rotating to a new value does not re-wrap existing ciphertext — it will simply fail to decrypt. | ### RetentionPolicy Fields are independently optional; a successful MysqlBackup is kept iff **any** enabled check says keep. `minKeep` is the safety floor that prevents a retention sweep from wiping the last good backup after a long outage of failing attempts. | Field | Type | Required | Default | Description | | --------------- | ---- | -------- | ------- | -------------------------------------------------------------------------------------------------------- | | `count` | int | No | `0` | Max number of successful CRs to keep. 0 disables count-based pruning. | | `maxAgeDays` | int | No | `0` | Max age in days of a successful CR before it becomes eligible for pruning. 0 disables age-based pruning. | | `minKeep` | int | No | `1` | This many newest successful CRs are always kept, regardless of the other knobs. | | `maxFailedKeep` | int | No | `10` | Cap on the Failed bucket, independent of the success retention policy. | ### PITRSpec (`spec.backup.pitr`) Enables continuous binary-log archival so restores can target an arbitrary `stopDatetime` on top of any retained full dump. See [Backup and restore → Point-in-time recovery](https://bloodraven.dev/docs/backup-and-restore/backup-restore#point-in-time-recovery-pitr) for the end-to-end story. | Field | Type | Required | Description | | --------------------- | -------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `enabled` | bool | No | Turns archival on. Omitting the whole block is equivalent to `false`. | | `profileName` | string | Yes when enabled | Name of a `spec.backup.profiles[]` entry. Binlog objects and per-site manifests live under a `binlogs/` subprefix inside that profile's storage. | | `maxBinlogSize` | string | No | Passed to MySQL as `max_binlog_size`. Controls rotation cadence (and therefore the RPO gap). Default `"100M"`. | | `archivePollInterval` | duration | No | Belt-and-suspenders scan cadence alongside inotify. Default `"60s"`. | ### InitFromBackupSpec (`spec.initFromBackup`) One-shot restore-on-first-boot. The operator runs a restore Job against the initial primary site before normal bootstrap is considered complete. Once `status.restore.phase == Succeeded`, subsequent reconciles skip the restore even if this field is still set. | Field | Type | Required | Description | | ------------- | -------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `source` | InitFromBackupSource | Yes | Tagged reference: exactly one of `mysqlBackupRef`, `s3`, or `pvc`. | | `loadOptions` | LoadOptions | No | `util.loadDump()` tuning (threads, resetProgress, skipBinlog, loadIndexes, schema filters). | | `pointInTime` | PointInTimeSpec | No | Post-load binlog replay to a specific target timestamp. Requires the source group to have had `spec.backup.pitr.enabled=true` at backup time. | | `decryption` | BackupDecryptionSpec | No | Passphrase to decrypt the source backup and any replayed binlogs. For same-group `mysqlBackupRef`, the operator falls back to the profile's own passphrase when omitted. | ### RestoreInPlaceSpec (`spec.restoreInPlace`) Re-triggerable destructive restore against the currently-active primary. Unlike `initFromBackup` (one-shot, greenfield), this field is meant to be edited repeatedly: bumping `confirm` to a newer RFC 3339 timestamp re-arms another restore. See `RestoreInPlaceSpec` in the source for full semantics (full-instance vs. per-schema restores). | Field | Type | Required | Description | | ------------- | -------------------- | -------- | -------------------------------------------------------------------------------------------------------------------- | | `confirm` | string | Yes | Anti-fat-finger token. Must be an RFC 3339 timestamp strictly greater than `status.restoreInPlace.confirmTokenUsed`. | | `source` | InitFromBackupSource | Yes | Tagged reference: one of `mysqlBackupRef`, `s3`, or `pvc`. | | `loadOptions` | LoadOptions | No | `util.loadDump()` tuning. For per-schema restores, `skipBinlog` is force-set to `false` so the DROP+load replicates. | | `pointInTime` | PointInTimeSpec | No | Post-load binlog replay. | | `decryption` | BackupDecryptionSpec | No | Same semantics as `InitFromBackupSpec.Decryption`. | ### PointInTimeSpec (`spec.initFromBackup.pointInTime`) Requests binlog replay on top of the loaded dump. Requires the source failover group to have had `spec.backup.pitr.enabled=true` at the time of the full dump — otherwise there is no archive to replay. | Field | Type | Required | Description | | -------------- | ------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `stopDatetime` | string | Yes | Replay target. Accepts RFC 3339 (`2026-04-15T09:30:00Z`) or MySQL's native form (`2026-04-15 09:30:00`). | | `excludeGtids` | string | No | Forwarded verbatim as `mysqlbinlog --exclude-gtids=...` to surgically skip known-bad transactions while replaying everything else. | ### DragonflySpec (`spec.dragonfly`) Optional cache/session sidekick management. Omitting this field or setting `enabled: false` preserves MySQL-only behavior and removes previously managed Dragonfly resources. | Field | Type | Required | Default | Description | | ----------------- | ---------------------------- | ---------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | bool | No | `false` | Creates and manages one Dragonfly StatefulSet per site when true. | | `image` | string | Yes when enabled | -- | Pinned Dragonfly image. `:latest` is rejected. Supported pin is Dragonfly `v1.38.0+`. Bloodraven reports whether the running image advertises `REPLTAKEOVER` on `status.dragonfly.replTakeoverSupported`; it does not admit or deny based on the tag. | | `port` | int | No | `6379` | Redis-compatible client port. | | `adminPort` | int | No | `9999` | Admin/control port used by the operator. | | `maxMemoryMb` | int | No | -- | Renders Dragonfly `--maxmemory=mb` when non-zero. | | `proactorThreads` | int | No | -- | Renders Dragonfly `--proactor_threads=` when non-zero. | | `args` | []string | No | -- | Extra Dragonfly args. Operator-owned flags such as `--port`, `--admin_port`, `--requirepass`, `--dir`, S3 flags, and `--break_replication_on_master_restart` are filtered out. | | `auth` | DragonflyAuthSpec | No | -- | Secret reference for the Dragonfly password. The operator renders `--requirepass` and authenticates its own connections. | | `resources` | ResourceRequirements | No | -- | Compute resources for each Dragonfly container. | | `plannedFailover` | DragonflyPlannedFailoverSpec | No | `{maxSyncWait: 30s, onSyncTimeout: proceed}` | Planned-failover sync and timeout behavior. | | `snapshot` | DragonflySnapshotSpec | No | -- | Native Dragonfly snapshot directory and optional S3-compatible settings for planned snapshot-restore upgrades. | #### DragonflyAuthSpec | Field | Type | Required | Default | Description | | ------------- | ------ | -------- | ---------- | ----------------------------------------- | | `secretName` | string | Yes | -- | Secret containing the Dragonfly password. | | `passwordKey` | string | No | `password` | Secret key containing the password. | #### DragonflyPlannedFailoverSpec | Field | Type | Required | Default | Description | | --------------- | -------- | -------- | --------- | -------------------------------------------------------------------------------------------------------------------- | | `maxSyncWait` | duration | No | `30s` | Maximum time to wait for target Dragonfly replica catch-up. Also used as the `REPLTAKEOVER` timeout. | | `onSyncTimeout` | string | No | `proceed` | `proceed` continues MySQL promotion and records `sessionsPreserved=false`; `fail` rolls back before MySQL promotion. | #### DragonflySnapshotSpec | Field | Type | Required | Description | | ----------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------ | | `dir` | string | No | Passed to Dragonfly as `--dir`. Use `s3://bucket[/prefix]` for S3-compatible snapshot/restore support. | | `serviceAccountName` | string | No | ServiceAccount assigned to Dragonfly pods for cloud IAM access. | | `credentialsSecretName` | string | No | Secret projected as AWS credential environment variables for S3-compatible snapshot access. | | `s3Endpoint` | string | No | S3-compatible endpoint passed as `--s3_endpoint`. | | `s3UseHTTPS` | bool | No | Passed as `--s3_use_https` when set. | | `s3SignPayload` | bool | No | Passed as `--s3_sign_payload` when set. | ### EncryptionAtRestSpec (`spec.encryptionAtRest`) Requires `spec.tls`: MySQL mandates a secure connection to clone encrypted data and Bloodraven bootstraps every replica with `CLONE INSTANCE`. The CRD rejects `enabled: true` without it. | Field | Type | Required | Default | Description | | ---------- | ---------------------- | -------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | bool | Yes | `false` | Turn on data-at-rest encryption for every site. Refused on a group that is already serving unless the `bloodraven.shipstream.io/encryption-adopt=confirm` annotation is set. | | `coverage` | EncryptionCoverageSpec | No | all `true` | Which encryption settings the operator enforces. See [EncryptionCoverageSpec](https://bloodraven.dev/#encryptioncoveragespec). | | `keyring` | KeyringSpec | No | -- | Keyring component paths and escrow lifecycle. See [KeyringSpec](https://bloodraven.dev/#keyringspec). | ### EncryptionCoverageSpec Every field defaults to `true`. These are operator-owned once encryption is enabled: a `spec.mysqlConf` entry naming the same option is ignored, so a stray override cannot silently downgrade a site. Turning one off narrows the security claim. | Field | Type | Default | MySQL setting | | ------------------ | ---- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tables` | bool | `true` | `default_table_encryption`. Applies to schemas and tables created afterwards; does not rewrite existing data. | | `privilegeCheck` | bool | `true` | `table_encryption_privilege_check`. Stops application users opting individual tables out of encryption. | | `redoLog` | bool | `true` | `innodb_redo_log_encrypt` | | `undoLog` | bool | `true` | `innodb_undo_log_encrypt` | | `binaryLog` | bool | `true` | `binlog_encryption` (covers relay logs on replicas too) | | `systemTablespace` | bool | `true` | `ALTER TABLESPACE mysql ENCRYPTION='Y'`, run once by the sidecar on the writable site. Not covered by `default_table_encryption`; without it the data dictionary (schema, table, and column names) stays readable on a stolen PVC. | ### KeyringSpec `mysqldDir` and `pluginDir` are image-specific. The defaults match the official `mysql:9.x` images — MySQL only reads its global component manifest from the directory holding `mysqld`, and its global component config from `plugin_dir`, so a wrong value shows up as InnoDB refusing to start. | Field | Type | Default | Description | | ---------------------- | ------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `dataFileDir` | string | `/run/mysql-keyring` | Keyring mount point. It must be outside the actual MySQL datadir. Admission rejects `/var/lib/mysql` and descendants; custom-image datadirs require manual verification. | | `mysqldDir` | string | `/usr/sbin` | Directory containing the `mysqld` binary; `mysqld.my` is subPath-mounted here. | | `pluginDir` | string | `/usr/lib64/mysql/plugin` | MySQL `plugin_dir`; `component_keyring_file.cnf` is subPath-mounted here. | | `retainVersions` | int | `5` | Superseded escrow Secret versions kept per site (2--50). The version a site is currently sealed against is never pruned. | | `escrowTimeoutSeconds` | int | `600` | How long before a site that has not escrowed its keyring is reported `Failed`. Minimum 30. The site stays unsealed and the operator keeps retrying either way. | ## Full example ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: image: mysql:9.7 sidecarImage: ghcr.io/shipstream/bloodraven-sidecar:1.1.0 sites: - name: iad zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.1.1 storage: storageClassName: fast-ssd size: 100Gi resources: requests: cpu: "2" memory: "16Gi" limits: cpu: "4" memory: "16Gi" - name: pdx zone: us-west-2a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.0.2.1 storage: storageClassName: fast-ssd size: 100Gi resources: requests: cpu: "2" memory: "16Gi" limits: cpu: "4" memory: "16Gi" - name: reader role: read-only zone: us-east-1b mysqlConf: innodb_buffer_pool_size: "4G" serviceTemplate: type: LoadBalancer externalTrafficPolicy: Local nodePort: 32006 annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" storage: storageClassName: fast-ssd size: 100Gi secretName: mysql-credentials sidecarResources: requests: cpu: "100m" memory: "64Mi" limits: cpu: "200m" memory: "128Mi" dns: hostname: orders.az.example.com ttl: 60 tls: issuerRef: name: letsencrypt kind: ClusterIssuer secretName: mysql-tls pollInterval: 2s failureThreshold: 3 recoveryThreshold: 2 failoverCooldown: 5m mysqlConf: innodb_buffer_pool_size: "8G" max_connections: "500" innodb_flush_log_at_trx_commit: "1" sync_binlog: "1" replication: maxLagSeconds: 300 readOnlyMaxLagSeconds: 30 updateStrategy: OrderedUpdate dragonfly: enabled: true image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.38.0 plannedFailover: maxSyncWait: 30s onSyncTimeout: proceed cloneTimeout: 3600 podLabels: cost-center: platform podAnnotations: prometheus.io/scrape: "true" prometheus.io/port: "9104" serviceTemplate: type: LoadBalancer externalTrafficPolicy: Local annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" extraContainers: - name: mysqld-exporter image: prom/mysqld-exporter:latest env: - name: DATA_SOURCE_NAME valueFrom: secretKeyRef: name: mysql-monitor-creds key: dsn ports: - containerPort: 9104 ``` ## Full example (credentials mode) ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: image: mysql:9.7 sidecarImage: ghcr.io/shipstream/bloodraven-sidecar:1.1.0 sites: - name: iad zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.1.1 storage: storageClassName: fast-ssd size: 100Gi - name: pdx zone: us-west-2a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.0.2.1 storage: storageClassName: fast-ssd size: 100Gi credentials: operatorSecret: mysql-operator-creds appSecret: mysql-app-creds monitorSecret: mysql-monitor-creds backupSecret: mysql-backup-creds dns: hostname: orders.az.example.com ttl: 60 ``` Each referenced Secret must contain `username` and `password` keys. The operator secret additionally requires `MYSQL_ROOT_PASSWORD`: ```yaml apiVersion: v1 kind: Secret metadata: name: mysql-operator-creds stringData: username: bloodraven password: MYSQL_ROOT_PASSWORD: --- apiVersion: v1 kind: Secret metadata: name: mysql-app-creds stringData: username: app password: ``` ## Status The operator writes the following status fields. ### Top-level status | Field | Type | Description | | ----------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `activeSite` | string | Name of the site currently serving as primary | | `sites` | []SiteStatus | Per-site status | | `conditions` | []Condition | Standard Kubernetes conditions | | `lastFailover` | timestamp | When the last failover occurred | | `lastFailoverTarget` | string | Which site was promoted in the last failover | | `promotionGtidExecuted` | string | GTID set recorded on the candidate at the moment of the most recent promotion, before it began accepting writes. Used for data-loss accounting. | | `updatePhase` | string | Current phase of an ordered update; empty when not updating. | | `restore` | RestoreStatus | In-flight or completed `initFromBackup` restore (phase, JobName, targetSite, startTime, completionTime, message). | | `restoreInPlace` | RestoreInPlaceStatus | In-flight or completed in-place restore (phase, JobName, targetSite, scope, confirmTokenUsed, startTime, completionTime, message). Mirrors the state machine described on [RestoreInPlaceSpec](https://bloodraven.dev/#restoreinplacespec-specrestoreinplace). | | `backupSchedules` | []BackupScheduleStatus | Per-schedule rollup (cronJobName, lastScheduleTime, lastSuccessfulTime, lastBackupName, lastBackupPhase, lastSuccessfulBackupName, lastRetryAttempt, nextRetryTime). | | `lastBackupTime` | timestamp | Completion time of the most recent successful MysqlBackup across all profiles, regardless of whether it was scheduled or on-demand. | | `pitr` | PITRStatus | Summary of the continuous binlog archive, populated by the operator from periodic polls of each sidecar's `/archiver/status` endpoint. Present when `spec.backup.pitr.enabled=true`. | | `plannedFailover` | PlannedFailoverStatus | Most-recent planned (admin-triggered) failover attempt. Retained until a newer annotation replaces it; how `kubectl describe` tells the story of a switchover after the fact. | | `dragonfly` | DragonflyStatus | Observed Dragonfly subsystem state when `spec.dragonfly.enabled=true`: active site, phase, per-site roles, reachability, replication health, last promotion, REPLTAKEOVER capability probe, and snapshot-upgrade status. | | `encryptionAtRest` | EncryptionAtRestStatus | Observed data-at-rest encryption state when `spec.encryptionAtRest.enabled=true`: whether every site is sealed, and per-site keyring phase, escrow Secret/version/digest, and observed coverage. See [EncryptionAtRestStatus](https://bloodraven.dev/#encryptionatreststatus). | ### SiteStatus | Field | Type | Description | | --------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Site identifier | | `state` | string | Current state: `writable`, `read-only`, `unreachable`, or `unknown` | | `lastSeen` | timestamp | Last time the operator successfully polled this site | | `replicating` | bool | Whether this site is replicating from another site | | `secondsBehindSource` | int | Replication lag in seconds (only set when replicating) | | `gtidExecuted` | string | GTID set executed on this site | | `sourceHost` | string | Source hostname reported by MySQL. Followers are expected to name the active primary's internal site Service. | | `sourceConvergenceState` | string | `Converged`, `Pending`, or `Blocked`. Generic direct-source state for every follower; independent of old-primary recovery state. | | `sourceConvergenceReason` | string | Stable reason: `DirectSource`, `SourceMismatch`, `ProbeFailed`, `MutationFailed`, or `GTIDDiverged`. Detailed errors remain in structured logs. | | `recoveryState` | string | Old-primary recovery state: empty (no recovery needed), `RecoveryInProgress` (the operator is reconfiguring or waiting for the site to stabilize as a replica), or `RecoveryBlocked` (divergent transactions detected — trigger a reclone to recover) | | `divergentGtid` | string | GTID set of transactions on this site that diverge from the current primary. Populated when `recoveryState` is `RecoveryBlocked`. | | `divergentTransactionCount` | int | Number of divergent transactions. Populated when `recoveryState` is `RecoveryBlocked`. | ### Conditions | Type | Meaning | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Ready` | `True` when there is exactly one confirmed writable `primary-candidate` and every non-reader follower has healthy direct replication. Reader reachability, lag, threads, and source state do not change this group condition. | | `Degraded` | `True` when core candidate/DR replication is broken or source-mismatched, lag exceeds `maxLagSeconds`, or a non-reader site is unreachable. Reader failures are excluded and remain visible per site and in metrics. | | `RecoveryPending` | `True` while old-primary recovery is running (`RecoveryInProgress`) or when an old primary has divergent transactions (`DivergentTransactions`). Use the reclone annotation only for `DivergentTransactions`. | | `Bootstrapping` | `True` when a clone operation is in progress (fresh-deploy, auto-clone, or reclone). Reason indicates the phase. | | `EncryptionAtRestReady` | `True` when every site is sealed against a verified escrowed keyring. `False` with reason `KeyringNotSealed` while sites are bootstrapping, cloning, or rotating; `KeyringFailed` when a site could not escrow or its escrow Secret is missing or corrupt; `AdoptionRefused` when encryption was enabled on a group that is already serving. | ### EncryptionAtRestStatus | Field | Type | Description | | -------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sealed` | bool | `True` only when every site is in the `Sealed` phase. **This is the field to alert on** -- `false` means at least one site is running with a writable keyring or failed to escrow. | | `sites` | []SiteEncryptionStatus | Per-site keyring lifecycle. | | `observedGeneration` | int | CR generation the encryption subsystem last reconciled. | ### SiteEncryptionStatus | Field | Type | Description | | ---------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Site identifier | | `phase` | string | `Pending`, `Unsealed`, `Escrowed`, `Sealed`, or `Failed`. The normal sequence is `Pending -> Unsealed -> Escrowed -> Sealed`. `Escrowed` means the keyring is captured and verified and the pod is rolling onto the sealed rendering. | | `unsealReason` | string | Why the site is unsealed: `Bootstrap`, `Clone`, or `Rotation`. Empty when sealed. | | `keyringSecret` | string | Escrow Secret the site is sealed against (or seeded from while unsealed). | | `keyringVersion` | int | Escrow version. Monotonic per site; each version is an immutable Secret. | | `keyringDigest` | string | `sha256:` over the escrowed keyring bytes. The operator refuses to seal until the sidecar reports the same digest for the live keyring file. | | `lastEscrowTime` | timestamp | When the current version was accepted. | | `unsealedSince` | timestamp | When the site entered `Unsealed`. Drives `keyring.escrowTimeoutSeconds`. | | `message` | string | Human-readable explanation of the current phase. | | `coverage` | SiteEncryptionCoverage | What was actually observed on the live instance. | ### SiteEncryptionCoverage Sampled from the live instance, not inferred from `my.cnf`. A site can be `Sealed` and still have incomplete coverage -- for example an adopted cluster whose pre-existing tables were never rebuilt. | Field | Type | Description | | --------------------------- | --------- | ------------------------------------------------------------------------------------------------------------- | | `keyringComponent` | string | `Component_name` from `performance_schema.keyring_component_status`. | | `keyringReadOnly` | bool | The component's `Read_only` status. Must be `true` in the `Sealed` phase. | | `systemTablespaceEncrypted` | bool | Whether the `mysql` tablespace is encrypted. | | `unencryptedTablespaces` | int | User tablespaces still reporting `ENCRYPTION='N'`. **Non-zero means part of your data is in the clear.** | | `redoLogEncrypted` | bool | `@@innodb_redo_log_encrypt` | | `undoLogEncrypted` | bool | `@@innodb_undo_log_encrypt` | | `binlogEncrypted` | bool | `@@binlog_encryption` | | `lastCheckTime` | timestamp | When coverage was last sampled. | ### DragonflyStatus | Field | Type | Description | | -------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | bool | Mirrors `spec.dragonfly.enabled` at last observation. | | `activeSite` | string | Site currently acting as Dragonfly master. Usually matches `status.activeSite`; may briefly diverge during planned failover. | | `phase` | string | `Disabled`, `Reconciling`, `ConfiguringReplication`, `Ready`, `Degraded`, or `Promoting`. | | `message` | string | Human-readable status summary. | | `lastPromotionTime` | timestamp | Last successful Dragonfly promotion time. | | `lastPromotionTarget` | string | Site promoted in the last successful Dragonfly promotion. | | `replTakeoverSupported` | bool | `true` when every successfully probed reachable site advertised `REPLTAKEOVER`. `false` when at least one probed site lacks it. Omitted until the first successful probe. | | `replTakeoverProbeTime` | timestamp | When `replTakeoverSupported` or `replTakeoverProbeMessage` last changed. | | `replTakeoverProbeMessage` | string | Explains a `false` result (which sites lacked the command). | | `sites` | []DragonflySiteStatus | Per-site Dragonfly observation. | | `upgrade` | DragonflyUpgradeStatus | Snapshot-restore upgrade progress when requested by annotation. | ### DragonflySiteStatus | Field | Type | Description | | ------------------ | ------ | --------------------------------------------------------------------------------- | | `name` | string | Site identifier. | | `role` | string | `master`, `replica`, `stale-master`, `unconfigured`, `unreachable`, or `unknown`. | | `reachable` | bool | Whether the operator completed an `INFO replication` call on the latest poll. | | `serviceName` | string | Site-local Dragonfly Service name. | | `podName` | string | Dragonfly pod name when known. | | `replicationState` | string | Raw Dragonfly `role` value from `INFO replication`. | | `linkStatus` | string | Replica `master_link_status`; empty on a master. | | `syncInProgress` | bool | Whether Dragonfly reports a full sync in progress. | | `lastIOSecondsAgo` | int | Seconds since the replica last received data from its master; `-1` means never. | | `ready` | bool | Operator-level readiness for the Dragonfly role. | | `message` | string | Human-readable per-site status. | ### Annotations | Annotation | Description | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bloodraven.shipstream.io/last-failover` | Operator-owned out-of-band copy of `status.lastFailover`, formatted as an RFC3339 UTC timestamp at second precision. Do not edit manually. Reset tooling clears it together with status history. | | `bloodraven.shipstream.io/last-failover-target` | Operator-owned out-of-band copy of `status.lastFailoverTarget`. It is always written and cleared as a pair with `last-failover`. Do not edit manually. | | `bloodraven.shipstream.io/reclone-site` | Triggers a `CLONE INSTANCE` from the current primary to the named site. Value format depends on the target's state: `:confirm=` for a cold reclone (no divergent GTID recorded), or `:` when `status.sites[].divergentGtid` is non-empty — the prefix must be 8+ characters and match the observed divergent GTID. Both forms act as a fat-finger interlock (cold path added by AUDIT L3 / WISHLIST #5). Invalid annotations are rejected with a `RecloneRejected` Warning Event and cleared. Valid ones emit `RecloneRequested` and report progress via the `Bootstrapping` condition. | | `bloodraven.shipstream.io/planned-failover` | Starts a planned failover to the named site. When Dragonfly is enabled, the planned-failover status includes a `dragonfly` substatus with session-preservation details. | | `bloodraven.shipstream.io/dragonfly-snapshot-upgrade` | Starts the Dragonfly snapshot-restore upgrade workflow to the target image. Requires `spec.dragonfly.snapshot` to be configured. | | `bloodraven.shipstream.io/rotate-keyring` | Rotates the named site's InnoDB master key. **Refused on the active primary** -- rotation is the one operation whose failure window would cost data rather than a re-clone, so the supported procedure is to rotate replicas, run a planned failover, then rotate the ex-primary. Also refused while an ordered update or planned failover is in flight. Cleared automatically once the target site is `Sealed` again. | | `bloodraven.shipstream.io/encryption-adopt` | Set to `confirm` to enable encryption on a failover group that already holds data, acknowledging that pre-existing tablespaces stay plaintext until they are rebuilt. `status.encryptionAtRest.sites[].coverage.unencryptedTablespaces` reports how much is still exposed. | ### Example status ```yaml status: activeSite: iad sites: - name: iad state: writable lastSeen: "2025-01-01T00:00:00Z" replicating: false - name: pdx state: read-only lastSeen: "2025-01-01T00:00:00Z" replicating: true secondsBehindSource: 0 gtidExecuted: "uuid:1-100" sourceHost: mysql-orders-iad-internal.orders.svc.cluster.local sourceConvergenceState: Converged sourceConvergenceReason: DirectSource - name: reader state: read-only lastSeen: "2025-01-01T00:00:00Z" replicating: true secondsBehindSource: 0 gtidExecuted: "uuid:1-100" sourceHost: mysql-orders-iad-internal.orders.svc.cluster.local sourceConvergenceState: Converged sourceConvergenceReason: DirectSource conditions: - type: Ready status: "True" - type: Degraded status: "False" lastFailover: "2025-01-01T00:00:00Z" lastFailoverTarget: iad dragonfly: enabled: true activeSite: iad phase: Ready message: all sites ready sites: - name: iad role: master reachable: true ready: true - name: pdx role: replica reachable: true linkStatus: up ready: true ``` --- ## `MysqlStandbyCluster` **Phase 1 of WISHLIST #7.** A passive DR relationship descriptor that lives on the DR cluster and continuously monitors a source `MysqlFailoverGroup`'s backup archive in a shared object store. The controller populates `status.discovered` and stamps readiness conditions on a configurable cadence. **No mysqld is started; no dump is loaded in Phase 1.** See [Multi-cluster DR](https://bloodraven.dev/docs/architecture/multi-cluster-dr) for the end-to-end recovery runbook and a full explanation of what Phase 1 does and does not provide. ### Top-level spec fields | Field | Type | Required | Default | Description | | ----------- | ---------------------------- | -------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `transport` | string | No | `ObjectStore` | Transport mode. Only `ObjectStore` is honored in v1alpha1; `Network` is reserved. | | `source` | StandbySource | Yes | -- | Identifies the source archive: bucket, prefix, profile name, optional decryption. | | `template` | StandbyFailoverGroupTemplate | **Yes** | -- | Embedded `MysqlFailoverGroupSpec` the controller will materialize on activation (Phase 3). Required (not optional) so the full activated topology is validated at standby-CR-create-time while the cluster is calm, rather than failing late mid-incident during a promote. Declare it now, not during an outage. | | `freshness` | StandbyFreshnessSpec | No | -- | Phase 1 bucket-discovery cadence (`discoveryInterval` only). | ### StandbySource | Field | Type | Required | Description | | ------------------- | -------------------- | -------- | ------------------------------------------------------------------------------------------------------------- | | `failoverGroupName` | string | Yes | Source `MysqlFailoverGroup` name. Informational; used in events and status. | | `namespace` | string | No | Source MFG namespace in its own cluster. Informational. | | `cluster` | string | No | Free-form source cluster identifier (e.g. `us-west-prod`). Informational. | | `storage` | BackupStorage | Yes | Object-store backend (same shape as `spec.backup.profiles[].storage`). The controller refuses to write to it. | | `profileName` | string | Yes | Backup profile name under which dumps and binlogs are stored. | | `decryption` | BackupDecryptionSpec | No | Passphrase Secret to decrypt source artifacts. Must exist in the DR namespace. | ### StandbyFreshnessSpec | Field | Type | Default | Description | | ------------------- | -------- | ------- | ------------------------------------------------------------------------------------------- | | `discoveryInterval` | duration | `5m` | How often the controller re-scans the bucket to refresh `status.discovered`. Minimum `30s`. | The Phase 2 verification cadence and staleness knobs (`verifySchedule`, `verifyTimeZone`, `maxStaleness`, `suspend`, `retentionFloorRefresh`) and the Phase 3 `spec.activate` block are **not** part of v1alpha1. They will be added back (backward-compatibly) when the code that consumes them ships. Until then they are absent from the structural schema, so they have no effect: the API server prunes them (they are silently dropped and never persisted), and clients using strict server-side field validation — the default for `kubectl apply`/`create` since Kubernetes 1.25 — reject them outright. ### Status | Field | Type | Description | | --------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `discovered` | StandbyDiscovered | Most recent successful bucket scan: dump location, name, completion time, GTID set, size, and binlog window timestamps. Encryption detection (BRV1 header) is deferred to Phase 2. | | `lastVerified` | StandbyLastVerified | Most recent terminal `MysqlBackupVerification` owned by this CR (Phase 2). | | `activation` | StandbyActivationStatus | One-shot promote audit trail: phase, source/target GTIDs, PITR stop datetime, replayed binlog count, active site, reason, message (Phase 3). | | `materializedFailoverGroup` | string | Name of the `MysqlFailoverGroup` created during activation. Empty until `Activated` (Phase 3). | | `conditions` | []Condition | Standard Kubernetes conditions. See below. | ### Conditions | Type | Phase populated | Meaning | | ---------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `BucketReadable` | Phase 1 | Bucket scan succeeded within the last discovery interval. Reason: `ListSucceeded` (True), `ListFailed` (False), `ScanIncomplete` (False — the `List` call did not complete before the scan deadline; a context deadline/cancel during `List`, distinct from a genuine `ListFailed`. The previous `status.discovered` is preserved), `AuthFailed` (False — store construction failed, e.g. bad credentials), or `ConfigError` (False — missing/invalid spec fields). | | `SourceConfigKnown` | Phase 1 | At least one full dump and one binlog manifest were found. Reason: `DumpFound` (True), `NoDumpFound` (False), `NoBinlogManifests` (False — **dump found, no PITR window yet**: a dump exists but no binlog manifests are under `/binlogs/`, so recovery is limited to the dump. This is the expected state for a dump-only source or a brand-new source, **not** a misconfiguration), `MetadataUnreadable` (False — dump `@.json` present but genuinely malformed/cannot be parsed), `ScanIncomplete` (False — the scan was cut short by a context deadline/cancel before reading every dump `@.json` or all manifests; the partial selection is **not** published, the previous `status.discovered` is preserved as last-known-good, and no `BucketScanned` event is emitted), or `ConfigError` (False — propagated from storage backend error). The default `kubectl get` printcolumn surfaces this as the neutral `SourceKnown`. | | `Restorable` | Phase 2 | Most recent owned `MysqlBackupVerification` is Succeeded and within a future staleness threshold (Phase 2). | | `ActivationInProgress` | Phase 3 | True while activation phase is non-terminal. | | `Active` | Phase 3 | True when `status.activation.phase == Activated`. | # CRD versioning and migration plan ![crd versioning infographic](https://bloodraven.dev/img/infographics/crd-versioning.webp) Bloodraven's API is currently `shipstream.io/v1alpha1`. This page sets the compatibility contract expected before promoting the API to `v1beta1` and then `v1`. ## Current state: v1alpha1 `v1alpha1` means the API is usable for pilots and early production evaluation, but the schema is still allowed to change. Consumers should: - Pin the operator and chart version. - Commit generated manifests rather than relying on floating chart refs. - Review release notes before applying CRD updates. - Avoid depending on undocumented fields or status internals. Bloodraven will preserve data-plane safety across upgrades, but it does not yet promise wire-compatible CRD evolution for every alpha field. ## Promotion criteria for v1beta1 Move to `v1beta1` only after these are true: 1. A conversion webhook exists and is covered by envtest. 2. All fields that may be renamed, reshaped, or removed from `v1alpha1` have a documented migration path. 3. Defaults and validation rules are stable enough to support server-side dry-run in production pipelines. 4. The chart installs webhook Service, Deployment flags, CA injection, and RBAC without manual post-install steps. 5. Generated CRDs in `config/crd/bases/` and `charts/bloodraven/crds/` are checked for drift in CI. 6. Backward compatibility is tested against at least one real older CR fixture per managed resource. ## Promotion criteria for v1 Move to `v1` only after `v1beta1` has survived real upgrades and the following compatibility contract is acceptable: - No breaking spec-field removals within `v1`. - Additive spec fields are allowed. - Status fields may grow additively; existing documented status fields keep their meaning. - Enum values are not removed without a deprecation window. - Defaults do not change in a way that causes existing manifests to reconcile into different data-plane behavior. - Destructive operations remain gated by explicit confirmation fields or annotations. ## Conversion strategy Bloodraven should use hub-and-spoke conversion with the newest stable API as the hub. Initial shape: ```text v1alpha1 <-> internal hub <-> v1beta1 ``` After `v1` exists: ```text v1alpha1 <-> internal hub <-> v1beta1 <-> v1 ``` The webhook must be lossless for fields that still exist in the target version. If a field has no equivalent, conversion should preserve it via annotations only when there is a concrete rollback requirement; otherwise the release notes must call out the one-way migration. ## Deprecation policy Use this policy once `v1beta1` exists: | Change type | Policy | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Add optional spec field | Allowed in any minor release. | | Add status field | Allowed in any minor release. | | Rename spec field | Add replacement first; support both through at least one minor release; remove only before `v1` or in a new API version. | | Remove enum value | Deprecate in docs and events first; reject new usage only after one minor release; keep conversion for old objects. | | Change default | Treat as breaking unless old behavior is unsafe. Require release-note callout and migration guidance. | | Change log/event contract | Update [Log schema](https://bloodraven.dev/docs/observability/log-schema) and release notes in the same PR. | ## Storage-version migration When introducing a new stored version: 1. Serve both old and new versions in the CRD. 2. Deploy the conversion webhook before changing `storage: true`. 3. Mark the new version as storage. 4. Run Kubernetes' storage-version migrator or force a safe read/write of all objects. 5. Verify `status.storedVersions` no longer lists the retired storage version before removing it from the CRD. Operational checks: ```bash kubectl get crd mysqlfailovergroups.shipstream.io \ -o jsonpath='{.status.storedVersions}{"\n"}' kubectl get mysqlfailovergroups -A -o yaml >/tmp/bloodraven-cr-backup.yaml ``` Always take an object backup before storage-version migration. This is a Kubernetes API safety net, not a MySQL data backup. ## Release-note requirements Every release that changes the CRD schema must include: - A list of added, changed, deprecated, and removed fields. - Whether `make manifests` changes `config/crd/bases/` and chart CRDs. - Whether a conversion webhook upgrade order is required. - A `kubectl apply --server-side --dry-run=server` command users can run against representative manifests. - Any data-plane behavior change caused by new defaults. ## Compatibility tests Before declaring `v1beta1`, add tests that: - Decode old `v1alpha1` YAML fixtures and convert them to the hub. - Convert hub objects back to `v1alpha1` and `v1beta1` without losing preserved fields. - Install the CRD and webhook in envtest. - Exercise status subresource writes across served versions. - Validate that destructive fields still require confirmation after conversion. ## What not to do - Do not silently change field meaning while keeping the same name. - Do not make the chart install a CRD version that the running operator cannot reconcile. - Do not remove a served version until every supported upgrade path has a conversion route. - Do not rely on Helm alone for storage-version migration; Helm updates CRD YAML but does not rewrite existing objects. # Placement Contract ![placement contract infographic](https://bloodraven.dev/img/infographics/placement-contract.webp) Bloodraven uses node labels and taints to control where MySQL pods run and to evict application workloads during failover. This page describes the labeling contract cluster administrators must satisfy. ## Why node labeling matters When the operator performs a failover, it taints nodes at the old active site with `NoExecute` to force Kubernetes to evict application pods that do not tolerate the taint. Taints are scoped per failover group, so a failover in one group does not disrupt pods belonging to another group. ## Required node labels Each site declares the exact nodes it controls with `spec.sites[].taintNodeSelector`: ```yaml spec: sites: - name: iad zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad ``` The selector is required. Bloodraven does not infer taint targets from the failover group name or site name. Apply matching labels to every node that should receive the site's taint: ```bash kubectl label node node-iad-1 \ shipstream.io/failover-group.orders=true \ shipstream.io/site.orders=iad kubectl label node node-pdx-1 \ shipstream.io/failover-group.orders=true \ shipstream.io/site.orders=pdx ``` You can label multiple nodes per site. The operator taints or untaints every node matching the selector for that site. ## Taint behavior When a site loses its primary role, the operator applies a per-group taint to all nodes selected by that site's `taintNodeSelector`: ```text shipstream.io/db-readonly-=true:NoExecute ``` For example, for a failover group named `orders`: ```text shipstream.io/db-readonly-orders=true:NoExecute ``` When a site becomes the active primary, the operator removes this taint from its selected nodes. ### Effect on pods The `NoExecute` effect means: - Pods that do not tolerate the taint are immediately evicted - Pods that do tolerate the taint continue running This gives you two categories of workloads: | Workload type | Toleration | Behavior on failover | | -------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------- | | Write-dependent apps | No toleration for `shipstream.io/db-readonly-:NoExecute` | Evicted from the old site, rescheduled to the new active site | | Read-only / stateless apps | Tolerates `shipstream.io/db-readonly-:NoExecute` | Continues running at both sites | ## Shared-node support Because taints and selectors are scoped per failover group, multiple groups can share the same physical nodes. Put one label pair per failover group on each shared node: ```bash kubectl label node node-iad-1 \ shipstream.io/failover-group.orders=true \ shipstream.io/site.orders=iad \ shipstream.io/failover-group.inventory=true \ shipstream.io/site.inventory=iad ``` Then configure each group with its own selector: ```yaml # orders MysqlFailoverGroup spec: sites: - name: iad taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad # inventory MysqlFailoverGroup spec: sites: - name: iad taintNodeSelector: shipstream.io/failover-group.inventory: "true" shipstream.io/site.inventory: iad ``` A failover in `orders` applies `shipstream.io/db-readonly-orders=true:NoExecute` only. Pods for `inventory` are unaffected if they tolerate other groups' taints. ### Cross-group tolerations On shared nodes, application pods must tolerate taint keys from other failover groups but not their own. This ensures they are only evicted by their own group's failover: ```yaml # Pod for the "orders" group on nodes shared with "inventory" spec: tolerations: - key: shipstream.io/db-readonly-inventory operator: Exists effect: NoExecute # Do not tolerate shipstream.io/db-readonly-orders. ``` A mutating admission webhook or Helm template can automate generating these tolerations across groups. ## Scheduling MySQL pods The operator schedules MySQL pods using the site's `zone` field via `topology.kubernetes.io/zone`. The `taintNodeSelector` controls application-node tainting; it is not used as the MySQL pod scheduler selector. # App Integration ![app integration infographic](https://bloodraven.dev/img/infographics/app-integration.webp) This page explains how applications connect to a MySQL failover group, optional Dragonfly cache/session endpoint, and handle failover events. ## Application checklist - Use `mysql--primary` or the external active-site DNS name for writes. - If `spec.dragonfly.enabled=true`, use `-dragonfly` for Redis-compatible cache/session traffic. - Keep DNS and connection-pool lifetimes short enough for your failover target. - Retry connection establishment and failed transactions that are safe to retry. - Configure Redis/Dragonfly clients to reconnect after socket drops; planned failover may intentionally close old-master client connections. - Do not pin writes to a site-local Service unless you are implementing warm-standby logic deliberately. - Alert on application write failures, connection pool exhaustion, and repeated read-only errors. ## DNS and pool guidance by runtime | Runtime | Guidance | | ------------------ | -------------------------------------------------------------------------------------------------- | | Go `database/sql` | Set connection max lifetime and idle lifetime; reopen failed connections after failover. | | Java/JDBC | Check JVM DNS cache TTL and pool validation query settings. | | Node.js | Recreate pools after connection loss; avoid process-wide DNS caches with long TTLs. | | Rails/ActiveRecord | Keep `reaping_frequency` and reconnect behavior enabled; restart workers if pools pin old sockets. | ## Connection examples Go DSN shape: ```text orders_app:password@tcp(mysql-orders-primary.orders.svc.cluster.local:3306)/orders?parseTime=true&timeout=5s&readTimeout=10s&writeTimeout=10s ``` JDBC: ```text jdbc:mysql://mysql-orders-primary.orders.svc.cluster.local:3306/orders?connectTimeout=5000&socketTimeout=10000 ``` Node.js `mysql2`: ```js import mysql from 'mysql2/promise'; const pool = mysql.createPool({ host: 'mysql-orders-primary.orders.svc.cluster.local', user: process.env.MYSQL_USER, password: process.env.MYSQL_PASSWORD, database: 'orders', waitForConnections: true, connectionLimit: 10, connectTimeout: 5000, }); ``` Rails: ```yaml production: adapter: mysql2 host: mysql-orders-primary.orders.svc.cluster.local username: <%= ENV.fetch("MYSQL_USER") %> password: <%= ENV.fetch("MYSQL_PASSWORD") %> database: orders reconnect: true ``` ## Connection endpoints For a `MysqlFailoverGroup` named `orders` in namespace `default`, the operator creates these Services: | Service | DNS name | Purpose | | ------------------------------ | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `mysql-orders-primary` | `mysql-orders-primary.default.svc.cluster.local:3306` | Writes. Always points to the active primary. | | `mysql-orders-replicas` | `mysql-orders-replicas.default.svc.cluster.local:3306` | Reads. Points to healthy read replicas. | | `mysql-orders-iad` | `mysql-orders-iad.default.svc.cluster.local:3306` | Client-facing access to the `iad` site instance. | | `mysql-orders-pdx` | `mysql-orders-pdx.default.svc.cluster.local:3306` | Client-facing access to the `pdx` site instance. | | `mysql-orders-reader` | `mysql-orders-reader.default.svc.cluster.local:3306` | Health-gated site-specific read pool for a `read-only` site. | | `mysql-orders--internal` | `mysql-orders--internal.default.svc.cluster.local` | Operator-only administrative route for MySQL `3306` and the sidecar. Do not use from applications. | Client-facing site Services expose only the named MySQL port. The sidecar is not externally exposed and never receives a NodePort. Internal Services are always `ClusterIP`, set `publishNotReadyAddresses: true`, and do not select on the reader's `healthy` label. This preserves the operator's route for probes, replication repair, clone, backup, and sidecar peer traffic even while the reader pod is not ready or its application endpoint has been shed. When `spec.tls` is enabled, operator Go clients dial the internal Service but continue verifying the existing client-facing site hostname as TLS `ServerName`; existing certificates do not need to add the `-internal` name. Strict hostname verification configured inside MySQL-native replication or clone is separate and must trust the internal Service hostname. When `spec.dragonfly.enabled=true`, Bloodraven also owns the cache/session endpoints for the same failover group: | Service | DNS name | Purpose | | ---------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------- | | `orders-dragonfly` | `orders-dragonfly.default.svc.cluster.local:6379` | Application Redis-compatible endpoint. Always selects the active Dragonfly master. | | `orders-dragonfly-iad` | `orders-dragonfly-iad.default.svc.cluster.local:6379` | Direct `iad` Dragonfly pod access for operator control and debugging. | | `orders-dragonfly-pdx` | `orders-dragonfly-pdx.default.svc.cluster.local:6379` | Direct `pdx` Dragonfly pod access for operator control and debugging. | Do not deploy separate tenant-chart Dragonfly CRs for a failover group where Bloodraven has `spec.dragonfly.enabled=true`. Bloodraven is the owner of the StatefulSets, Services, replication role labels, active Service selector, and planned/emergency promotion status. Tenant charts should either omit their Dragonfly templates or gate them behind a disabled-by-default value, then point application settings at the Bloodraven-created `orders-dragonfly` Service. Recommended app-facing environment variables: ```yaml - name: REDIS_HOST value: orders-dragonfly.default.svc.cluster.local - name: REDIS_PORT value: "6379" - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: tenant-dragonfly key: password ``` Redis/Dragonfly clients must reconnect after connection drops. During planned failover Bloodraven may remove the old master from the active Service and issue `CLIENT KILL TYPE NORMAL` so clients reconnect to the newly promoted site instead of holding stale sockets. For planned Dragonfly image upgrades that can tolerate a short cache/session outage, configure native snapshot restore before using the snapshot-upgrade workflow: ```yaml spec: dragonfly: snapshot: dir: s3://tenant-dragonfly/orders/prod serviceAccountName: dragonfly-backup credentialsSecretName: dragonfly-s3 s3Endpoint: rustfs.default.svc.cluster.local:9000 s3UseHTTPS: false ``` Bloodraven passes `snapshot.dir` to Dragonfly as `--dir`, renders S3-compatible flags such as `--s3_endpoint` and `--s3_use_https`, projects `snapshot.credentialsSecretName` as AWS credential environment variables when set, and assigns `snapshot.serviceAccountName` to the Dragonfly pods so cloud IAM systems such as EKS IRSA can grant bucket access. Treat this as session-continuity insurance for planned maintenance, not as the durable backup for application state. To run the planned snapshot-restore upgrade, annotate the failover group with the target image: ```bash kubectl -n default annotate --overwrite mysqlfailovergroup orders \ bloodraven.shipstream.io/dragonfly-snapshot-upgrade=docker.dragonflydb.io/dragonflydb/dragonfly: ``` Track progress in `status.dragonfly.upgrade`. During `SavingSnapshot`, `UpdatingActive`, and `WaitingForActiveRestore`, Bloodraven intentionally removes the active Dragonfly endpoint so clients see a planned cache outage instead of writing to a half-restored pod. ### Default connection pattern For workloads that should move with the active database site: ```text # Writes mysql-orders-primary..svc.cluster.local:3306 # Reads mysql-orders-replicas..svc.cluster.local:3306 ``` The `-primary` Service selector is updated by the operator whenever a failover occurs. Use this for workers, cron jobs, runners, and other write-dependent workloads that Bloodraven should evict and reschedule to the active site. For warm-standby web pods that intentionally keep one deployment per site, use the [site-local pattern](https://bloodraven.dev/#strategy-3-site-local-warm-standby) instead. ## Failover strategies Bloodraven supports two strategies for handling application failover. Choose the one that fits your deployment model. ### Strategy 1: Taint-based failover Best for applications that run on **dedicated or shared nodes** and should be fully migrated when the database fails over. **How it works:** 1. The operator taints nodes at the old active site with `shipstream.io/db-readonly-=true:NoExecute` 2. Kubernetes evicts pods that do not tolerate this taint 3. Those pods are rescheduled to the new active site's nodes (which have the taint removed) **Application requirements:** - Deploy on nodes selected by `spec.sites[].taintNodeSelector`, such as `shipstream.io/failover-group.orders=true` and `shipstream.io/site.orders=iad` - Do **not** add a toleration for `shipstream.io/db-readonly-:NoExecute` (your own group's taint) - On shared nodes, **add** tolerations for other groups' taints (see [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract#cross-group-tolerations)) - Use `mysql-orders-primary` for writes (the Service follows the active site) **Example pod spec:** ```yaml apiVersion: v1 kind: Pod metadata: name: order-processor spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: shipstream.io/failover-group operator: In values: ["orders"] containers: - name: app image: myapp:latest env: - name: MYSQL_HOST value: mysql-orders-primary.default.svc.cluster.local - name: MYSQL_PORT value: "3306" # No toleration for shipstream.io/db-readonly-orders -- pod will be evicted on failover # On shared nodes, add tolerations for OTHER groups' taints: # tolerations: # - key: shipstream.io/db-readonly-inventory # operator: Exists # effect: NoExecute ``` ### Strategy 2: Service-based failover Best for applications that are **deployed independently per site** and only need to follow the database. **How it works:** 1. Each site's application deployment connects to `mysql-orders-primary` for writes 2. On failover, the `-primary` Service selector shifts to the new active site 3. Applications at both sites see the endpoint change through normal Kubernetes Service discovery **Application requirements:** - Connect to `mysql-orders-primary` for writes - Optionally connect to `mysql-orders-replicas` for reads - Handle brief connection errors during failover (seconds, not minutes) ### Strategy 3: Site-local warm standby Best for web/API pods that run in every site and should serve a maintenance page when their local MySQL is not writable. This avoids cross-site app-to-database traffic and lets each site's frontend reflect the state of the local database instance. **How it works:** 1. Each site deployment connects to its own site Service: `mysql-orders-iad` in `iad`, `mysql-orders-pdx` in `pdx`. 2. The app checks `@@global.read_only` or handles MySQL error 1290. 3. If local MySQL is read-only, the app serves maintenance / read-only mode instead of sending writes across the WAN. 4. User-facing DNS or traffic steering moves users to the active site; Bloodraven still updates the failover group's external DNS record for clients that use `spec.dns.hostname` directly. **Application requirements:** - Run one deployment per site, pinned by node affinity or your GitOps generator. - Set `MYSQL_HOST` to the site-local Service, not `mysql-orders-primary`. - Do not tolerate your own group's `db-readonly` taint for workers that must move; web pods that intentionally stay warm in both sites may tolerate it and rely on the local read-only check. - Keep database writes behind retry/maintenance handling. A local read-only response is expected during normal failover. Example per-site deployment fragment: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: orders-web-iad spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: shipstream.io/site operator: In values: [iad] tolerations: - key: shipstream.io/db-readonly-orders operator: Exists effect: NoExecute containers: - name: web image: example/orders-web:latest env: - name: MYSQL_HOST value: mysql-orders-iad.default.svc.cluster.local - name: MYSQL_PORT value: "3306" ``` For an Argo CD ApplicationSet or Helm chart, make `site` a parameter and derive both the node selector and `MYSQL_HOST` from it: ```yaml env: - name: MYSQL_HOST value: mysql-orders-{{ .Values.site }}.default.svc.cluster.local ``` Use the moving `mysql-orders-primary` Service for background workers in the same application if those workers should only run where writes are accepted. ## Handling failover in application code During a failover, there is a brief window (typically under 10 seconds) where: 1. The old primary becomes read-only 2. The new primary is being promoted 3. DNS and Service endpoints are being updated Applications should: - **Retry** failed write operations with backoff - **Expect** `read-only` errors (`Error 1290: The MySQL server is running with the --read-only option`) and retry after a short delay - **Not** cache DNS indefinitely -- respect TTLs so that DNS changes propagated by external-dns take effect ## External access via DNS For applications outside the Kubernetes cluster, the operator manages a `DNSEndpoint` CR whose A record points `spec.dns.hostname` to the active site's load balancer IP (`spec.sites[].lbIP`). external-dns watches this CR and syncs it to your configured DNS provider. When a failover occurs, the operator updates the `DNSEndpoint` to point to the new active site. Changing `spec.dns.hostname` or `spec.dns.ttl` on a live group rewrites that same object on the next poll — the previous name is replaced, not left alongside the new one. Applications should CNAME their own DNS names to the failover group's hostname (i.e. `spec.dns.hostname`). For example, if the failover group has `dns.hostname: orders.az.example.com`, an application might create: ```text orders.myapp.example.com CNAME orders.az.example.com ``` External applications should: - CNAME to the group's `spec.dns.hostname` (not hardcoded IPs) - Respect DNS TTLs - Implement connection retry logic ## Read replicas The `mysql-orders-replicas` Service selects pods with: - `shipstream.io/role: replica` - `shipstream.io/healthy: yes` A replica is considered healthy when: - It is reachable - It is replicating (`replicating: true` in status) - Its replication lag is within `spec.replication.maxLagSeconds` If no healthy replicas exist, the Service will have no endpoints. Applications reading from the replicas Service should handle this by falling back to the primary or surfacing an error. ## Site-specific reader pools A site with `role: read-only` is a non-promotable follower intended for a local application read pool. Connect to its client-facing site Service, for example `mysql-orders-reader.default.svc.cluster.local:3306`. Never send writes to it: readers are held in `super_read_only`, are excluded from promotion and active DNS, and do not participate in application node taints. The reader Service publishes an endpoint only when the latest debounced topology snapshot confirms all of the following: - MySQL is read-only. - Both replication I/O and SQL threads are healthy. - `sourceConvergenceState` is `Converged` and the canonical `Source_Host` directly names the uniquely confirmed active primary. - Replication lag is known and is less than or equal to `spec.replication.readOnlyMaxLagSeconds`. When omitted, this inherits `maxLagSeconds`; explicit zero means only zero lag is accepted. If any check fails, Kubernetes removes the reader from the client Service at the normal topology poll cadence. Existing connections are not an operator-managed proxy and are not forcibly killed; applications must handle connection errors and retry. Direct pod access and the internal Service remain available to the operator for diagnosis and recovery. Reader failures remain visible in site status and metrics but do not set the failover group's shared `Ready=False` or `Degraded=True` conditions. # Credentials And TLS ![credentials and tls infographic](https://bloodraven.dev/img/infographics/credentials-and-tls.webp) Bloodraven supports two credential models. Use per-role credentials for new deployments. ::warning **Legacy DSN model** `spec.secretName` points at a single Secret containing a `dsn` key. It is convenient for tests but gives every component the same MySQL privileges. Do not use it for new production failover groups. Migrate by creating the role Secrets below, setting `spec.credentials`, and removing `spec.secretName` in the same apply. :: ## Credential roles | Field | Required keys | Purpose | | ---------------- | -------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | | `operatorSecret` | `username`, `password`, `MYSQL_ROOT_PASSWORD`; optional `MYSQL_REPLICATION_USER`, `MYSQL_REPLICATION_PASSWORD` | Operator, sidecar, bootstrap, replication | | `appSecret` | `username`, `password` | Application read-write user | | `readOnlySecret` | `username`, `password` | Application read-only user | | `monitorSecret` | `username`, `password` | Exporter and monitoring user | | `backupSecret` | `username`, `password` | Backup and restore Jobs | ## Create Secrets ```bash kubectl create namespace orders kubectl create secret generic orders-mysql-operator -n orders \ --from-literal=username=bloodraven \ --from-literal=password='replace-with-random-operator-password' \ --from-literal=MYSQL_ROOT_PASSWORD='replace-with-random-root-password' \ --from-literal=MYSQL_REPLICATION_USER=replicator \ --from-literal=MYSQL_REPLICATION_PASSWORD='replace-with-random-replication-password' kubectl create secret generic orders-mysql-app -n orders \ --from-literal=username=orders_app \ --from-literal=password='replace-with-random-app-password' kubectl create secret generic orders-mysql-readonly -n orders \ --from-literal=username=orders_readonly \ --from-literal=password='replace-with-random-readonly-password' kubectl create secret generic orders-mysql-monitor -n orders \ --from-literal=username=orders_monitor \ --from-literal=password='replace-with-random-monitor-password' kubectl create secret generic orders-mysql-backup -n orders \ --from-literal=username=orders_backup \ --from-literal=password='replace-with-random-backup-password' ``` ## cert-manager TLS Secret `spec.tls.secretName` must reference a Secret with `ca.crt`, `tls.crt`, and `tls.key`. With cert-manager, create a Certificate in the same namespace as the failover group: ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: orders-mysql-tls namespace: orders spec: secretName: orders-mysql-tls duration: 2160h renewBefore: 360h issuerRef: name: mysql-ca kind: ClusterIssuer commonName: orders.az.example.com dnsNames: - orders.az.example.com - mysql-orders-primary.orders.svc.cluster.local - mysql-orders-replicas.orders.svc.cluster.local - mysql-orders-iad.orders.svc.cluster.local - mysql-orders-pdx.orders.svc.cluster.local ``` Adopting `spec.tls`, changing `spec.tls.secretName`, or rotating the TLS Secret bytes restarts the topology manager so the operator's own MySQL checkers pick up the new material. MySQL pods roll independently via the spec-hash drift path. The operator uses verified TLS for credentials-mode MySQL connections when `spec.tls.secretName` is set. Include `ca.crt` in the Secret and make sure the certificate covers the primary Service plus every per-site Service (`mysql--..svc.cluster.local`) so failover probes, promotion, bootstrap/reclone, and credential reconciliation can validate the server identity. The per-site Service SAN is required for a second reason: `spec.tls` also makes the operator set `require_secure_transport=ON`, so each sidecar's MySQL connection is TLS as well. The sidecar connects over loopback, which appears in no certificate, so it verifies the server against its own site's Service name. Without that SAN the sidecar cannot query MySQL at all — `/health` returns 503 and the liveness probe restarts the container, which also stops the self-fencing monitor and the `super_read_only` safety net. Backup and restore Jobs get the same Secret mounted at `/etc/mysql/tls` and connect with `ssl-mode=VERIFY_CA` against its `ca.crt` — both the `mysqlsh` session and the `mysqlbinlog | mysql` PITR replay. A TLS-enabled Job fails before connecting if the CA path is empty, missing, unreadable, or does not contain usable CA certificates; it never downgrades to unverified TLS. Verification Jobs are different. They load the backup into an ephemeral mysqld that listens on loopback without a certificate, so that local connection stays plaintext and does not use the group's TLS material. ::note In legacy `spec.secretName` mode you supply the MySQL DSN yourself. When you enable `spec.tls` , add a `tls=` parameter to that DSN (for example `?tls=true` ) — the operator's own site connections use it verbatim. The sidecar is the exception: it is given verified TLS automatically unless your DSN already sets `tls=` , in which case your choice wins. :: ## Manual TLS Secret ```bash kubectl create secret generic orders-mysql-tls -n orders \ --from-file=ca.crt=./ca.crt \ --from-file=tls.crt=./tls.crt \ --from-file=tls.key=./tls.key ``` ## Failover group example ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: image: mysql:9.7 sidecarImage: ghcr.io/shipstream/bloodraven-sidecar:1.1.0 credentials: operatorSecret: orders-mysql-operator appSecret: orders-mysql-app readOnlySecret: orders-mysql-readonly monitorSecret: orders-mysql-monitor backupSecret: orders-mysql-backup tls: issuerRef: name: mysql-ca kind: ClusterIssuer secretName: orders-mysql-tls dns: hostname: orders.az.example.com ttl: 60 sites: - name: iad zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.1.1 storage: storageClassName: fast-ssd size: 100Gi - name: pdx zone: us-west-2a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.0.2.1 storage: storageClassName: fast-ssd size: 100Gi ``` ## Client connection examples ```bash mysql --host=orders.az.example.com \ --user=orders_app \ --password \ --ssl-mode=VERIFY_IDENTITY \ --ssl-ca=/etc/mysql/ca.crt ``` JDBC: ```text jdbc:mysql://orders.az.example.com:3306/orders?sslMode=VERIFY_IDENTITY&serverSslCert=/etc/mysql/ca.crt ``` Go `go-sql-driver/mysql` should register a TLS config with the CA and use `tls=` in the DSN. See [App Integration](https://bloodraven.dev/docs/configuration/app-integration) for application reconnect guidance. ## Rotation expectations - Updating a referenced Secret is the supported rotation path. - Rotate one role at a time and watch operator Events. - Keep old app credentials valid until application pods have restarted or connection pools have reconnected. - Treat the TLS private key and backup encryption passphrases as recovery material. ## Related pages - [Security Model](https://bloodraven.dev/docs/configuration/security-model) - [Production Hardening](https://bloodraven.dev/docs/configuration/production-hardening) - [CRD Reference](https://bloodraven.dev/docs/configuration/crd-reference) # Tenant Databases `MysqlDatabase` is a namespaced CRD that declares one database on a `MysqlFailoverGroup`, the user that owns it, and grants for principals that already exist. It exists to answer one question: **how does a caller provision a tenant database without being handed MySQL admin?** ## Why this exists Before this CRD, the only way to create a per-tenant database on a Bloodraven-managed group was to connect to MySQL as an administrator. That meant the provisioner held the group's operator credential — `GRANT ALL PRIVILEGES ON *.* WITH GRANT OPTION`, plus `MYSQL_ROOT_PASSWORD`. Standing, long-lived, and effectively root on every tenant database in every group it provisioned into. The obvious alternative — leasing a short-lived credential from a secrets engine such as OpenBao's `database` engine — does not work here, for reasons that are structural rather than incidental: 1. **The secrets engine dials the database.** It opens the connection itself and fails closed when it cannot reach the host. A MySQL instance reachable only on a private network cannot be configured as a target at all. 2. **Bloodraven is already the credential authority.** `reconcileRole` issues `CREATE USER IF NOT EXISTS … IDENTIFIED BY` and `ALTER USER … IDENTIFIED BY` from a referenced Secret's bytes. The Secret is *desired state for the MySQL user*, not a credential to a user that already exists. An external engine rotating the same principals would be a second writer with no arbitration. 3. **The operator credential cannot be leased even in principle.** It carries `MYSQL_ROOT_PASSWORD` — the value MySQL is initialized with — and the operator falls back to `root` with it when the operator user does not yet exist. A credential that must be known before the database exists cannot be minted by something that connects to the database. So the component that holds MySQL admin is Bloodraven, because it already must. `MysqlDatabase` is the way to ask Bloodraven to create a tenant database without being given the keys to do it yourself. The caller's MySQL credential is replaced by Kubernetes RBAC on a namespaced CRD. **The security property:** a caller can provision a tenant database while holding **no MySQL credential and no Secret access**. ## Example ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlDatabase metadata: name: tenant-acme namespace: bloodraven spec: groupRef: name: main # MysqlFailoverGroup in the same namespace databaseName: acme_wms # ^[A-Za-z0-9_]{1,64}$ characterSet: utf8mb4 # default collation: utf8mb4_unicode_ci # default owner: # Secret with keys `username` and `password`, same contract as # spec.credentials.*Secret on MysqlFailoverGroup. secretName: acme-mysql-owner privileges: [ALL PRIVILEGES] # ON acme_wms.* only, never WITH GRANT OPTION # Principals that must ALREADY exist. Grant-only: never CREATE USER. grants: - username: maester privileges: [SELECT, DELETE] deletionPolicy: Retain # default ``` The owner's password arrives the way every other Bloodraven credential does: you write a Secret, Bloodraven applies it. Bloodraven never generates, returns, or stores a password — if it generated one it would need somewhere to put it, which reintroduces the custody problem this CRD exists to remove. ## Status ```yaml status: phase: Ready # Pending | Creating | Ready | Failed | Deleting observedGeneration: 3 databaseCreated: true ownerUser: acme_app # echoed from the Secret; NOT the password appliedGrants: [acme_app, maester] activeSite: dc1 lastAppliedHash: a1b2c3d4e5f6 message: database acme_wms ready on site dc1 conditions: - type: Ready status: "True" reason: DatabaseReconciled ``` `observedGeneration` and the `Ready` condition are the contract. They are how a provisioner reports provisioning state back to its own callers without opening a MySQL connection — treat them as API surface, not diagnostics. **`status` never carries credential material.** `ownerUser` is a username. `lastAppliedHash` fingerprints the Secret's *revision* (UID + resourceVersion), never a digest of its bytes — status is caller-readable, and a content digest would let a status reader offline-check password guesses. ### Phases | Phase | Meaning | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Pending` | A dependency is not ready: the group is absent or has no active site, the owner Secret has not been written, a group credential Secret is mid-rotation, the primary is fenced by an in-place restore or a planned failover, or a transient MySQL/connection error hit mid-apply (an unplanned failover, say). Not an error. | | `Creating` | Applying DDL. | | `Ready` | Database, owner and grants applied on the current active primary. | | `Failed` | Invalid identifier, a MySQL system schema name, a `grants[]` user that does not exist, a pre-existing schema or account this CR did not create, an ownership conflict with another CR, a reserved owner username, or a MySQL verdict about the CR's own statements. | | `Deleting` | Finalizer running under `deletionPolicy: Delete`. | A `MysqlDatabase` applied before its `MysqlFailoverGroup` goes `Pending`, not `Failed` — that ordering is normal, not a fault. ## Privileges `privileges` is an allowlist, not a passthrough string: ```text ALL PRIVILEGES, SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, INDEX, REFERENCES, LOCK TABLES, SHOW VIEW, TRIGGER, EVENT, EXECUTE ``` Anything outside the list is rejected by the API server (the field is an enum) and rejected again in Go before any SQL is rendered. `ALL PRIVILEGES` cannot be combined with other entries. `GRANT OPTION` is not on the list and never will be. Bloodraven never emits `WITH GRANT OPTION` from this CRD — an owner that can grant is an owner that can escape its own database. `ALL PRIVILEGES` here means `GRANT ALL PRIVILEGES ON \`acme\_wms\`.\*\`, which confers nothing outside that schema. Identifiers are validated before they reach SQL rendering, not merely escaped: `databaseName`, `characterSet` and `collation` must match `^[A-Za-z0-9_]{1,64}$`, and usernames `^[A-Za-z0-9_][A-Za-z0-9_.$-]{0,31}$`. Rejection is the contract; escaping is the belt on top of the braces. MySQL's own schemas — `mysql`, `sys`, `information_schema`, `performance_schema`, case-insensitively — are rejected outright as `databaseName`: a tenant CR must never hold privileges on the grant tables. ## `grants[]` is grant-only A `MysqlDatabase` can bring exactly one MySQL principal into existence: its own `owner`, and only because you placed that user's password in a Secret first. Every `grants[]` entry names a user that must already exist. The reconciler verifies it (`SELECT 1 FROM mysql.user WHERE user = ? AND host = '%'`) and **fails the CR** with `reason: GrantUserMissing` if it does not. It never creates the user. Without that split, "create a database" would imply "create arbitrary MySQL users", and this CRD would be a privilege-escalation primitive rather than a narrowing of one. If you hit `GrantUserMissing`, the fix is to create the principal as a group-level concern — it is shared across tenants, so it does not belong to any one `MysqlDatabase`. The CR re-checks on its own and goes `Ready` once the user exists; no CR edit is needed. ## The owner cannot be a group-level principal Bloodraven applies `ALTER USER … IDENTIFIED BY` from the owner Secret's bytes. That is correct desired-state behaviour for a user this CRD owns — it is what makes rotation a Secret write. Pointed at a Secret whose `username` is `root`, the operator user, or any other account named by `spec.credentials`, the same statement would instead reset a privileged account's password to whatever the Secret's author chose. So the reconciler refuses. If `spec.owner.secretName` resolves to a username belonging to the group — `root`, `replicator`, MySQL's built-in system accounts (`mysql.sys`, `mysql.session`, `mysql.infoschema`), or any `spec.credentials` principal — the CR fails with `reason: OwnerUserReserved` and no statement is built, let alone executed. The check fails closed: if a group credential Secret cannot be read for any reason other than not existing, the reconcile errors and retries rather than proceeding with a partial reserved set. This matters most if you deviate from the recommended split. In the intended deployment the caller has no `secrets` verbs at all (the Secret is rendered by an external controller), so it cannot name anything. If your provisioner does write its own owner Secrets, this check is what stops "provision a tenant database" from becoming "reset the operator's password". ::warning **Residual risk** The check covers group-level principals, and a separate conflict check (below) covers principals that belong to another `MysqlDatabase`. What remains: a caller who can write Secrets in the namespace can still collide with MySQL users created entirely outside Bloodraven. If your callers write their own owner Secrets and you host mutually-untrusting tenants in one namespace, give each tenant its own namespace — the CRD is namespaced and the caller Role is namespaced precisely so that this is available. :: ## One database, one CR Two `MysqlDatabase` CRs on the same group must not claim the same `databaseName` or the same owner principal: desired-state `ALTER USER` means shared owners take turns resetting each other's password, and `deletionPolicy: Delete` on one duplicate would drop the other's live data. The reconciler refuses: the older CR wins, the newer fails with `reason: DatabaseNameConflict` or `OwnerConflict` before any SQL is rendered. `spec.databaseName` and `spec.groupRef` are both **immutable**, enforced by the API server. MySQL has no schema rename and the reconciler has no way to move a schema between groups; either edit would apply fresh state and orphan what already exists, and a later `deletionPolicy: Delete` would aim cleanup at the wrong objects. Renaming or re-grouping a tenant database is a migration, not a spec edit. ## Rotation Rotating the owner password is a Secret write and nothing else: ```bash kubectl -n bloodraven patch secret acme-mysql-owner \ --type merge -p '{"stringData":{"password":"new-password"}}' ``` The reconciler watches the referenced Secret, so it applies `ALTER USER` on the next reconcile. Re-applying an unchanged CR issues **zero** MySQL statements — the reconciler compares a fingerprint of the spec, the Secret's revision, the active site and the group's identity against `status.lastAppliedHash` — which is what keeps a tenant-dense cluster from hammering the primary. Rotating the **username** in the Secret is also just a Secret write, and it revokes what it replaces. Rotation is create-before-drop: the reconciler creates the new owner, grants it, applies the database, and only then drops the previous owner account (`OwnerUserRotated` event fires after the full handover). A failure mid-handover leaves both accounts alive and retried — never a window in which the tenant has no owner. A rotation performed because a credential leaked actually revokes the leaked credential — `status.ownerUser` keeps the old name until the old account is really gone, so a failure mid-rotation retries rather than leaving a shadow account. Privileges are desired state in both directions, for the owner and for every `grants[]` entry: the declared set is granted first, then only the surplus is revoked, so narrowing `[ALL PRIVILEGES]` to `[SELECT]` actually narrows it — while a failure mid-sequence leaves the principal over-granted for one requeue interval rather than with zero privileges on its own database. The revoke is scoped to this database, and the one-database-one-CR rule (above) is what makes it safe — no other CR can be managing the same grant. Removing an entry from `grants[]` revokes it on the next apply, and deletion revokes the union of the current list and the previously applied one, so no grant row outlives the CR. ## Deletion `deletionPolicy` defaults to `Retain`, and the default is the point. | Policy | Behaviour on CR delete | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Retain` (default) | Remove the finalizer, leave MySQL untouched, emit `DatabaseRetained`. No connection is opened. | | `Delete` | Revoke every grant this CR applied (current list union `status.appliedGrants`), `DROP DATABASE`, `DROP USER` the owner(s), then remove the finalizer. Never drops a `grants[]` user — those principals are shared. | Dropping a tenant database because a CR was garbage-collected by a GitOps prune, a namespace delete or a bad label selector is an unrecoverable data-loss incident. Offboarding should be an audited human action, so it takes an explicit `deletionPolicy: Delete` to express it. The zero value resolves to `Retain` too — a CR stored before the field existed, or one round-tripped by a client that dropped it, is never read as permission to drop data. `Delete` only removes what the CR actually applied and exclusively owns: - `status.databaseCreated` is a write-ahead record, stamped once the admin connection is open and before the first statement executes. A CR that failed before any SQL ran (invalid spec, reserved owner, ownership conflict, unreachable primary) releases with `DatabaseDropSkipped` and touches nothing — it must not drop a database something else created under the same name. A CR that failed *mid*-apply is covered in the other direction: its owner user is recorded and gets dropped rather than surviving as an orphaned privileged account. - If another live CR on the group still declares the same `databaseName`, shares the owner principal, or still lists the owner in its `grants[]`, the corresponding drop is skipped with a `DatabaseDropSkipped`/`OwnerUserDropSkipped` warning, and an owner that has since become a group-level principal is never dropped (`OwnerUserReservedSkipped`). - The `grants[]` revoke uses `REVOKE ... IGNORE UNKNOWN USER`, so a CR that failed on `GrantUserMissing` still deletes cleanly instead of wedging on the revoke of a user that never existed. Under `Delete`, if the group has no active site — or its primary is fenced by an in-place restore or planned failover, or the connection fails — the drop is **deferred**, not skipped: the CR emits `DatabaseDropDeferred` and waits. If the group is gone entirely there is nothing to connect to, so the finalizer is released with a `DatabaseCleanupSkipped` warning rather than wedging the CR forever. While the finalizer runs, `status.phase` is `Deleting`. ## Adoption is refused Bloodraven only manages what it created. If the schema named by `databaseName` already exists on the group and this CR's `status.databaseCreated` does not say it created it, the CR fails with `reason: DatabasePreExists` and runs no SQL. Likewise, if the owner username already exists in MySQL and is not recorded as this CR's owner, the CR fails with `reason: PreExistingOwnerUser` before any `ALTER USER` resets a foreign account's password. Neither refusal authorizes a later `deletionPolicy: Delete` to touch the foreign schema or account. To hand an existing schema over to a `MysqlDatabase`, drop and recreate the CR only if you own the schema — otherwise pick a new `databaseName`. ## Failover Reconciliation runs against the primary only. Grants replicate, so after a failover the rows are already on the new primary — but a CR must not report `Ready` against a primary the operator has not spoken to since the flip, because that is reporting something it does not know. The active site is part of the hash, so a failover invalidates the "nothing changed" short-circuit and forces a re-apply. The controller watches `MysqlFailoverGroup` and re-enqueues every `MysqlDatabase` in the namespace whose `groupRef` matches when `status.activeSite` changes, or when the group enters or leaves a fenced state. `status.activeSite` on the CR follows the group. During an in-place restore or a planned failover the primary is fenced, and reconciliation backs off to `Pending` rather than erroring — a maintenance window working as designed should not turn every tenant CR red. The fence uses the same classifiers the topology manager freezes on, so "fenced" here always means what it means to the operator. The same logic applies to *unplanned* failovers, where there is no fence to observe: the group watch re-enqueues tenants the moment `status.activeSite` moves, which can be before the promoted site has actually left `super_read_only`. Read-only and connection errors during an apply are classified as transient — the CR stays `Pending` with `reason: PrimaryUnavailable` and retries — rather than latching `Failed` on every ordinary failover. `Failed` is reserved for MySQL's verdicts about the CR itself. ## RBAC Two separate pieces, and the distinction is the security story. **The operator** gets `get;list;watch;update;patch` on `mysqldatabases` and `get;update;patch` on `mysqldatabases/status`. Deliberately no `create` and no `delete`: it reconciles tenant databases that a caller declared, it never invents them. `update` is what lets it add and remove the finalizer. **The caller** binds a namespaced Role, shipped as an example at `config/rbac/mysqldatabase_caller_role.yaml` and not installed by default: ```yaml rules: - apiGroups: [shipstream.io] resources: [mysqldatabases] verbs: [create, get, list, watch, update, patch, delete] - apiGroups: [shipstream.io] resources: [mysqldatabases/status] verbs: [get] ``` What is absent is the point: - no `secrets` rule, so the caller cannot read the owner password it provisions against, nor the group's operator credential; - no `mysqlfailovergroups` rule, so it cannot read the DSN, the credential Secret names, or the topology; - no MySQL credential of any kind. `status` is a subresource, so `update` on `mysqldatabases` does not let a caller forge `Ready`. Only Bloodraven writes status, which is what makes `Ready` mean "Bloodraven applied this". If a future change makes a `secrets` rule necessary in that Role, it has reintroduced the standing root-equivalent credential this API exists to remove. Writing the owner Secret is a separate concern and deliberately a separate principal — in ShipStream's deployment, External Secrets Operator renders it from OpenBao and the provisioner never touches it. ## Quotas Nothing in Bloodraven bounds how many databases a namespace may create. Use a Kubernetes `ResourceQuota` on the CRD count: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: tenant-databases namespace: bloodraven spec: hard: count/mysqldatabases.shipstream.io: "200" ``` ## Relationship to `spec.credentials` `MysqlDatabase` does not replace `spec.credentials` on `MysqlFailoverGroup`. The five group-level roles (operator, app, readonly, monitor, backup) stay exactly as they are; this is per-tenant databases, a different granularity. Both paths connect to the same primary through the same function — `openAdminConnection` in `internal/controller/credentials.go` — and manage disjoint principals, enforced in both directions: the tenant reconciler refuses a Secret that names a group-level principal (`OwnerUserReserved`), and group credential reconciliation fails closed before any SQL when a role username is claimed by a live `MysqlDatabase`. That function is the only place in Bloodraven that assembles MySQL admin credentials, and it has exactly these two callers. A third would be a design decision, not a refactor. ## Known gaps - **Host scoping is `'%'`,** consistent with every account in `credentials.go`. Scoping tenant owners to a pod CIDR is worth doing, but as one change across both paths rather than here alone. - **Out-of-band drift is not self-healed.** A database or grant dropped directly in MySQL is not detected: the hash short-circuit means an unchanged Ready CR issues no statements, which is the deliberate trade for not hammering the primary. Force a re-apply by rotating the owner Secret or editing the spec. - **`sql_mode=NO_BACKSLASH_ESCAPES` is unsupported** on the target group: password escaping assumes MySQL's default backslash semantics, so a password containing `\` would be stored literally under that mode. # Data-at-rest encryption Bloodraven can run MySQL with InnoDB data-at-rest encryption turned on, keeping the encryption key off the MySQL data PVC and off worker-node disks. It uses the GPL `component_keyring_file` keyring component that ships in MySQL Community Edition, so it needs no Oracle Enterprise licence and no CSI-level encrypted storage. ```yaml spec: tls: # required issuerRef: {name: ca, kind: Issuer} secretName: mysql-tls encryptionAtRest: enabled: true ``` The operator installation must also enable the chart's dedicated escrow TLS listener. Everything below explains that prerequisite, what the feature protects, and how the keyring lifecycle works. ## What this is not ::warning **Not Oracle "MySQL Enterprise TDE"** Oracle reserves the encrypted local keyring (`component_keyring_encrypted_file`) and the KMS integrations (AWS, OCI, KMIP) for Enterprise Edition, and calls the centrally-managed product "MySQL Enterprise Transparent Data Encryption". Oracle states explicitly that the file-based keyring components are **not intended as regulatory-compliance solutions**. Do not present this feature as compliance-grade TDE. If you have a PCI/FIPS-style requirement that names a key-management system, this design does not satisfy it on its own. :: ## Threat model **In scope — what this protects against:** | Threat | How it is addressed | | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | A stolen or decommissioned MySQL data PVC | Tablespaces, redo, undo, binary and relay logs are all encrypted. The key is not on the volume. | | A volume snapshot copied out of the cluster | Same. | | A worker-node disk read offline | The keyring only ever exists on tmpfs (a Secret volume or a memory-backed `emptyDir`), never on a persistent disk. | | An operator or application user rotating keys ad hoc | In the sealed steady state the keyring is read-only, so MySQL itself rejects `ALTER INSTANCE ROTATE INNODB MASTER KEY`. | **Out of scope — what this does not protect against:** - **A live root-level compromise of a node, or of the MySQL process.** A running MySQL necessarily holds decrypted keys in memory. Anything that can read that memory, or that can simply ask MySQL for the data, wins. - **A compromised Kubernetes control plane.** See [etcd is now part of your key custody](https://bloodraven.dev/#etcd-is-now-part-of-your-key-custody). - **Backup, restore, and verification staging.** See [Boundaries](https://bloodraven.dev/#boundaries). ## etcd is now part of your key custody The live keyring is projected from a Kubernetes Secret. **Kubernetes stores Secrets unencrypted in etcd by default.** Without API-server encryption at rest, enabling this feature does not protect your keys — it just moves them from the MySQL data disk to the control-plane disk. Before enabling `encryptionAtRest`, you must: 1. **Turn on API-server encryption at rest for Secrets**, ideally KMS-backed (`kms` provider v2), not `aescbc` with a local key file. See the [Kubernetes encryption-at-rest guide](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/){rel=""nofollow""}. 2. **Restrict RBAC on Secrets** in the failover group's namespace. Anyone who can `get` secrets there can read the keyring and decrypt a stolen PVC. 3. **Disable or encrypt swap on worker nodes.** Kubernetes asks for `tmpfs` with `noswap`, but kernel support for that mount option is only official from Linux 6.3. On older kernels kubelet logs a warning and memory-backed volume contents can reach swap. Verify with `mount | grep tmpfs` on a node and check kubelet's startup logs. None of these are optional. Bloodraven cannot verify them for you. ## Requirements - **`spec.tls` must be set.** MySQL requires a secure connection to clone encrypted data, and Bloodraven bootstraps every replica with `CLONE INSTANCE`. The CRD rejects `encryptionAtRest.enabled: true` without TLS. - **The `spec.tls` certificate must cover the per-site Services.**`spec.tls` makes the operator set `require_secure_transport=ON`, so the sidecar's own MySQL connection is TLS too. It dials loopback — a name no certificate carries — and therefore verifies against `mysql--..svc.cluster.local`. That SAN is already required for the operator's failover probes (see [Credentials and TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls)); if it is missing, the sidecar cannot connect, `/health` fails, and the liveness probe restarts the container. - **The operator escrow TLS listener must be enabled.** Set `auxiliary.escrowTLS.enabled=true` and `auxiliary.escrowTLS.existingSecret=` on the Bloodraven chart. The Secret must contain `tls.crt` and `tls.key`; the certificate SAN must cover the auxiliary Service FQDN, and every encrypted group's `spec.tls` `ca.crt` must trust its issuer. Keyring bytes and their bearer token are never sent over the plaintext auxiliary listener. - **MySQL 9.x Community** (or any build shipping `component_keyring_file`). Verified against `mysql:9.7`. - **Image-specific paths.** Bloodraven copies the image's `mysqld` binary to a private runtime path and places the global component manifest next to that copy. It places the global component config in the image's `plugin_dir`. The private executable path also prevents a host-level, path-attached `/usr/sbin/mysqld` AppArmor profile from breaking component loading inside kind. Defaults match the official `mysql:9.x` images. If you run a different image, check the source binary and plugin paths: ```bash docker run --rm --entrypoint sh -c \ 'command -v mysqld; mysqld --verbose --help 2>/dev/null | grep "^plugin_dir"' ``` :brand set `spec.encryptionAtRest.keyring.mysqldDir` / `.pluginDir` accordingly. ## How the keyring lifecycle works Every site has its own independent keyring. That is not a simplification — MySQL's encrypted clone decrypts the donor's tablespace keys in transit and the recipient re-encrypts them under a **new master key of its own**, so sites cannot share one. A site is in one of two renderings at any moment. ### Sealed — the steady state ```text Kubernetes Secret mysql---keyring-v (immutable) │ │ kubelet projects it onto node-local tmpfs ▼ /run/mysql-keyring/keyring (read-only, mode 0444) │ │ component_keyring_file with "read_only": true ▼ mysqld ──────────────► encrypted /var/lib/mysql ``` `"read_only": true` is doing real work here, not documentation. With it set, MySQL physically cannot add a key: ```sql mysql> ALTER INSTANCE ROTATE INNODB MASTER KEY; ERROR 3185 (HY000): Can't find master key from keyring, ... ``` That is the guarantee that makes the design safe: in the steady state there is **no window** in which MySQL can create a key that is not already escrowed, because it cannot create keys at all. ### Unsealed — bootstrap, clone, and rotation Three operations genuinely need to write the keyring: | Operation | Why | | ------------------------------- | -------------------------------------------------------------------------------------------------------- | | Initial bootstrap | MySQL creates the master key, redo key, and binlog master key while initializing a fresh data directory. | | `CLONE INSTANCE` into this site | The recipient re-wraps every tablespace key under a new master key of its own. | | Master-key rotation | By definition. | For those, the operator re-renders the site with the keyring on a memory-backed `emptyDir`, seeded from the current escrow version by an init container, and arms the sidecar's escrow agent: ```text keyring-init ──seed──► /run/mysql-keyring/keyring (tmpfs, rw) │ mysqld writes ────┤ │ sidecar reads, hashes │ HTTPS POST /keyring/escrow (bearer token) ▼ operator writes a new immutable Secret version │ operator re-reads it and compares against the live digest ▼ re-render sealed, roll pod ``` The operator will not seal a site until **all** of the following hold: 1. an escrow Secret exists for the site, 2. its contents, re-hashed by the operator, match the live keyring digest the sidecar reports, 3. the Deployment is actually rendered with the Secret projection and the pod is ready, 4. MySQL reports `Read_only: Yes` in `performance_schema.keyring_component_status`. ## Key loss is bounded by construction The only window in which a keyring can be lost is while a site is deliberately unsealed. That window is made survivable rather than merely short: - **Bootstrap:** the site has no data worth keeping. Recovery is to wipe and re-bootstrap. - **Clone:** the site is being overwritten from a donor anyway. Recovery is to re-clone. - **Rotation:** **the operator refuses to rotate the active primary, and refuses to promote a site whose `UnsealReason` is `Rotation` until it is `Sealed`.** A replica that loses its keyring mid-rotation is re-cloneable from the primary; a primary that lost one would not be. See [Rotating keys](https://bloodraven.dev/#rotating-keys). So a lost keyring always costs you a re-clone of one site, never your data. The one thing that would be unrecoverable — losing the escrow Secret of a sealed site — is why the operator raises a `Failed` phase and a `KeyringEscrowMissing` event the moment it notices one is gone. ## Status and monitoring ```bash kubectl get mysqlfailovergroup orders -o jsonpath='{.status.encryptionAtRest}' | jq ``` ```json { "sealed": true, "sites": [ { "name": "iad", "phase": "Sealed", "keyringSecret": "mysql-orders-iad-keyring-v1", "keyringVersion": 1, "keyringDigest": "sha256:9f2c…", "coverage": { "keyringComponent": "component_keyring_file", "keyringReadOnly": true, "systemTablespaceEncrypted": true, "unencryptedTablespaces": 0, "redoLogEncrypted": true, "undoLogEncrypted": true, "binlogEncrypted": true } } ] } ``` `status.conditions[type=EncryptionAtRestReady]` summarizes the group. ### When a site will not seal A site that stays at `phase: Unsealed` carries the reason in `sites[].message`, including the sidecar's own last error where it has one. The three common causes: | `message` contains | Cause | | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | `waiting for the Deployment to roll onto the writable keyring rendering` | The pod is not Ready. Check the sidecar container — if it is restarting, look at its logs before anything else. | | `sidecar reports: ... connection refused` | The operator's escrow listener is not running. Set `auxiliary.escrowTLS.enabled=true` on the chart. | | `sidecar reports: ... x509: certificate signed by unknown authority` | The escrow listener's certificate is not trusted by this group's `spec.tls` `ca.crt`. | ```bash kubectl get mysqlfailovergroup orders \ -o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}' ``` ### Metrics | Metric | Meaning | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `bloodraven_keyring_phase{namespace,group,site,phase}` | One-hot per site. **Alert on `phase="sealed"` being 0 for more than a few minutes.** | | `bloodraven_keyring_escrow_version{namespace,group,site}` | Current escrow version per site. | | `bloodraven_keyring_escrow_pushes_total{group,site,outcome}` | Sidecar → operator escrow pushes. Sustained failures mean a site cannot be sealed. | | `bloodraven_keyring_rotations_total{group,site,outcome}` | Master-key rotations. | | `bloodraven_keyring_promotions_blocked_total{namespace,group,site,outcome}` | Promotion decisions that skipped or refused a mid-rotation site. `outcome` is `skipped` or `refused`. | | `bloodraven_encryption_unencrypted_tablespaces{namespace,group,site}` | User tablespaces still reporting `ENCRYPTION='N'`. **Non-zero means part of your data is in the clear.** | | `bloodraven_encryption_coverage{namespace,group,site,aspect}` | Per-aspect coverage (`system_tablespace`, `redo_log`, `undo_log`, `binlog`, `keyring_read_only`). | Suggested alert: ```yaml - alert: BloodravenKeyringNotSealed expr: bloodraven_keyring_phase{phase="sealed"} == 0 for: 15m annotations: summary: "{{ $labels.site }} is running with a writable keyring" ``` ## Coverage With the defaults, Bloodraven enforces these as operator-owned settings — a `spec.mysqlConf` entry naming the same key is ignored, so a stray override cannot silently downgrade a site: ```ini default_table_encryption=ON table_encryption_privilege_check=ON innodb_redo_log_encrypt=ON innodb_undo_log_encrypt=ON binlog_encryption=ON # covers relay logs on replicas too ``` Plus `ALTER TABLESPACE mysql ENCRYPTION='Y'`, run once by the sidecar on the writable site. This one matters more than it looks: `default_table_encryption` does **not** cover the `mysql` system tablespace, which holds the data dictionary. Leave it unencrypted and every schema, table, and column name is readable on a stolen PVC even though the row data is not. Each can be turned off individually — and each one you turn off narrows the claim: ```yaml spec: encryptionAtRest: enabled: true coverage: binaryLog: false # binlogs and relay logs land in plaintext ``` ::note **`default_table_encryption` is forward-looking** It encrypts schemas and tables created *after* it takes effect. It does not rewrite anything already on disk. That is what `status.…coverage.unencryptedTablespaces` counts. :: ## Enabling encryption ### On a new failover group Set `spec.encryptionAtRest.enabled: true` and `spec.tls` before the group first bootstraps. The first primary-candidate comes up unsealed, creates its keys, escrows, and seals. Every other site is seeded with `CLONE INSTANCE`: the operator unseals the recipient, holds it unsealed until the clone and the post-clone mysqld restart finish, then re-escrows and reseals. A single-site group has no clone step. ::warning **Operator and sidecar images must both be current** Because `spec.tls` sets `require_secure_transport=ON`, the sidecar's own MySQL connection needs TLS, which the operator configures through environment variables on the sidecar container. An operator new enough to render them paired with a sidecar too old to read them — or the reverse — leaves the sidecar unable to reach MySQL: it fails its health check, the liveness probe restarts it, and no site ever seals. Keep `spec.sidecarImage` at the same release as the operator. The operator emits a `SidecarVersionSkew` Warning event when the two image tags differ. It is advisory only — a re-tagged or locally built image looks the same as a real mismatch, so blocking on it would turn a cosmetic difference into an outage. :: ### On an existing group The operator **refuses** by default: ```text Warning EncryptionAdoptionRefused spec.encryptionAtRest.enabled was turned on for a group that is already serving from site "iad". Existing tablespaces stay plaintext — MySQL only encrypts data written after the fact. ``` This is not a mistake in the refusal — it is the honest answer. Flipping the flag on a live group gives you a cluster that *reports* encrypted while most of the bytes on the PVC are not. **Supported path — replica-first, no plaintext left behind:** 1. Stand up the encrypted group (or add an encrypted site) and let it bootstrap and seal. 2. Load it from the existing cluster (`spec.initFromBackup`, or replication from the old primary). 3. Verify `status.encryptionAtRest.sealed == true` and `unencryptedTablespaces == 0` on every site. 4. Cut traffic over with a [planned failover](https://bloodraven.dev/docs/operations/planned-failover). 5. Decommission the old cluster and **wipe its PVCs** — they still hold plaintext. **Escape hatch — accept partial coverage:** If you understand that pre-existing tables stay in the clear until they are rebuilt: ```bash kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/encryption-adopt=confirm ``` Adoption rolls one site at a time. Encrypted configuration is written to a new immutable, content-addressed ConfigMap; an unrolled Deployment continues referencing its original unencrypted ConfigMap. The ordered updater switches the ConfigMap reference and keyring volume/init wiring together in one pod template revision. A restart of a site that has not taken its update therefore still uses a coherent unencrypted configuration rather than requesting a keyring the pod does not provide. New writes are then encrypted, redo/undo/binlog encryption engages, and `unencryptedTablespaces` tells you how much is still exposed. Rebuild the remainder at your own pace: ```sql ALTER TABLE mydb.orders ENCRYPTION='Y'; -- rebuilds the table ``` Note that this rewrites each table in place, so plan it like any other large `ALTER`. ## Rotating keys Rotation is the one operation where a lost keyring would cost you data rather than a re-clone, because `ALTER INSTANCE ROTATE INNODB MASTER KEY` re-wraps the tablespace keys under a new master key. **The operator therefore refuses to rotate the active primary.** The supported procedure rotates every site while it is a replica: ```bash # 1. Rotate each replica, one at a time. Wait for Sealed between them. kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/rotate-keyring=pdx --overwrite kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="pdx")].phase}' # Pending → Unsealed → Escrowed → Sealed # 2. Planned failover onto a rotated site. kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/planned-failover=pdx # 3. Rotate the old primary, now a replica. kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/rotate-keyring=iad --overwrite ``` The annotation is cleared automatically once the target site is Sealed again. Rotation is also refused while an ordered update or a planned failover is in flight. The inverse coupling also holds: **a site mid-rotation is not a promotion target.** Emergency failover skips it (or refuses entirely if it is the only remaining candidate). A planned failover whose target is still `UnsealReason=Rotation` is rejected with `status.plannedFailover.reason: KeyringRotation` — wait for `Sealed`, then re-annotate. There is no cancel-rotation command; if rotation is stalled, repair escrow or the sidecar and let the site reach `Sealed`. If that leaves the group with no promotion target, the operator stays read-only on purpose: promoting would make the unescrowed keyring the sole authoritative copy. Each rotation mints a new immutable escrow Secret version. `keyring.retainVersions` (default 5) controls how many superseded versions are kept; the version a site is currently sealed against is never pruned, even if retention would otherwise reach it. ## Disaster recovery **A site's pod is gone.** Nothing to do. The keyring lives in the escrow Secret; kubelet re-projects it when the pod is rescheduled. **A site's escrow Secret was deleted.** The site is `Failed` and a `KeyringEscrowMissing` event fires. If the pod is still running, its keyring is alive on tmpfs — **do not delete or restart that pod.** Recover by recreating the exact missing Secret directly from the running pod. This avoids a rollout while the only surviving key copy is in memory. ```bash set -o pipefail POD=$(kubectl get pod \ -l shipstream.io/failover-group=orders,shipstream.io/site=iad \ -o jsonpath='{.items[0].metadata.name}') SECRET=$(kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringSecret}') VERSION=$(kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringVersion}') DIGEST=$(kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringDigest}') KEYRING_DIR=$(kubectl get mysqlfailovergroup orders \ -o jsonpath='{.spec.encryptionAtRest.keyring.dataFileDir}') KEYRING_DIR=${KEYRING_DIR:-/run/mysql-keyring} FG_UID=$(kubectl get mysqlfailovergroup orders -o jsonpath='{.metadata.uid}') kubectl exec "$POD" -c mysql -- cat "$KEYRING_DIR/keyring" | \ kubectl create secret generic "$SECRET" --from-file=keyring=/dev/stdin ACTUAL_DIGEST=$(kubectl get secret "$SECRET" -o jsonpath='{.data.keyring}' | \ base64 -d | sha256sum | cut -d' ' -f1) if [[ "sha256:${ACTUAL_DIGEST}" != "$DIGEST" ]]; then kubectl delete secret "$SECRET" echo "restored keyring digest does not match status; refusing to continue" >&2 exit 1 fi kubectl label secret "$SECRET" \ app.kubernetes.io/name=mysql-keyring \ app.kubernetes.io/managed-by=bloodraven \ shipstream.io/failover-group=orders shipstream.io/site=iad \ shipstream.io/keyring-version="$VERSION" kubectl annotate secret "$SECRET" shipstream.io/keyring-digest="$DIGEST" kubectl patch secret "$SECRET" --type=merge -p="{ \"metadata\":{\"ownerReferences\":[{ \"apiVersion\":\"shipstream.io/v1alpha1\",\"kind\":\"MysqlFailoverGroup\", \"name\":\"orders\",\"uid\":\"${FG_UID}\",\"controller\":true,\"blockOwnerDeletion\":true }]},\"immutable\":true }" SITE_INDEX=$(kubectl get mysqlfailovergroup orders -o json | \ jq -er '.status.encryptionAtRest.sites | map(.name) | index("iad")') kubectl patch mysqlfailovergroup orders --subresource=status --type=json -p="[ {\"op\":\"test\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/name\",\"value\":\"iad\"}, {\"op\":\"replace\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/phase\",\"value\":\"Sealed\"} ]" ``` If the pod is already gone, that site's data is unrecoverable. Delete its unreadable PVC and use the confirmed reclone annotation; the operator starts the replacement with a fresh keyring and clones from the healthy primary. **The whole namespace was lost.** Restore from a backup ([backup encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption) is a separate, independent key), then bootstrap a fresh encrypted group and load it. Backup artifacts are logical dumps, so they do not depend on any site's keyring. **Rolling a site back onto an older keyring version.** Patch the site's `keyringSecret`/`keyringVersion`/`keyringDigest` in status to an older retained version and let the pod roll. Only do this if you know the older version still decrypts the data on that PVC — a master-key rotation happened in between if the version changed, and the newer key is the one the tablespace headers reference. ## Boundaries Live MySQL encryption does not cover Bloodraven's logical backup/restore/verification workflows: | Workflow | What still touches node storage in plaintext | | ------------------- | ------------------------------------------------------------------------------------------ | | Backup | `mysqlsh` stages the dump in a local `emptyDir` before Bloodraven encrypts and uploads it. | | Restore | The artifact is decrypted into an `emptyDir` before `mysqlsh` loads it. | | Backup verification | An ephemeral MySQL data directory is created on an `emptyDir`, not TDE-enabled. | Backup *artifacts* at rest in S3 or on a PVC are separately protected by [`spec.backup.profiles[].encryption`](https://bloodraven.dev/docs/backup-and-restore/backup-encryption) — turn that on too. The gap above is the transient staging inside the pod's own `emptyDir`, which lands on the node's ephemeral storage. If your requirement is "no plaintext database content ever touches node storage", this feature alone does not satisfy it and you should also constrain where backup Jobs schedule, or use encrypted ephemeral storage on those nodes. ## Interaction with other features | Feature | Behaviour | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Failover / promotion | A site with `UnsealReason=Rotation` is not a promotion candidate until it is `Sealed`. Other sealed candidates are still promoted. If none remain, the group stays without a writable primary and emits `KeyringPromotionRefused`. | | Reclone (`bloodraven.shipstream.io/reclone-site`) | The operator unseals the recipient first and keeps it unsealed until bootstrap finishes (clone, mysqld restart, replication setup), then re-escrows and re-seals. The clone is deferred (not failed) while the pod rolls onto the writable keyring. | | Ordered updates | Encryption ConfigMaps are immutable and content-addressed. Each site switches its ConfigMap reference and keyring wiring in the same ordered pod-template update; unrolled sites retain their prior restartable revision. Superseded ConfigMaps are removed only after the rollout reports no old pod remaining. | | PITR / binlog archiver | Archived binlogs are encrypted by MySQL on disk *and* re-encrypted by the archiver if `spec.backup` encryption is on. | | Dragonfly | Unaffected — Dragonfly is a separate subsystem with no keyring. | ## Reference | Field | Default | Description | | ------------------------------ | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `false` | Turn on data-at-rest encryption. Requires `spec.tls`. | | `coverage.tables` | `true` | `default_table_encryption` | | `coverage.privilegeCheck` | `true` | `table_encryption_privilege_check` | | `coverage.redoLog` | `true` | `innodb_redo_log_encrypt` | | `coverage.undoLog` | `true` | `innodb_undo_log_encrypt` | | `coverage.binaryLog` | `true` | `binlog_encryption` (binary + relay logs) | | `coverage.systemTablespace` | `true` | `ALTER TABLESPACE mysql ENCRYPTION='Y'` | | `keyring.dataFileDir` | `/run/mysql-keyring` | Keyring mount point. Must be outside the actual MySQL datadir. Admission rejects `/var/lib/mysql` and descendants; custom-image datadirs require manual verification. | | `keyring.mysqldDir` | `/usr/sbin` | Directory holding the image's source `mysqld`; Bloodraven copies it to a private runtime path and places the global manifest beside the copy. | | `keyring.pluginDir` | `/usr/lib64/mysql/plugin` | MySQL `plugin_dir`; Bloodraven materializes the global component config here before startup. | | `keyring.retainVersions` | `5` | Superseded escrow versions kept per site (2–50). | | `keyring.escrowTimeoutSeconds` | `600` | How long before a site that has not escrowed is reported `Failed`. It stays unsealed and keeps retrying either way. | ### Resources the operator creates | Resource | Purpose | | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mysql---keyring-v` (Secret, immutable) | Escrowed keyring. Owned by the failover group. | | `mysql---keyring-token` (Secret) | Per-site escrow bearer token. Mounted into the sidecar only while unsealed. | | `mysql---config-` (immutable ConfigMap) | Content-addressed encrypted MySQL and keyring-component configuration. The Deployment references the exact revision matching its keyring wiring; superseded revisions remain until rollout completion. Unencrypted groups retain the canonical `mysql---config` name. | ### Annotations | Annotation | Value | Effect | | ------------------------------------------- | --------- | ------------------------------------------------------------------------------------ | | `bloodraven.shipstream.io/rotate-keyring` | site name | Rotate that site's master key. Refused on the active primary. Cleared on completion. | | `bloodraven.shipstream.io/encryption-adopt` | `confirm` | Accept partial coverage when enabling encryption on a live group. | ## Further reading - [InnoDB data-at-rest encryption](https://dev.mysql.com/doc/refman/9.7/en/innodb-data-encryption.html){rel=""nofollow""} - [Keyring component installation](https://dev.mysql.com/doc/refman/9.7/en/keyring-component-installation.html){rel=""nofollow""} - [`component_keyring_file`](https://dev.mysql.com/doc/refman/9.7/en/keyring-file-component.html){rel=""nofollow""} - [Encrypting binary and relay logs](https://dev.mysql.com/doc/refman/9.7/en/replication-binlog-encryption.html){rel=""nofollow""} - [Cloning encrypted data](https://dev.mysql.com/doc/refman/9.7/en/clone-plugin-encrypted-data.html){rel=""nofollow""} - [Kubernetes: encrypting data at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/){rel=""nofollow""} - [Kubernetes: swap and memory-backed volumes](https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/#memory-backed-volumes){rel=""nofollow""} # Security model ![security model infographic](https://bloodraven.dev/img/infographics/security-model.webp) This page documents Bloodraven's security posture: the trust boundaries between its components, the authorizations each component holds, and the blast radius when a specific secret or identity is compromised. It complements [Production hardening](https://bloodraven.dev/docs/configuration/production-hardening), which lists the settings to turn on; this page explains *why* they matter and what remains exposed when they are all correct. Bloodraven is an **in-cluster control plane for a tenant MySQL service**. It is not a zero-trust system on its own. It relies on the hosting Kubernetes cluster for authentication, namespace isolation, and network policy. The guarantees below all assume a competent cluster operator is running it. ## Security quick wins - Use `spec.credentials` with five distinct role Secrets. - Enable `spec.tls` and require TLS in MySQL clients. - Keep `auxiliary.service.enabled=false` unless a dashboard or integration needs it. - Add NetworkPolicy for operator, sidecar, MySQL, Prometheus, app, and backup paths. - Pin and mirror operator, sidecar, MySQL, and backup images. - If Dragonfly is enabled, protect its password Secret, active Service, and admin port with the same care as the MySQL data plane. - Link Secret rotation to [Credentials And TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls). ## Threat-model assumptions | Assumption | Consequence | | -------------------------------------------------------- | ---------------------------------------------------------------------- | | Kubernetes API authn/authz is trusted | RBAC boundaries are meaningful. | | CNI enforces NetworkPolicy correctly | Unauthenticated sidecar and auxiliary endpoints can be isolated. | | Secret storage is protected | MySQL, Dragonfly, TLS, S3, and encryption secrets remain confidential. | | Node administrators are trusted | A node admin can read mounted Secrets and MySQL data on that node. | | Application workloads are less trusted than the operator | Apps should only receive app or read-only MySQL credentials. | ## Trust model at a glance Four identities participate in the required MySQL data path. Managed Dragonfly adds an optional cache/session data path when enabled. Everything else in the cluster is considered untrusted. | Identity | What it is | Trust level | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | **Operator** | Single-replica Deployment running `bloodraven`, bound to a cluster-wide ServiceAccount. Owns the reconcile loop and DNS writes. | Fully trusted — it is the control plane. | | **Sidecar** | Per-MySQL-pod container running `bloodraven-sidecar`. Serves read-only status HTTP, manages MySQL startup, writes binlog manifests, and self-fences its MySQL. | Trusted for the single pod it runs in. | | **MySQL** | `mysqld` itself, with the per-role users Bloodraven creates (`operator`, `app`, `readonly`, `monitor`, `backup`). | Trusted for query execution; each MySQL user is trusted only for its role's grants. | | **Dragonfly** | Optional per-site Dragonfly pods created when `spec.dragonfly.enabled=true`. The operator configures replication, role/traffic labels, and promotion. | Trusted for cache/session data only. Do not store durable application state here. | | **Application** | Workloads that connect to the primary/replica Services using `appSecret` / `readOnlySecret`. | Untrusted by the operator — apps are tenants. | All intra-cluster traffic between these identities flows over the pod network. Bloodraven does **not** authenticate HTTP calls between the operator and the sidecars, and it does **not** mTLS-protect the sidecar HTTP endpoint. Isolation is the hosting cluster's job (NetworkPolicy, namespace boundaries, CNI-level encryption if you require it). ## Authorization — who can do what ### Operator ServiceAccount The operator runs with a **cluster-scoped** ClusterRole (`config/rbac/role.yaml`, mirrored in `charts/bloodraven/templates/clusterrole.yaml`). It holds the minimum set of verbs to reconcile `MysqlFailoverGroup` and `MysqlBackup` resources across any namespace. Notably: - **Full CRUD on `Secrets`, `ConfigMaps`, `Services`, `PersistentVolumeClaims`, `Jobs`, `CronJobs`, `Deployments`, `PodDisruptionBudgets`** — cluster-wide. The operator reads credential Secrets, renders MySQL config into ConfigMaps, and creates PVCs, Services, Pods, and Jobs per site. - **Full CRUD on `dnsendpoints.externaldns.k8s.io`** — the failover path writes `DNSEndpoint` CRs that `external-dns` then pushes to the DNS provider. - **`get`/`list`/`watch`/`patch` on `pods` and `nodes`**, plus `get` on `pods/log` — used for the tainter, the backup-sentinel log tail, and placement contract enforcement. - **Full CRUD on `leases.coordination.k8s.io`** — leader election. **There is no namespaced partitioning.** A single operator install reconciles groups in every namespace. If you need tenant isolation (one operator per tenant), deploy one operator per namespace scope and use a `Role` instead — the code supports this but the shipped chart does not. ### MySQL user roles When `spec.credentials` is used, the operator creates five MySQL users with the following effective grants (see [CRD Reference → CredentialsSpec](https://bloodraven.dev/docs/configuration/crd-reference#credentialsspec)): | Role | Effective authority | | ---------- | ------------------------------------------------------------------------------------------------------------------- | | `operator` | `ALL PRIVILEGES ON *.* WITH GRANT OPTION` — full admin. Used by the operator and sidecar only. | | `app` | `ALL PRIVILEGES ON *.*` without `GRANT OPTION` or `SUPER` — read/write DML + DDL for tenant workloads. | | `readonly` | `SELECT, SHOW VIEW, SHOW DATABASES, PROCESS` — read replicas. | | `monitor` | `PROCESS, REPLICATION CLIENT` + `SELECT` on `performance_schema.*` — `mysqld_exporter` and similar scrapers. | | `backup` | `SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER, RELOAD, BACKUP_ADMIN, REPLICATION CLIENT` — MySQL Shell dump jobs. | The legacy `spec.secretName` DSN mode collapses all of the above into a single user whose credentials the DSN carries. **Do not use it in production** — it defeats the entire per-role isolation story. ### Destructive annotations on the CR `patch` on `mysqlfailovergroups` is the narrow privilege that controls every operator-initiated destructive action. The annotations the operator acts on are: - `bloodraven.shipstream.io/reclone-site=:` — Triggers `CLONE INSTANCE` on the named site, overwriting its data directory from the current primary. The GTID-prefix confirmation (≥ 8 chars, must match the current `status.sites[].divergentGtid`) prevents typos from wiping the wrong site. See [Failover → Reclone flow](https://bloodraven.dev/docs/operations/failover#divergence-detected-manual-intervention-required). - `bloodraven.shipstream.io/planned-failover=` — Drains writes, honours the anti-flap cooldown, waits for the target to catch up, and promotes the named site. When Dragonfly is enabled, this also drives the Dragonfly sync and `REPLTAKEOVER` phases. - `bloodraven.shipstream.io/dragonfly-snapshot-upgrade=` — Runs the Dragonfly snapshot-restore upgrade workflow when `spec.dragonfly.snapshot` is configured. - `bloodraven.shipstream.io/last-failover` and `bloodraven.shipstream.io/last-failover-target` — Operator-owned durable anti-flap history. A writer can extend or clear the effective cooldown and change the remembered fencing target. The operator rejects malformed, unpaired, unknown-site, and implausibly future-dated values, but these keys must still be protected from manual or third-party mutation. An attacker with `patch mysqlfailovergroups` in the target namespace can therefore cause a data wipe of one site (bounded by the GTID confirmation interlock), an unscheduled failover, or a planned Dragonfly cache/session outage during snapshot-restore upgrade. RBAC on the CRD is the only gate — scope `MysqlFailoverGroup` edit rights as tightly as you would scope `kubectl delete pvc`. ## Network surface | Listener | Port | Authenticated? | Purpose | | ----------------------------- | ------------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | Operator metrics | `:8080` | No | Prometheus scrape (`/metrics`). | | Operator health probes | `:8081` | No | Kubernetes liveness/readiness. | | Operator auxiliary HTTP | `:8082` | No | `/status`, `/active-site`, `/pitr-cutoff`, `/ws/status` (WebSocket). | | Operator keyring escrow HTTPS | `:8443` by default | Bearer token + TLS | `/keyring/escrow`; enabled with `auxiliary.escrowTLS`. | | Sidecar HTTP | `:8080` | No | `/health`, `/status`, `/peer/ping`, `/archiver/status`, `/keyring/status`. Consumed by the operator and the peer sidecar. | | MySQL | `:3306` | Yes (password; TLS if `spec.tls` set) | Data plane. | | Dragonfly client | `:6379` by default | Password when `spec.dragonfly.auth` is set | Optional Redis-compatible cache/session data plane. | | Dragonfly admin | `:9999` by default | Password when `spec.dragonfly.auth` is set | Optional operator control surface for replication, promotion, and snapshot commands. | **With one exception, none of the HTTP surfaces above authenticate callers.** The unauthenticated ones expose read-only status and GTID coordinates and accept no state-changing requests. The design assumes the pod network is protected by NetworkPolicy and that the operator + sidecars run in a namespace that does not accept untrusted ingress. In particular: - Do not expose `:8082` outside the cluster. The `/active-site` and `/status` endpoints do not leak credentials but do leak topology, site names, zone labels, and GTID coordinates — useful reconnaissance for an attacker already inside the cluster. - The WebSocket push stream (`/ws/status`) is likewise unauthenticated. Anyone that can reach port `8082` can read real-time failover events. - If you require scrape auth, front the metrics endpoint with a sidecar (e.g. `kube-rbac-proxy`). The chart does not ship one. - `/keyring/status` on the sidecar reports digests and encryption coverage — never key material. It leaks the same class of reconnaissance as `/status`. **The exception: `/keyring/escrow`.** It is the only operator HTTP endpoint that mutates cluster state, so it is isolated on a dedicated TLS listener and is the only one that authenticates. Callers present a per-site bearer token that the operator mints into a Secret mounted **only** into that site's pod, and **only** while that site is deliberately unsealed. A sealed site carries no token at all. Beyond the token: the body is size-capped, the claimed digest must match a fresh hash of the payload, escrow Secrets are immutable (so an accepted push can only add a version, never rewrite the one a running MySQL is sealed against), and the reconcile loop independently re-reads the stored Secret before it will seal anything. See [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest). The sidecar validates the listener certificate against the CA mounted from the failover group's `spec.tls` Secret. There is no plaintext fallback and redirects are rejected so neither the bearer token nor the keyring can be sent to another endpoint. ## TLS and data-in-flight When `spec.tls.secretName` references a Secret with `ca.crt`, `tls.crt`, and `tls.key`, the operator mounts it into every MySQL pod and renders config that sets `ssl-ca`/`ssl-cert`/`ssl-key` **and**`require-secure-transport=ON`. Without TLS set, MySQL accepts plaintext connections from anything on the pod network. - Replication between sites uses the same MySQL port and the same server certificate — cross-site replication is TLS-protected only when `spec.tls` is set. Mirror a single issuer across sites, or ensure both sites' certs chain to a CA each other trusts. - The sidecar, operator, backup Jobs, and MySQL Shell dump workers all trust the CA in `ca.crt`. When you rotate the CA, rotate the mounted Secret; the operator detects the hash change and rolls pods. - Cert-manager integration is not automatic — Bloodraven expects the Secret to exist. Use a cert-manager `Certificate` resource to populate it. Dragonfly traffic is not covered by `spec.tls`; configure NetworkPolicy and, where required, Dragonfly-native TLS or a service mesh outside Bloodraven's managed flags. Do not expose the Dragonfly admin port to applications. ## Secrets at rest and in memory - **Nothing is written to disk that wasn't already on disk.** The sidecar receives its DSN via `MYSQL_DSN` env var (or reconstructs one from `MYSQL_USER`/`MYSQL_PASSWORD`) and keeps it in memory. It does not write credentials to a file. - **Logs never include passwords.** DSNs are built for in-process use and are never logged; SQL statement errors are truncated before logging. Operator events and condition messages are safe to ship to a log aggregator. - **Secret content is copied into env and files as MySQL requires.** The MySQL pod's environment holds `MYSQL_ROOT_PASSWORD` and the per-role passwords during first boot; after user creation these are no longer required and could be unset, but the operator keeps them in the pod spec for simplicity. Anything that can `kubectl exec` or `kubectl describe pod` can see them. - **Dragonfly auth is optional but recommended for shared namespaces.** When `spec.dragonfly.auth` is set, the operator projects the password Secret into Dragonfly pods as an environment variable and uses it for its own Redis-protocol connections. - **MySQL data at rest is plaintext unless you enable `spec.encryptionAtRest`.** With it enabled, tablespaces, redo, undo, binary and relay logs are encrypted and the keyring lives only on tmpfs — but the key then lives in a Kubernetes Secret, which means **etcd becomes part of your key custody**. Enabling the feature without API-server encryption at rest for Secrets moves the plaintext key from the MySQL disk to the control-plane disk rather than protecting it. See [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#etcd-is-now-part-of-your-key-custody). ## Compromise scenarios ### The operator ServiceAccount token leaks This is the worst-case credential compromise. With operator-level cluster access, an attacker can: 1. **Read every namespace's Secrets.** Including MySQL credential Secrets for every `MysqlFailoverGroup` the operator manages. Treat this as equivalent to leaking all per-tenant MySQL credentials and all TLS private keys referenced by `spec.tls`. 2. **Create or patch arbitrary `Deployment`, `Pod`, `Job`, `CronJob`, `Service`, and `PVC` objects** cluster-wide. Because the operator has full CRUD on these, the attacker has a general-purpose tenancy breakout through pod spec injection. 3. **Write `DNSEndpoint` CRs** that external-dns will push to the public DNS zone, enabling a DNS-hijack attack against any name external-dns manages (not just MySQL names). 4. **Exfiltrate PVC contents** by scheduling a pod that mounts the target PVC — the ServiceAccount can create pods and PVC mounts. 5. **Create or patch Dragonfly StatefulSets and Services** when the feature is enabled, including changing cache/session routing labels. **Mitigations.** Pin the chart's `serviceAccountName`, bind it to a namespace-scoped Role where possible, run the operator in its own namespace, and rely on the audit log to detect token reuse from unexpected source IPs. Rotate the token (delete the ServiceAccount's token Secret) on any suspected compromise — Kubernetes issues a new projected token to the pod automatically. **This is the single largest blast radius in the product.** A multi-operator, namespace-scoped deployment model is on the wishlist but not implemented today. ### The MySQL `operator` / `root` password leaks Inside the DB, `operator` and `root` are equivalent — both carry `ALL PRIVILEGES ... WITH GRANT OPTION`. An attacker with either can: 1. **Read, modify, or drop any schema.** Including running `SELECT ... INTO OUTFILE` on any path MySQL can reach (bounded by `secure_file_priv`, which Bloodraven sets to the data directory). 2. **Create new MySQL users** and grant them arbitrary privileges, surviving a password rotation of the known users. 3. **Run `STOP REPLICA` / `RESET REPLICA ALL` / `CHANGE REPLICATION SOURCE TO ...`** — directly breaking replication or repointing a replica at an attacker-controlled primary. 4. **Trigger `SHUTDOWN`** (unless `SUPER` is gated), rebooting the MySQL instance. The blast radius is **per-failover-group**, not cluster-wide. The operator will detect the resulting replication breakage and attempt recovery or raise `Degraded=True`, but it cannot distinguish malicious writes from legitimate ones — divergent GTIDs are the only signal that reaches the `status.sites[].divergentGtid` field. **Mitigations.** Store credential Secrets with a rotating provider (ESO, sealed-secrets, Vault). Set alerts on `bloodraven_divergent_transactions > 0`. Keep the operator's Secret in the same namespace as the failover group so a namespace-compromise does not cascade. ### The MySQL `app` password leaks Bounded by the `app` grants: read/write across every schema, but **no `SUPER`, no `GRANT OPTION`, no `REPLICATION` privileges**. The attacker cannot: - Create new MySQL users or escalate within MySQL. - Break replication (lacks `REPLICATION CLIENT` / `REPLICATION SLAVE`). - Run `KILL` on other sessions (lacks `PROCESS` unless you deviate from the default). - Drop the MySQL server itself. The attacker *can* run arbitrary SQL against the tenant data. This is the same blast radius as an application SQL-injection — assume any compromise of `app` yields full data read/write for the application's schemas. **Mitigations.** Rotate `appSecret` on detection. The operator propagates password changes via `ALTER USER` on the next reconcile. Consumers that cache connection strings need to reconnect. ### A sidecar pod is compromised A sidecar pod shares the pod network namespace with its `mysqld`. An attacker with code-execution in the sidecar can: 1. **Connect to `mysqld` on `127.0.0.1:3306` using the `operator` credentials** the sidecar holds. This is the same authority as "operator password leaks", but scoped to the one MySQL instance. 2. **Self-fence the MySQL** (write `read-only=ON`, kill its own pod) — the sidecar already does this legitimately, but an attacker can use it for denial-of-service. The peer sidecar will eventually promote, so the net effect is an unscheduled failover. 3. **Write the binlog archive manifest** for that site — corrupting PITR coverage for that site only. Objects already uploaded to the backup bucket are **not** deleted by the sidecar; recovery is possible from the bucket manifest + objects, but `status.pitr.archiveCoverageTo` will mislead operators in the meantime. 4. **Read the MySQL data directory** — the sidecar's mount point on `/var/lib/mysql` is shared with `mysqld`. The blast radius is **one MySQL site** of one failover group. The attacker does **not** gain operator-level Kubernetes privileges — the sidecar runs with a minimal ServiceAccount — and does **not** reach the peer site directly (the peer only trusts its own sidecar and the operator). The worst observable consequence is a forced failover plus data exfiltration from the one site. **Mitigations.** Use a distroless sidecar image, keep its process footprint small, and audit the sidecar image in the same supply-chain pipeline as `mysqld`. Alert on `bloodraven_failovers_total` spikes that coincide with sidecar restarts. ### A backup bucket credential leaks Bounded by the bucket IAM policy — not by Bloodraven. Bloodraven passes the credentials (from `spec.backup.profiles[].storage.secretRef`) to `mysqlsh`, `mysqlbinlog`, and the sidecar archiver as env vars. With the credential, an attacker can: - **Delete or overwrite backup objects** (full dumps and binlogs), destroying the PITR chain. - **Read every dump** — they contain the full database content. **Mitigations.** Scope bucket IAM to the specific profile's prefix. Enable object-lock / versioning / MFA-delete on the bucket. Rotate bucket credentials independently of MySQL credentials. ### The DNS zone trust is compromised Bloodraven writes `DNSEndpoint` CRs; `external-dns` translates them into DNS records. If the DNS provider or external-dns itself is compromised, an attacker can repoint MySQL traffic to an attacker-controlled endpoint. **Bloodraven cannot prevent this.** The only mitigation is defense in depth at the DNS layer — zone- signing, IAM scoping of external-dns, and short TTLs that limit the window during which a malicious repoint holds. ## What Bloodraven does not defend against Explicit non-goals, so the list above is complete: - **Container escape from `mysqld`** — Out of scope; treat `mysqld` as you would any other privileged workload in your cluster. - **Kernel-level exfiltration** of data in MySQL memory — out of scope; use confidential-computing primitives at the node level if required. - **Malicious application SQL** — the operator cannot distinguish a legitimate write from a malicious one. Application-level access control is the app's responsibility. - **Kubernetes API server compromise** — nothing below the API server can be trusted if the API server is compromised. - **Supply-chain attacks on the operator or sidecar images** — pin and mirror per [Production hardening](https://bloodraven.dev/docs/configuration/production-hardening#image-supply-chain). - **DoS of the control plane** — an attacker with `patch` on a CR can trigger reconciles at whatever rate the API server allows. The operator is rate-limited by controller-runtime's workqueue, but a motivated abuser can still produce noise. - **Live compromise of a MySQL pod when `spec.encryptionAtRest` is on** — a running `mysqld` holds decrypted keys in memory by necessity, and the keyring file is readable inside its own pod. Data-at-rest encryption defends stolen volumes, snapshots, and node disks; it does not defend a live process. It is also **not** Oracle "MySQL Enterprise TDE" and Oracle does not present file keyrings as a regulatory-compliance solution. ## Hardening summary If you only read one section, read this. The minimum for a production deployment: 1. Use `spec.credentials` (per-role) — never `spec.secretName` (legacy DSN). 2. Set `spec.tls` and require it in the MySQL clients. 3. Scope `patch` on `MysqlFailoverGroup` tightly — it is the reclone and planned-failover vector. 4. Keep the operator in its own namespace and do not grant its ServiceAccount to anyone else. 5. Apply a NetworkPolicy that permits only the operator to reach sidecar `:8080` and the dashboard to reach operator `:8082`. 6. Store credential Secrets with a rotating provider and rotate on a schedule. 7. Lock down the backup bucket IAM to the prefix the profile uses, and enable versioning / object-lock. 8. Subscribe to divergent-GTID alerts (`bloodraven_divergent_transactions > 0`) — that is your "someone wrote to the wrong primary" tripwire. For the full checklist see [Production hardening](https://bloodraven.dev/docs/configuration/production-hardening). # Architecture ![architecture infographic](https://bloodraven.dev/img/infographics/architecture.webp) Bloodraven manages MySQL async replication failover groups. Each `MysqlFailoverGroup` resource describes **two or more sites**, each running a MySQL pod with a sidecar container. When `spec.dragonfly.enabled=true`, each site also gets a single-replica Dragonfly StatefulSet that acts as a cache/session sidekick for the same failover group. Sites carry explicit roles — `primary-candidate` (promotable on failover) or `dr-only` (passive replica) — and at least two primary-candidate sites are required so an active primary always has a promotion target. The operator continuously polls every site and takes corrective action when the topology deviates from the desired state. Bloodraven uses a **star replication topology** for MySQL: one site is the active primary, and every other site replicates from it. Managed Dragonfly uses the same active-site model: one Dragonfly pod is the master, and non-active sites are configured as replicas of that master. There are no replication chains. See [Multi-site topology](https://bloodraven.dev/docs/architecture/multi-site) for the full role model and sizing guidance. ## High-level overview ```mermaid graph TB subgraph "Kubernetes Cluster" OP[bloodraven operator] subgraph "Site: iad" POD_A[MySQL Pod] SC_A[Sidecar] SVC_A["mysql-orders-iad (Service)"] DF_A[Dragonfly Pod] DFSVC_A["orders-dragonfly-iad (Service)"] end subgraph "Site: pdx" POD_B[MySQL Pod] SC_B[Sidecar] SVC_B["mysql-orders-pdx (Service)"] DF_B[Dragonfly Pod] DFSVC_B["orders-dragonfly-pdx (Service)"] end SVC_PRI["mysql-orders-primary (Service)"] SVC_REP["mysql-orders-replicas (Service)"] DFSVC["orders-dragonfly (active Service)"] end DE[DNSEndpoint CR] ED[external-dns] OP -- "poll (2s base, 30s backoff cap)" --> POD_A OP -- "poll (2s base, 30s backoff cap)" --> POD_B OP -- "INFO replication" --> DF_A OP -- "INFO replication" --> DF_B OP -- "REPLICAOF / REPLTAKEOVER" --> DF_A OP -- "REPLICAOF / REPLTAKEOVER" --> DF_B OP -- "create/update on failover" --> DE ED -- "watches" --> DE ED -- "syncs to DNS provider" --> DNS[DNS Provider] SVC_PRI -- "selector follows active" --> POD_A SVC_REP -- "selector follows healthy replicas" --> POD_B DFSVC -- "selector follows Dragonfly master" --> DF_A DF_B -- "replicates from active" --> DF_A SC_A -- "self-fence if isolated" --> POD_A SC_B -- "self-fence if isolated" --> POD_B ``` ## Operator (`bloodraven`) The operator runs as a single-replica Deployment with controller-runtime leader election enabled as a safety belt. See [Operator availability](https://bloodraven.dev/docs/architecture/operator-availability) for the complete breakdown of what happens — and what doesn't — when the operator is down. For each `MysqlFailoverGroup`, it: 1. Creates a MySQL Deployment, Service, and PVC per site 2. Creates a `mysql--primary` Service whose selector tracks the active site 3. Creates a `mysql--replicas` Service whose selector matches healthy read replicas 4. Runs a polling loop with a 2-second default base interval that connects to each site's MySQL instance and evaluates the replication topology; after a confirmed persistent failure, the group cadence backs off exponentially to a 30-second cap 5. Applies the [state machine](https://bloodraven.dev/docs/operations/failover#state-machine) logic to decide whether to failover, alert, or do nothing 6. On failover, executes the [promotion sequence](https://bloodraven.dev/docs/operations/failover#failover-sequence), updates the DNSEndpoint, and reconciles node taints and Service selectors from subsequent topology observations 7. When Dragonfly is enabled, creates per-site Dragonfly StatefulSets and Services, observes `INFO replication`, keeps replicas attached to the active master, and promotes Dragonfly during planned and emergency failover The operator exposes: - **Prometheus metrics** on `:8080/metrics` - **Health probes** on an auxiliary port - **Status API** on `:8082` (`GET /status`, `GET /active-site`, `GET /ws/status`) ## Sidecar (`bloodraven-sidecar`) Each MySQL pod includes a sidecar container that: - Responds to health and readiness probes - Monitors connectivity to both the operator and **every peer site** (the sidecar is given a comma-separated list of peer sidecar addresses at pod-spec time) - **Self-fences on topology mismatch**: every `peerCheckInterval` tick the sidecar polls the operator's `GET /active-site` endpoint. If the operator-authoritative active site is a different site than this one and MySQL is still writable, the sidecar fences immediately regardless of lease timing. When the operator is unreachable to this sidecar but reachable to a peer, the sidecar also reads the peer's cached view via `GET /peer/active-site` and adopts it when strictly newer than its own. This closes the gap where a stale primary returns after a failover: even if the peer is reachable (so the lease-only rule would never fire), the mismatched `activeSite` triggers an immediate fence. - **Self-fences on full isolation** as a backstop: if the operator AND every peer are unreachable for `leaseTimeout` (default `20s`), it sets `SET GLOBAL super_read_only=ON` on its local MySQL instance. A single reachable peer is enough to keep the primary writable under this rule. - **Startup safety net**: on boot, queries the operator's `GET /active-site` endpoint to determine the current active site. The sidecar fences MySQL first and only clears `super_read_only` if the operator confirms this is the active site; the answer also seeds the shared topology cache so `/peer/active-site` is usable from tick zero. - **Binlog archiver (PITR)**: when `spec.backup.pitr.enabled=true`, a goroutine watches `mysql-bin.index` via inotify and uploads each sealed binlog to the referenced backup profile's storage. Only the primary uploads (gated on `@@read_only`); after a failover the new primary's archiver takes over within one scan cycle. Archived files are pruned once the oldest retained `MysqlBackup` moves past them, using a cutoff timestamp pulled from the operator's `/pitr-cutoff` endpoint. See [Backup and restore → PITR](https://bloodraven.dev/docs/backup-and-restore/backup-restore#point-in-time-recovery-pitr). This self-fencing behavior is a critical safety net. Even if the operator is down, an isolated MySQL instance will not accept writes that could diverge from the other site. The full timing of operator-down + primary-failure is documented in [Operator availability](https://bloodraven.dev/docs/architecture/operator-availability#operator-down--primary-failure). Architectural note: the archiver runs in the sidecar rather than the operator because inotify + direct binlog file reads need the MySQL data PVC mounted. PVCs are typically `ReadWriteOnce` and bound to one node, so a centralized archiver in the operator pod can't mount every failover group's data volume. The sidecar is already per-pod and already knows its local MySQL's role, so it's the natural home. ## Services Each `MysqlFailoverGroup` named `orders` produces these Services: | Service | Purpose | | ----------------------- | ----------------------------------------------------------- | | `mysql-orders-iad` | Direct access to the `iad` site MySQL instance | | `mysql-orders-pdx` | Direct access to the `pdx` site MySQL instance | | `mysql-orders-primary` | Follows the active primary. Apps connect here for writes. | | `mysql-orders-replicas` | Matches healthy read replicas. Apps connect here for reads. | The `-primary` and `-replicas` Services use label selectors that the operator updates when the topology changes. See [App Integration](https://bloodraven.dev/docs/configuration/app-integration) for connection details. When `spec.dragonfly.enabled=true`, the group also produces these Dragonfly Services: | Service | Purpose | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `orders-dragonfly-iad` | Direct access to the `iad` Dragonfly pod for replication wiring and debugging. | | `orders-dragonfly-pdx` | Direct access to the `pdx` Dragonfly pod for replication wiring and debugging. | | `orders-dragonfly` | Redis-compatible application endpoint. Selects only the pod labeled `shipstream.io/dragonfly-role=master` and `shipstream.io/dragonfly-traffic=enabled`. | The active Dragonfly Service does not use DNS steering. Its endpoint set changes when the operator updates role and traffic labels during planned failover, emergency failover, Dragonfly-only promotion, or Dragonfly image rollout. ## PodDisruptionBudget Each failover group gets a `PodDisruptionBudget` named `mysql-` with `minAvailable: 1`. This ensures at least one MySQL instance remains running during voluntary disruptions such as node drains and cluster upgrades. When `spec.dragonfly.enabled=true`, the operator also creates one Dragonfly `PodDisruptionBudget` per site named `-dragonfly-` with `minAvailable: 1`. These PDBs do not make Dragonfly durable, but they avoid voluntary disruptions taking a site's cache/session pod down during routine node drains. ## Labels All resources managed by the operator carry a standard set of labels: | Label | Value | | ------------------------------ | ----------------------- | | `app.kubernetes.io/name` | `mysql` | | `app.kubernetes.io/instance` | `` | | `shipstream.io/failover-group` | `` | | `shipstream.io/site` | `` | | `app.kubernetes.io/managed-by` | `bloodraven` | Pods receive additional labels: | Label | Value | | ----------------------- | ---------------------- | | `shipstream.io/role` | `primary` or `replica` | | `shipstream.io/healthy` | `yes` or `no` | These labels drive Service selectors and can be used for monitoring queries and policy rules. Dragonfly resources use the same `app.kubernetes.io/instance`, `shipstream.io/failover-group`, `shipstream.io/site`, and `app.kubernetes.io/managed-by` labels, with `app.kubernetes.io/name=dragonfly`. Dragonfly pods also receive `shipstream.io/dragonfly-role` (`master` or `replica`) and `shipstream.io/dragonfly-traffic=enabled`; the active Service selector requires both `dragonfly-role=master` and `dragonfly-traffic=enabled`. ## Replication topology Bloodraven uses MySQL asynchronous replication with GTID-based positioning. At any given time: - One site is the **primary** (`read_only=0`, `super_read_only=0`) and accepts writes - The other site is the **replica** (`read_only=1`) and replicates from the primary The operator does not configure multi-source replication or group replication. It manages a simple primary-replica pair and handles promotion and demotion through direct SQL commands. When Dragonfly is enabled, Bloodraven treats it as cache/session state rather than durable application data. Planned failover waits for the target Dragonfly replica to catch up and then promotes it with `REPLTAKEOVER`. Emergency failover attempts Dragonfly promotion best-effort within a bounded budget; if Dragonfly cannot preserve sessions, MySQL failover still completes. # Multi-site topology ![multi site infographic](https://bloodraven.dev/img/infographics/multi-site.webp) Bloodraven supports any number of sites (≥ 2) per `MysqlFailoverGroup`. Each site declares a **role** that controls its behaviour during failover: | Role | Auto-promoted? | Typical use | | ----------------------------- | -------------- | ------------------------------------------------- | | `primary-candidate` (default) | yes | active/standby pair inside a region | | `dr-only` | no | cross-region follower, kept for disaster recovery | | `read-only` | no | site-specific application read pool | The spec must contain at least **two** `primary-candidate` sites so the operator always has a promotion target for the current primary. Any number of `dr-only` or `read-only` sites may be appended. A reader does not count toward the two-candidate minimum. Bloodraven uses these terms consistently: | Term | Meaning | | -------------- | ----------------------------------------------------------------------------------------- | | Reader | A site with effective role `read-only`. | | Follower | Any non-active site with replication metadata, including candidate, DR, and reader sites. | | Standby | A healthy, promotable `primary-candidate` follower. | | Active primary | The unique observed writable site, which must have role `primary-candidate`. | ## Replication topology Bloodraven uses a **star topology**: one site is the active primary, and every other site replicates from it. There are no replication chains. ```text primary (iad, primary-candidate) / | \ / | \ (pdx) (fra) (reader) primary-candidate dr-only read-only ``` During planned or emergency failover, the newly promoted primary becomes the direct source for every remaining follower, including `dr-only` and `read-only` sites. Bloodraven also detects a healthy-looking follower that still points through another replica after restart or topology drift. Before repointing, the operator requires exactly one writable promotable primary and checks that its `GTID_EXECUTED` contains the follower's executed set. It repeats containment after `STOP REPLICA` to close the applier race, then uses `CHANGE REPLICATION SOURCE TO` and `START REPLICA` without resetting replication metadata. Divergence is reported as `sourceConvergenceState: Blocked` with reason `GTIDDiverged`; Bloodraven does not silently abandon follower transactions. ## Failover target selection When the active site becomes unreachable and at least one `primary-candidate` replica is reachable, the operator promotes the best candidate using this order: 1. **GTID freshness.** The operator queries `@@gtid_executed` on every eligible primary-candidate replica in parallel and picks the one whose executed set is strictly a superset of the others. This is the primary selector: promoting the freshest replica minimises transactions lost to the async-replication RPO window. 2. **Priority tiebreaker.** When multiple replicas have equivalent GTID sets (common in healthy clusters), the first entry in `spec.splitBrainPolicy.sitePriorities` that is currently eligible wins. Entries not in the list fall through to declared site order. 3. **Declared order.** Final tiebreaker for replicas that share GTID sets and are not named in the priority list. `dr-only` and `read-only` sites are never auto-promoted, selected by planned failover, used as active DNS targets, or chosen as clone donors. A reader that becomes writable is fenced on every topology poll, even if it is the only writable site. Readers also never apply or remove node taints; supplied `lbIP` and `taintNodeSelector` fields are accepted for manifest compatibility but ignored. If *no* `primary-candidate` replica is reachable when the primary fails, the operator emits a `NoPrimary` alert and takes no action. Manual promotion of a `dr-only` site is deliberately outside automatic failover; convert it to `primary-candidate` first if you intend to make it eligible for [planned failover](https://bloodraven.dev/docs/operations/planned-failover). ## Split-brain resolution `spec.splitBrainPolicy.sitePriorities` doubles as the split-brain tiebreaker. When more than one site is simultaneously writable and the operator cannot infer a winner from its own recent failover history (for example, after a fresh deploy or an operator restart that lost in-memory state), the first entry in `sitePriorities` that is currently writable and primary-candidate is promoted; every other writable site is fenced (`SET GLOBAL super_read_only=ON`). If `sitePriorities` is empty — or if no entry in the list is currently a writable primary-candidate — the operator alerts only and requires manual resolution. This matches the default "don't guess" behaviour of earlier Bloodraven versions. ::warning Split-brain auto-resolution is a policy decision, not a safety feature. Writes accepted on losing sites that did not replicate to the winner are lost when those sites are fenced. The existing divergent- GTID detection will block auto-rejoin of any losing site whose GTID set contains transactions the winner never saw; those transactions are only recoverable via re-clone. :: ## Example: three-site cross-region DR Two in-region `primary-candidate` sites (active/standby HA) plus a cross-region `dr-only` follower: ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: sites: - name: iad role: primary-candidate zone: iad-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.0.10 storage: { storageClassName: gp3, size: 500Gi } - name: pdx role: primary-candidate zone: pdx-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: pdx lbIP: 10.1.0.10 storage: { storageClassName: gp3, size: 500Gi } - name: fra role: dr-only zone: fra-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: fra lbIP: 10.2.0.10 storage: { storageClassName: gp3, size: 500Gi } splitBrainPolicy: sitePriorities: [iad, pdx] # ... rest of spec ... ``` Failover behaviour: - If `iad` goes down, `pdx` is promoted (first priority, primary- candidate, reachable replica). - If `iad` *and* `pdx` both go down, the operator alerts — it will **not** auto-promote `fra` because `fra` is `dr-only`. Manual promotion would use the forthcoming planned-failover API. - If two sites become simultaneously writable (split-brain), the winner is the first `sitePriorities` entry currently writable; others are fenced. ## Sidecar peer awareness Each pod's sidecar is given the internal Service addresses for every non-self site, including `dr-only` sites and readers, via the `PEER_ADDRESSES` env var. The sidecar tracks per-peer liveness and only self-fences when the operator **and every peer** are unreachable beyond `spec.sidecar.leaseTimeout`. A single reachable peer is enough to keep the primary writable. This quorum rule matters as the number of sites grows: a split where the primary can still reach at least one peer is preserved as a legitimate-writes window rather than collapsing to a self-fenced outage. Reader peers provide connectivity and relay fresh operator-authoritative active-site observations; they gain no promotion or primary authority. A reachable peer without fresh authoritative topology can still suppress the lease-only all-peers-unreachable fence. This is retained compatibility behavior, not a quorum guarantee. ## Example: two candidates and a reader The reader omits candidate/DR-only placement fields, uses a lower endpoint lag threshold, and customizes only its own client Service and MySQL config: ```yaml spec: replication: maxLagSeconds: 300 readOnlyMaxLagSeconds: 30 sites: - name: iad role: primary-candidate zone: iad-1a taintNodeSelector: shipstream.io/site.orders: iad lbIP: 10.0.0.10 storage: {storageClassName: gp3, size: 500Gi} - name: pdx role: primary-candidate zone: pdx-1a taintNodeSelector: shipstream.io/site.orders: pdx lbIP: 10.1.0.10 storage: {storageClassName: gp3, size: 500Gi} - name: reader role: read-only zone: iad-1b mysqlConf: innodb_buffer_pool_size: 4G serviceTemplate: type: LoadBalancer externalTrafficPolicy: Local annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" storage: {storageClassName: gp3, size: 500Gi} splitBrainPolicy: sitePriorities: [iad, pdx] ``` ## Sizing and compatibility - Minimum: 2 sites, both `primary-candidate`; readers are additional sites. - Maximum: 16 sites. Practical limits derive from replication cost on the primary (each replica opens an I/O thread) and from DNS/LB churn at failover time. - Every site receives a Deployment, PVC, client MySQL Service, internal administrative Service, and per-site MySQL ConfigMap. Reader client endpoints have stricter health gating than other site Services. ## Known limitations - `dr-only` sites cannot be auto-promoted. Region-level loss of every primary-candidate site requires manual intervention. - Planned failover targets only `primary-candidate` sites. A `dr-only` or `read-only` site must be deliberately reclassified, with `lbIP` and `taintNodeSelector` supplied, before it can be promoted. # Multi-cluster disaster recovery This page is the operational runbook for recovering a `MysqlFailoverGroup` into a separate Kubernetes cluster when the source cluster is lost. It covers every step from topology to DNS cutover using **only the existing `MysqlFailoverGroup` CRD surface** (available today on `main`). No new CRDs or controllers are required to execute this runbook. WISHLIST #7 introduces `MysqlStandbyCluster` as a **passive observability layer** that surfaces source-archive freshness via `BucketReadable` / `SourceConfigKnown` conditions. Phase 2 adds a readiness gate (`Restorable`); Phase 3 adds one-command activation. See [What MysqlStandbyCluster adds](https://bloodraven.dev/#what-mysqlstandbycluster-phase-1-adds-today) at the bottom of this page for details on what ships in that first phase and what remains in later phases. --- ## Topology overview Cross-cluster DR in Bloodraven v1 uses the shared object store (S3 or compatible) as the **only** channel between the source and DR clusters. There is no cross-cluster replication link, no operator-to-operator RPC, and no federation. ```mermaid flowchart LR subgraph SourceCluster["Source cluster (us-west-prod)"] direction TB SOp["Bloodraven operator"] MFG["MysqlFailoverGroup (orders)"] SidePri["Sidecar on active primary\n!read_only → writes binlogs"] SideRep["Sidecar on replicas\nread_only → idle"] SOp -->|"reconciles"| MFG MFG --> SidePri MFG --> SideRep end subgraph Bucket["Shared S3 bucket (cross-cluster bus)"] direction TB Dumps["prefix/backup-name/\n(full dumps + @.json sentinel)"] Binlogs["prefix/binlogs/\n(sealed binlogs + per-site manifest)"] end subgraph DRCluster["DR cluster (us-east-prod)"] direction TB DOp["Bloodraven operator"] NewMFG["MysqlFailoverGroup\n(bootstrapped from dump + PITR)"] DOp -->|"reconciles"| NewMFG end SidePri -->|"PUT sealed binlogs"| Binlogs SOp -->|"PUT full dumps via Job"| Dumps NewMFG -->|"GET dump + binlogs\nduring initFromBackup"| Dumps NewMFG -->|"GET binlog manifests"| Binlogs ``` Key points: - The **source cluster writes**: full dumps (operator-driven backup Jobs) and sealed binlogs (sidecar archiver on the active primary, gated on `!@@read_only`). The upload switches to a new primary within one archiver scan cycle after an intra-cluster failover. - The **DR cluster only reads** during bootstrap. It does not run a binlog archiver of its own — there is nothing to archive from a cluster that does not yet exist. - `dr-only` site roles (`api/v1alpha1/types.go:280-283`, [Multi-site topology](https://bloodraven.dev/docs/architecture/multi-site)) are a separate concept: they are passive replicas inside the **same** cluster as the source MFG and are never auto-promoted. Cross-cluster DR is distinct from this. --- ## Bucket and IAM layout ### Bucket structure Bloodraven writes to the bucket prefix configured in `spec.backup.profiles[].storage.s3.prefix`. The layout under that prefix: ```text / / # one directory per successful full dump @.json # dump sentinel: GTIDs, size, completion time .sql.gz # mysqlsh dumpInstance artifacts binlogs/ # PITR archive (when spec.backup.pitr.enabled=true) manifest-.json # per-site sealed-binlog manifest / .enc # sealed binlogs (possibly encrypted) dr-cursors/ # Phase 2: reserved, not written in this release ``` The DR cluster only needs read access to `/` to execute this runbook. Write access to `dr-cursors/` is reserved for the `MysqlStandbyCluster` verification/readiness feature (Phase 2 of WISHLIST #7). ### Minimum IAM policy — AWS Grant the DR cluster's service account or IAM role the following policy. Replace `shipstream-backups` and `orders/west` with your actual bucket name and prefix. **DR cluster read-only policy:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DRReadOnly", "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetObject"], "Resource": [ "arn:aws:s3:::shipstream-backups", "arn:aws:s3:::shipstream-backups/orders/west/*" ] } ] } ``` Scope the `ListBucket` resource to the bucket name (not a prefix — S3 requires bucket-level `ListBucket`), and the `GetObject` resource to the prefix subtree. The DR restore Job uses the `credentialsSecret` you reference in `spec.initFromBackup.source.s3`; that Secret must contain AWS credentials (or the pod must have IRSA/Workload Identity access) that the above policy covers. ::note **Phase 2 cursor writes** When `MysqlStandbyCluster` ships (Phase 2), the DR side needs one additional write permission scoped to `dr-cursors/*` only. Do not add it until that phase is in use. :: ### GCS / S3-compatible (MinIO, Ceph) Bloodraven uses the AWS SDK v2 with a configurable `endpoint`. Set `spec.initFromBackup.source.s3.endpointURL` (or `spec.backup.profiles[].storage.s3.endpointURL` on the source) to your endpoint. Minimum equivalent permissions: | Cloud | Equivalent permissions | | ------------- | ------------------------------------------------------------ | | GCS (interop) | `storage.objects.list` + `storage.objects.get` on the prefix | | MinIO | `s3:ListBucket` + `s3:GetObject` — same policy syntax | | Ceph RGW | Same as MinIO; set `endpointURL` to your RGW endpoint | --- ## Encryption passphrase distribution When the source profile uses `spec.backup.profiles[].encryption`, every dump artifact and archived binlog is wrapped in the Bloodraven BRV1 format (AES-256-GCM envelope). The DR restore Job must have the same passphrase available. The operator does not distribute passphrases across clusters. Follow these steps before triggering a recovery: **Step 1.** Copy the passphrase from the source cluster into a temporary file on a trusted workstation: ```bash PASSFILE=$(mktemp) kubectl -n orders get secret orders-backup-passphrase \ -o jsonpath='{.data.passphrase}' | base64 -d > "$PASSFILE" ``` **Step 2.** Point `kubectl` at the DR cluster and create the same Secret in the DR namespace from that file: ```bash kubectl -n orders create secret generic orders-backup-passphrase \ --from-file=passphrase="$PASSFILE" shred -u "$PASSFILE" ``` **Step 3.** Reference it in the DR restore manifest via `spec.initFromBackup.decryption.passphraseSecret.name`. See [Backup encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption) for full field semantics. If the passphrase Secret is missing or wrong, the restore Job fails fast with: ```text Warning RestoreBuildFailed initFromBackup source is encrypted but no passphrase is available; set initFromBackup.decryption.passphraseSecret or restore the profile's encryption.passphraseSecret ``` This is a clean, recoverable error — fix the Secret and delete the failed Job; the reconciler rebuilds it. --- ## Source fencing checklist Before you trigger a recovery into the DR cluster, confirm the source is genuinely down. Promoting the DR copy while the source is still accepting writes produces two writable clusters. Bloodraven v1 does not automatically detect or resolve cross-cluster split-brain — divergent GTIDs accumulate on the source side and can only be audited post-hoc (see [Durability and RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo#the-exact-bound-divergentgtid)). Use at least two of these three signals before proceeding: **Signal 1 — Source operator `/active-site` endpoint returns 5xx:** ```bash curl -f 'http://:8082/active-site?namespace=orders&group=orders' # Healthy: {"activeSite":"iad"} # Down: curl: (22) The requested URL returned error: 503 ``` The operator's auxiliary HTTP server (`cmd/bloodraven/main.go:361`) serves `/active-site` on port `8082` and requires `namespace` plus `group` query parameters. It returns 503 when it cannot reach any writable site. **Signal 2 — Source cluster API server is unreachable:** ```bash kubectl --context= get nodes --request-timeout=10s # Error from server: timeout waiting for server response ``` **Signal 3 — Source MySQL endpoint is TCP-unreachable from a third vantage point:** ```bash # From a network location that is NOT one of the two clusters nc -zv 3306 # Connection refused / timeout ``` If any signal is ambiguous — for example, the API server is reachable but appears degraded — wait for a clearer signal rather than promoting. The cost of delaying a few minutes is far lower than the cost of a split-brain incident. ::warning **Split-brain risk** Bloodraven v1 does not fence the source before or after you promote the DR cluster. Issuing `initFromBackup` on the DR cluster when the source is still alive produces two writable groups. Use the three signals above to be certain before proceeding. :: --- ## Recovery procedure This walkthrough assumes: - The source `MysqlFailoverGroup` is named `orders`, in namespace `orders`, and had PITR enabled under a profile named `nightly-s3` stored at `s3://shipstream-backups/orders/west`. - The DR cluster has network access to the same S3 bucket. - The passphrase Secret has been mirrored (see above). - The DR cluster has a fresh namespace `orders` with Bloodraven installed (same operator version as the source; see [Install production](https://bloodraven.dev/docs/get-started/install-production)). ### Step 1 — Identify the recovery point Locate the most recent successful dump in the source bucket: ```bash aws s3 ls s3://shipstream-backups/orders/west/ --recursive \ | grep '@\.json' | sort -k1,2 | tail -5 ``` Each directory under `orders/west/` that contains `@.json` is a completed dump. The `@.json` holds the completion timestamp, GTID set, and dump size. Pick the most recent `@.json` whose directory you want to recover from. Note its prefix — for example, `orders/west/orders-nightly-20260520`. To target a specific point in time (e.g., 14:32 UTC before a bad migration), note the desired `stopDatetime`. Binlogs replayed after the dump will be bounded by the newest sealed binlog archived before the event; the unarchived tail on the lost primary's PVC is not recoverable (see [PITR and the RPO window](https://bloodraven.dev/docs/architecture/durability-and-rpo#pitr-and-the-rpo-window)). ### Step 2 — Apply the DR `MysqlFailoverGroup` Create a `MysqlFailoverGroup` on the DR cluster configured for the DR environment's nodes, IPs, and DNS. Set `spec.initFromBackup` to pull from the source bucket. **Source group configuration (reference — source cluster):** ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: sites: - name: iad role: primary-candidate zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad lbIP: 10.0.10.11 storage: { storageClassName: gp3, size: 500Gi } - name: iad-2 role: primary-candidate zone: us-east-1b taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: iad-2 lbIP: 10.0.10.12 storage: { storageClassName: gp3, size: 500Gi } secretName: mysql-operator-creds dns: hostname: orders-east.example.com ttl: 60 backup: profiles: - name: nightly-s3 storage: type: S3 s3: bucket: shipstream-backups prefix: orders/west region: us-west-2 credentialsSecret: s3-backup-creds encryption: passphraseSecret: name: orders-backup-passphrase pitr: enabled: true profileName: nightly-s3 ``` **DR cluster recovery manifest:** ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: # DR-local node topology — different nodes, IPs, and zones from source sites: - name: east-1 role: primary-candidate zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: east-1 lbIP: 10.1.20.11 # DR cluster's LB IP storage: { storageClassName: gp3, size: 500Gi } - name: east-2 role: primary-candidate zone: us-east-1b taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: east-2 lbIP: 10.1.20.12 storage: { storageClassName: gp3, size: 500Gi } secretName: mysql-operator-creds # must exist in DR namespace dns: hostname: orders-east.example.com ttl: 60 initFromBackup: source: s3: bucket: shipstream-backups prefix: orders/west/orders-nightly-20260520 # exact dump directory region: us-west-2 credentialsSecret: s3-dr-readonly-creds # DR read-only creds decryption: passphraseSecret: name: orders-backup-passphrase # mirrored from source pointInTime: # omit if recovering to dump GTID stopDatetime: "2026-05-20T14:32:00Z" ``` Apply to the DR cluster: ```bash kubectl --context= apply -f dr-orders.yaml ``` ### Step 3 — Monitor the restore ```bash kubectl --context= -n orders get mysqlfailovergroup orders -w ``` Watch `status.restore.phase`. The sequence: | Phase | Meaning | | ----------- | ------------------------------------------------------------------------ | | `Pending` | Restore Job not yet created | | `Running` | `util.loadDump()` is running (and optionally `mysqlbinlog` replay after) | | `Succeeded` | Dump and any PITR replay completed; bootstrap continues | | `Failed` | Inspect `status.restore.message` and Job logs | Full status: ```bash kubectl --context= -n orders describe mysqlfailovergroup orders ``` The `initFromBackup` path is one-shot and runs before the failover group is considered ready (`api/v1alpha1/backup_types.go:552-557`). Once `status.restore.phase == Succeeded`, bootstrap (clone, replication, fencing) completes normally. ### Step 4 — Verify the DR group is healthy ```bash kubectl --context= -n orders get mysqlfailovergroup orders -o yaml \ | grep -A5 'conditions:' ``` Wait for `Ready=True`. Check that at least one site is `writable` and replication is up on the peer: ```bash kubectl --context= -n orders get mysqlfailovergroup orders \ -o jsonpath='{.status.activeSite}' ``` ### Step 5 — DNS cutover Bloodraven writes a `DNSEndpoint` CR per site using `external-dns/endpoint-controller.io/v1` (`api/v1alpha1/types.go:371-384`, `internal/platform/dns.go:23-31`). The operator controls the **per-cluster** DNS record for `spec.dns.hostname` in whatever cluster it runs in. It does **not** control the global application-facing record. Choose one of these cutover patterns: **Option A — Weighted CNAME with health checks (recommended for production)** Maintain a weighted CNAME (Route53, Google Cloud DNS, Akamai GTM) or a GSLB record that points at both clusters. On source loss: 1. Set source weight to 0 (or let health checks drop it automatically if the source LB IP is unreachable). 2. Set DR weight to 100. 3. TTL propagation time depends on your DNS TTL. Pre-reducing TTL to ≤ 60 s before an expected maintenance window shortens this. **Option B — Manual flip (simplest)** ```bash # Point the application CNAME at the DR cluster's DNS name aws route53 change-resource-record-sets --hosted-zone-id Z123 --change-batch '{ "Changes": [{ "Action": "UPSERT", "ResourceRecordSet": { "Name": "db.example.com", "Type": "CNAME", "TTL": 60, "ResourceRecords": [{"Value": "orders-east.example.com"}] } }] }' ``` **Option C — GSLB / Akamai GTM** Mark the source origin as offline in the GTM policy. The GSLB health monitor will have already removed it from rotation if the TCP probe to MySQL is failing. After DNS propagation, confirm applications are connecting to the DR cluster (check `status.activeSite` and replication lag on the DR group). --- ## Failback narrative When the source cluster returns, the original path is reversible. This is a manual, prose-only procedure in v1; Phase 4 of WISHLIST #7 will document and partially automate it. The high-level steps: 1. **Confirm the DR cluster is steady-state primary.** New writes are landing on the DR cluster; intra-cluster replication (if multi-site) is healthy. 2. **Wipe the source cluster's MySQL state.** Delete the old `MysqlFailoverGroup` CR. The operator cascades to Deployments, Services, and DNSEndpoints; PVCs are **not** cascaded — delete them explicitly: ```bash kubectl --context= -n orders delete mysqlfailovergroup orders kubectl --context= -n orders delete pvc \ -l bloodraven.shipstream.io/failover-group=orders ``` 3. **Ensure the DR group's backup prefix does not collide with the original source prefix.** A directional prefix convention avoids ambiguity — for example, use `orders/east/` for the DR group's dumps and `orders/west/` for the original source. Update the DR group's `spec.backup.profiles[].storage.s3.prefix` accordingly before the DR group takes its first backup. 4. **Stand up a fresh `MysqlFailoverGroup` on the source cluster** with `spec.initFromBackup` pointing at the DR cluster's new prefix (the backup that the now-primary DR cluster took). This is the same pattern as the original recovery, but in reverse. Wait for `status.restore.phase=Succeeded`. 5. **Cut DNS back if desired.** This step is optional and human-driven. Bloodraven v1 does not auto-rebalance traffic across clusters. Phase 4 (WISHLIST #7) introduces `MysqlStandbyCluster` symmetric failback: a second standby CR on the returning source cluster pointing at the DR cluster's new prefix, with the same continuous verification and one-command activation path that Phase 3 provides for the forward direction. --- ## What `MysqlStandbyCluster` (Phase 1) adds today WISHLIST #7 Phase 1 introduces `MysqlStandbyCluster` — a new CRD that lives on the DR cluster and continuously monitors the source bucket. **What Phase 1 ships:** - A new `MysqlStandbyCluster` CR (`shipstream.io/v1alpha1`, short name `msc`) that names the relationship between a source MFG's backup archive and this DR cluster. - A passive reconciler (`MysqlStandbyClusterReconciler`) that runs a discovery loop on a configurable cadence (default 5 minutes): - Lists objects under the configured S3 prefix to find the most recent successful full dump and per-site binlog manifests. - Populates `status.discovered` with the dump location, completion time, GTID set, and binlog window timestamps. - Stamps `BucketReadable=True/False` based on whether the bucket scan succeeded. - Stamps `SourceConfigKnown=True/False` based on whether at least one dump and one binlog manifest were found. - No mysqld is started, no dump is loaded, and no `MysqlFailoverGroup` is materialized. Phase 1 is purely observational. **What Phase 1 does NOT ship:** - Continuous restore verification (`Restorable` condition) — Phase 2. - One-command activation (`dr-activate` kubectl plugin + a future activate-request state machine) — Phase 3. - Failback tooling — Phase 4. The spec fields that drive Phase 2 verification and Phase 3 activation are **not** part of v1alpha1 yet. They will be added back (backward-compatibly) when that code ships, rather than shipping inert fields that enforce nothing today. The `template` field is the exception: it is required at create-time so the full activated topology is validated while the cluster is calm, not mid-incident during a promote. An on-call engineer recovering from a cluster loss still executes this runbook manually. Phase 1's value is operational visibility: you can see at a glance whether the source bucket is readable and whether a dump + binlog window exist, without having to run `aws s3 ls` yourself. **Sample `MysqlStandbyCluster` CR:** ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlStandbyCluster metadata: name: orders-east-from-west namespace: orders spec: transport: ObjectStore source: failoverGroupName: orders cluster: us-west-prod # informational namespace: orders profileName: nightly-s3 storage: type: S3 s3: bucket: shipstream-backups prefix: orders/west region: us-west-2 credentialsSecret: s3-dr-readonly-creds decryption: passphraseSecret: name: orders-backup-passphrase # template declares the MysqlFailoverGroup to materialise on activate # (Phase 3). It is validated as a full MysqlFailoverGroupSpec at create # time — dns and at least two primary-candidate sites are required — so fill # it in now, not during an incident. See examples/standby-cluster.yaml for # the canonical reference. template: name: orders-east-dr spec: image: mysql:9.7 credentials: operatorSecret: orders-mysql-operator appSecret: orders-mysql-app dns: hostname: orders-east.example.com ttl: 60 sites: - name: east-az1 role: primary-candidate zone: us-east-1a taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: east-az1 lbIP: 10.0.1.1 storage: storageClassName: fast-ssd size: 100Gi - name: east-az2 role: primary-candidate zone: us-east-1b taintNodeSelector: shipstream.io/failover-group.orders: "true" shipstream.io/site.orders: east-az2 lbIP: 10.0.2.1 storage: storageClassName: fast-ssd size: 100Gi # freshness.discoveryInterval (default 5m) controls the scan cadence. # The Phase 2 verification/staleness knobs and the Phase 3 activate block # are not part of v1alpha1 yet — they will be added back # (backward-compatibly) when that code ships. Phase 1 reads source.storage # and source.decryption to drive the discovery loop. ``` Check status: ```bash kubectl -n orders get mysqlstandbycluster orders-east-from-west -o yaml ``` Look for `status.discovered.dumpLocation`, `status.discovered.dumpCompletionTime`, and the `BucketReadable` and `SourceConfigKnown` conditions. **Planned follow-up phases (tracking issue: WISHLIST #7):** - **Phase 2** — Continuous DR readiness: the reconciler schedules periodic `MysqlBackupVerification` runs against the discovered dump + binlog window, publishing a `Restorable` condition and a `bloodraven_dr_restorable_timestamp_seconds` gauge. The source operator gains awareness of `dr-cursors/` sentinel objects to prevent premature binlog pruning. Prerequisite: WISHLIST #43 (PITR E2E scenarios). - **Phase 3** — One-command activation: `kubectl bloodraven dr-activate` - a future confirm-token gate materializes a writable `MysqlFailoverGroup` from the discovered dump. The activation state machine is phase-stamped and durable across operator restarts. The spec field that carries the confirm token will be added when this lands. - **Phase 4** — Failback tooling: symmetric `MysqlStandbyCluster` on the returning source cluster + extended runbook. Until Phase 3 ships, activation always requires the manual `spec.initFromBackup` steps described in this runbook. # Why not Group Replication? ![why not group replication infographic](https://bloodraven.dev/img/infographics/why-not-group-replication.webp) The single most common question about Bloodraven is: *"Why run async replication with an external operator when MySQL ships both Group Replication (GR) and InnoDB Cluster?"* This page is the architectural answer so readers can stop asking and pick the right tool for their situation. ## TL;DR Bloodraven is optimized for the **two-site, geographically-separated, accept-non-zero-RPO** deployment. Group Replication is optimized for the **three-or-more-node, low-latency, zero-RPO** deployment. They're different design points, and Bloodraven solves problems Group Replication doesn't — most importantly: staying writable when the cross-site link is slow, flappy, or down. ## What Bloodraven trades away - **RPO is not zero.** A hard primary loss can lose every transaction that committed on the dying primary but hadn't yet shipped to the replica over async replication. Under healthy operation this window is typically sub-second, but it exists. If your data model cannot accept this, Bloodraven is the wrong tool — use Group Replication (or a higher-tier system like Spanner / CockroachDB). - **No in-database conflict resolution.** Because only one site writes at a time, Bloodraven cannot merge concurrent writes from two sites. Conflicts are resolved by the operator's "one primary at a time" invariant; anything that breaks that invariant (split brain) is surfaced as a condition for a human, not silently merged. ## What Bloodraven keeps **Zero commit latency.** A primary write acknowledges as soon as it has fsynced to the local binlog — the same latency profile as a standalone MySQL. No quorum round-trip, no cross-site ACK. For two sites separated by ≥ 20 ms of network latency, GR's certification - quorum on every commit typically adds 40–80 ms to p50 write latency. Bloodraven's async model adds zero. **Single-node write availability.** The primary accepts writes even when the other site is unreachable. GR requires a majority of the group online to accept writes, so losing half of a two-node group means the remaining node is read-only; losing two of three means the whole cluster is read-only. Bloodraven will promote a surviving site to writable the moment it detects the primary is gone and the replica is healthy (see [Failover sequence](https://bloodraven.dev/docs/operations/failover#failover-sequence)). **No quorum requirement.** Two sites is a legitimate topology. GR with two members is a pathological configuration — any partition or node loss makes the group inquorate. Operators who want two-site HA with GR are forced to invent a "witness" third node somewhere, which introduces its own set of cross-region headaches (where does the witness live? What happens when the witness is isolated?). Bloodraven's sidecar self-fencing layer performs the "am I still authoritative?" check without a quorum. **Simpler mental model.** There is always exactly one primary; the other site is a replica or is fenced. Topology never has to pick between N possible primaries, negotiate a view change, or resolve certification conflicts. The operator's state machine has [four per-site states](https://bloodraven.dev/docs/operations/failover#state-machine) and a small cross-site truth table — a single developer can hold the whole thing in their head, which is load-bearing when you're debugging at 03:00. **Works across zones with real latency.** Group Replication's paper-published performance numbers assume sub-millisecond inter-node latency, because every commit serializes through the group's certification protocol. At 20-100 ms cross-region latency (the typical two-datacenter or two-region deployment), GR is functional but expensive on write throughput, and any network blip triggers a view change. Async replication + a supervisor is the standard answer at that latency tier for a reason. **Sidecar self-fencing.** The sidecar on each MySQL pod refuses to accept writes if it can reach neither the operator nor the peer for `spec.sidecar.leaseTimeout` (default 20 s). This closes the split-brain window that async replication alone would leave open. See the [Sidecar description](https://bloodraven.dev/docs/architecture/architecture#sidecar-bloodraven-sidecar). ## When Group Replication is actually the right answer Bloodraven is not *always* the right answer — be honest about when it isn't: - **Zero-RPO is a product requirement.** Financial ledgers, inventory-as-source-of-truth, anything where "we lost a second of writes" is a customer-visible failure. Group Replication is what you want. - **Three or more nodes are already on the table.** If your topology already has three MySQL nodes for HA (and the write path is willing to pay quorum latency), GR turns that into synchronous-ish replication with no external operator. Bloodraven's two-site architecture doesn't fit. - **Low inter-node latency.** Single-AZ, single-DC, single-rack deployments don't pay GR's latency cost, because the cost is small in that environment. - **You cannot tolerate split-brain resolution by human.** Bloodraven's response to `writable/writable` is to alert and wait for an operator (or, opt-in, to fence a pre-configured loser — see [`spec.splitBrainPolicy`](https://bloodraven.dev/docs/operations/failover#split-brain-resolution)). If your runbook requires the cluster to auto-pick a winner in every case without data reconciliation, use GR. ## The honest tradeoff Bloodraven and Group Replication solve the same top-level problem ("keep MySQL writable when bad things happen") from two different vantage points: | Concern | Group Replication | Bloodraven | | -------------------------------------------- | --------------------------------------------- | ------------------------------------------------- | | RPO on hard primary loss | 0 | ≈ `secondsBehindSource` of the replica at failure | | Commit latency | 1 cross-node round-trip | 1 local fsync | | Minimum nodes to tolerate 1 failure | 3 | 2 | | Write availability during a partition | Majority side only | The reachable side (operator arbitrates) | | Conflict resolution | Certification (may abort commits) | Single-writer invariant (no conflicts possible) | | Operational complexity | View changes, certification, group membership | Primary/replica + one external operator | | Typical inter-node latency sweet spot | < 5 ms | Doesn't care; tested at 20–100+ ms | | Supervisor required for DNS/traffic steering | Yes (MySQL Router / InnoDB Cluster) | Yes (Bloodraven itself) | Bloodraven picks the column on the right of every row. If the column on the left describes your situation better, run Group Replication. ## Related reading - [Architecture](https://bloodraven.dev/docs/architecture/architecture) — how the operator, sidecars, and Services fit together. - [Failover](https://bloodraven.dev/docs/operations/failover) — state machine, failover sequence, anti-flap cooldown, split-brain handling. - [Getting started](https://bloodraven.dev/docs/get-started/getting-started) — stand up a two-site failover group end-to-end. # Operator availability ![operator availability infographic](https://bloodraven.dev/img/infographics/operator-availability.webp) Bloodraven is deliberately **not** a highly-available control plane. It runs as a single-replica Deployment with controller-runtime leader election enabled as a safety belt, and this page documents exactly what that means for the data plane — especially for the uncomfortable case of a primary failure that happens *while the operator is also down*. ## Design stance - **One active operator at a time.** Leader election is on by default (`charts/bloodraven/values.yaml`), but scaling `replicaCount` above 1 does not buy a faster failover. Whichever replica holds the lease is the sole writer of status, DNS, and promotion commands; every other replica is idle and just waiting to take over. - **The MySQL data plane does not need the operator to serve traffic.** A healthy primary and replica keep serving reads and writes with zero operator involvement. The operator is on the failure-detection and promotion path, not the request path. - **Correctness is preserved during operator downtime.** The [sidecar self-fencing](https://bloodraven.dev/docs/architecture/architecture#sidecar-bloodraven-sidecar) layer stops any MySQL instance from accepting writes when it can reach neither the operator nor its peer for `spec.sidecar.leaseTimeout` (default `20s`). See [Operator-down + primary failure](https://bloodraven.dev/#operator-down--primary-failure) below. - **Availability is not preserved during operator downtime.** A primary failure while the operator is down is an outage for writes until the operator comes back. This is the explicit tradeoff. If you need lower MTTR, invest in faster operator recovery (small image / fast image pulls, short liveness probe backoff, favorable scheduling or `priorityClass`) rather than in a multi-replica control plane. ## What the operator is responsible for For context, here is every action that *only* the operator performs. Nothing in this list happens during operator downtime. - Polling each site's MySQL instance and debouncing `writable` / `read-only` / `unreachable` transitions (see [state machine](https://bloodraven.dev/docs/operations/failover#state-machine)). - Deciding whether a cross-site combination warrants a failover, an alert, or no action. - Running the [failover sequence](https://bloodraven.dev/docs/operations/failover#failover-sequence): promote the replica, flip the `-primary` Service selector, update the `DNSEndpoint` record, taint the losing site's nodes. - Reconciling Pods, PVCs, Services, and the two MySQL Deployments from the `MysqlFailoverGroup` spec. - Reconciling backup `CronJob`s, creating `MysqlBackup` Jobs, and running retention cleanup. - Writing `status.*` fields on the CR (active site, conditions, PITR coverage, divergent GTIDs, etc.). - Answering the sidecars' `GET /active-site` probe, which is what the sidecar uses to decide whether to self-fence on boot. ## What happens when the operator is down ### Healthy steady state Nothing visible to applications. MySQL keeps serving reads and writes on whatever site is active. Replication keeps flowing. Sidecar health probes keep succeeding. The only observable signal is that the operator pod is down (liveness probe / Prometheus `up{job="bloodraven"}`), and `status.*` fields on `MysqlFailoverGroup` objects stop updating. ### Rolling restart / image upgrade A normal pod restart is ≈ 5–10 seconds. Within that window: - No MySQL state changes are observed. - Sidecars count down `leaseTimeout` (default `20s`). The restart finishes well before that, and on the next periodic fencing check (`peerCheckInterval`, default `5s`) the sidecar can reach the operator's `/healthz` endpoint again, so the self-fence timer is refreshed before it expires. - In-flight `RecloneRequested` / `RecloneRejected` decisions are idempotent: the annotation is either still present (will be handled on the next sync) or already cleared. No special procedure is required for operator image upgrades. ### Operator-down + primary failure This is the case the wishlist asked us to document explicitly. **Timeline.** Assume the operator has been down for longer than `leaseTimeout` (20 s) when the active primary crashes or is partitioned. | T | Event | Observable state | | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 0 | Operator pod crashes / is being upgraded. Liveness fails. | `-primary` Service still points at the active primary; replica is healthy. | | ≤ 20 s | Sidecars on both sites notice the operator is unreachable. Each one starts its `leaseTimeout` timer. | Nothing user-visible. Writes still work. | | tp | The primary pod goes down for any reason (node crash, OOM, partition, manual kill). | Apps connecting to `-primary` fail: first `ECONNREFUSED` once the pod is gone, then DNS name resolves but the Service has no endpoints. Reads from `-replicas` continue uninterrupted. | | tp + 20 s | If the primary pod is still alive but isolated (network partition), its own sidecar observes that both the operator *and* the peer replica are unreachable for longer than `leaseTimeout` and self-fences the primary (`SET GLOBAL super_read_only = ON`). If the primary simply crashed, there is nothing to fence — the replica stays read-only by virtue of its existing `read_only=1` (the replica's fencing monitor is a no-op on a read-only instance). Either way, no writable node exists. | Reads still work. Writes still fail. **No split brain is possible.** | | …operator still down… | Nothing happens. There is no component elected or authorized to promote the replica. | Writes remain unavailable. Operator's CR status is stale (no new polls have been recorded). | | tr | Operator pod is rescheduled and starts leading. | Operator begins polling both sites. | | tr + `failureThreshold × pollInterval` ≈ 6 s | Operator declares the dead primary `unreachable` and the live replica `read-only`. | Debounced state visible in `status.sites[]`. | | tr + ≈ 6 s + [relay drain](https://bloodraven.dev/docs/operations/failover#failover-sequence) (up to 30 s) | Operator promotes the replica, records the promotion, and best-effort flips the DNSEndpoint. | `FailoverExecuted` Event. MySQL is writable on the target; Service-label and taint convergence is still poll-driven. | | tr + ≈ 37 s | MySQL promotion completes; Service-label and taint convergence follows successful topology polls. | Apps using the new site's direct endpoint or updated DNS can reconnect first; the `-primary` Service follows after the candidate is debounced to `writable`. Transactions committed on the dead primary but never received by the replica are outside normal asynchronous-replication RPO. | These post-restart numbers use the base cadence because a newly started topology manager has no retained in-memory failure count. During normal operation, a manager that has already observed a sustained site outage backs its group poll loop off to as much as 30 seconds. A second site failure in that state can take up to about 90 seconds to collect the default three failed polls; do not apply the 6-second first-fault estimate to that compound case. **Application perspective.** From the application side, an operator-down + primary-failure outage looks like this: 1. Writes start failing immediately when the primary becomes unreachable. Most MySQL clients surface this as a connection error or a `read_only` error if the pod comes back briefly before being fenced. 2. Reads keep succeeding throughout (the `-replicas` Service still has a healthy endpoint, as long as the replica stays up). 3. The target MySQL becomes writable once the operator finishes its first post-restart failover cycle, roughly **operator-startup time + 6 s (debounce) + up to 30 s (relay drain) ≈ 10–40 s** after the operator pod comes back. Clients using updated DNS or the target's site endpoint can reconnect then. The `-primary` Service follows after successful polls debounce the target to `writable`. If the old primary is still alive but isolated, there may also be an additional stale-primary fencing window driven by `leaseTimeout` plus the sidecar check interval; that fencing only applies to a still-writable old primary and may already have happened before the operator restarts. **Write-path unavailability is the cost of not running a multi-replica control plane.** Correctness — no split brain, no silent divergence, no applying-the-wrong-writes-to-the-wrong-site — is preserved by the sidecar fencing layer regardless of how long the operator is gone. ### Operator-down + replica failure Symmetric case, and less dramatic: writes continue on the active primary; reads start failing if the replica was serving them via `-replicas`. When the operator returns, the replica is detected as `unreachable`, no failover is triggered (nothing to promote to), and the Service selector narrows to exclude the down replica until it recovers. ### Operator-down + partial partition (stale-primary scenario) This is the subtle case that WISHLIST #4 called out. A primary has already been failed over — the operator promoted the peer after the original primary lost connectivity — and the original primary now comes back into a state where it can reach its peer sidecar but still cannot reach the operator. Under a naive "fence only when operator AND every peer are silent" rule, this stale primary never fences: its peer is reachable, so the lease counter is refreshed every tick. Two MySQL instances could both report `read_only=OFF` until the operator's link comes back and it force-fences the wrong one. Bloodraven closes this by having the sidecar poll authoritative topology on every fencing tick, not only at boot: 1. The sidecar `GET`s the operator's `/active-site?namespace=&group=` and caches the reply (`activeSite`, `observedAt`). The cache is also seeded by the boot-time safety net. 2. It `GET`s each peer's `/peer/active-site`. If a peer returns a view with a strictly newer `observedAt`, the sidecar adopts it — this is how a sidecar that has lost its own operator link can still learn that a failover has happened. 3. Before running the lease-expiry rule, the sidecar compares the cached authoritative `activeSite` against its own site. If they differ and MySQL is still writable, it fences immediately: `SET GLOBAL super_read_only=ON` and kill open app connections. The sequence for the scenario above: ```mermaid sequenceDiagram participant App participant SiteA as Site A (stale primary) participant SiteB as Site B (new primary) participant Op as Operator Note over Op,SiteB: Earlier: operator failed over A→B
after A lost connectivity. Op->>SiteB: /active-site → {activeSite: B, observedAt: t0} SiteB-->>Op: cached Note over SiteA: Site A comes back. Operator
still unreachable to A. SiteA-)Op: /active-site (fails) SiteA-)SiteB: /peer/ping → 200 SiteA->>SiteB: /peer/active-site SiteB-->>SiteA: {activeSite: B, observedAt: t0} Note over SiteA: adopt: observedAt(t0) > own(stale) SiteA->>SiteA: cached.activeSite = B ≠ mySite(A)
SET super_read_only=ON App-xSiteA: writes now fail (super_read_only) ``` The stale primary fences within one fencing tick (`peerCheckInterval`, default 5 s) of coming back — it does not wait for `leaseTimeout`, and it does not depend on the operator being reachable. The only prerequisite is that at least one peer has ever successfully observed the authoritative `/active-site`. The boot-time safety net guarantees this for any peer that has been up since the last failover. Rolling upgrade note: a peer running an older sidecar that does not yet implement `/peer/active-site` returns 404; the adopting sidecar silently falls back to the operator-only path. No lease behavior changes in the mixed-version window. ### Operator-down + both-site outage Rare, but by construction, nothing can recover this automatically — neither the sidecars (both peers unreachable and self-fenced) nor the operator (down). Human intervention is required. See [Total loss recovery](https://bloodraven.dev/docs/operations/operations#total-loss-recovery). ## Recommended mitigations These are things you can do to minimize the blast radius of operator downtime without adopting a multi-replica control plane: - **Keep the operator pod up.** Use a tight liveness probe, a readiness probe that only reports ready once the manager is healthy, and immutable image tags (or an explicit pull policy) to avoid rollout-time pull failures. Rely on Kubernetes to restart or reschedule the pod promptly; a healthy node restart is ≈ 5 s, well inside `leaseTimeout`. - **Size the PodDisruptionBudget so the operator isn't evicted during routine node drains.** The shipped Helm chart does not set a PDB on the operator; add one if your cluster does aggressive scheduling. - **Monitor `up{job="bloodraven"}` and `kube_pod_status_phase` for the operator namespace.** Page on operator down, not just on MySQL alerts — a quiet operator during a MySQL failure is the worst of both worlds. - **Rehearse the outage.** Use the playground's scenarios 2 (operator kill + restart while healthy) and 5 (operator kill mid-failover) to confirm the timing in your own environment. - **Never run the operator image from a floating tag in production.** A failed image pull during a routine Deployment rollout turns into the case documented above. ## FAQ **Can I run two operator replicas?** You can, and leader election will keep exactly one active. This protects against a split-brain scenario during a rolling Deployment update (the old pod is stopped before the new pod acquires the lease, so only one side writes status at a time). It does **not** meaningfully reduce failover MTTR — by the time the new leader is elected and starts polling, a single-replica Deployment would have restarted too. **Does operator downtime cause data loss?** No. Committed writes stay on whichever site accepted them. The sidecar self-fence prevents any post-primary-crash writes from being accepted on a partitioned-away primary, so you never have two versions of "the latest state" to reconcile. Async replication's normal RPO applies to the crashed-primary case — see [Durability and RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo) for the full contract, or [Old primary recovery](https://bloodraven.dev/docs/operations/failover#old-primary-recovery) for the operator's post-failover reconciliation. **Does operator downtime cause split brain?** No, by design. Two rules cooperate. (1) Sidecars poll the operator's `/active-site` every tick and also read each peer's `/peer/active-site`; if the authoritative active site disagrees with this site and MySQL is still writable, the sidecar fences immediately. (2) As a backstop, any MySQL instance that can reach neither its peer nor the operator beyond `leaseTimeout` self-fences. Rule (1) is what prevents a returning stale primary from continuing to accept writes even when its peer is reachable — see [Operator-down + partial partition](https://bloodraven.dev/#operator-down--partial-partition-stale-primary-scenario). **Why not scale the operator horizontally?** Because the failure-detection loop is the bottleneck, not the operator process. Polling starts at the `pollInterval` base (default 2 seconds) with `failureThreshold` debounce (default 3), then backs off to a 30-second cap after a confirmed persistent failure. Adding a second replica cannot shorten that loop — both replicas would see the same polls — and leader election adds its own election latency when the lease expires. The current design keeps the writable path on one goroutine with observable state, which is far easier to reason about than a coordinator-of-coordinators. # Durability and RPO ![durability and rpo infographic](https://bloodraven.dev/img/infographics/durability-and-rpo.webp) Bloodraven drives **asynchronous** primary→replica replication. The primary commits locally and sends binlog events to the replica; it does not wait for the replica to persist those events before returning success to the application. That single design choice is what gives Bloodraven its sub-second write latency and its ability to keep serving writes when the replica is down — and it's also the sole reason the RPO of an emergency failover is **not zero**. This page makes the RPO contract explicit so you don't have to infer it from source code. ## The contract in one line > An emergency failover can lose every transaction that committed on > the dying primary but had not yet replicated to the surviving site. Everything below is the detail behind that sentence — the theoretical bound on how much can be lost, what the operator surfaces so you can audit it after the fact, and how Point-in-Time Recovery (PITR) interacts with the bound. ## What gets persisted where Bloodraven configures each MySQL pod with: | Setting | Value | What it means for durability | | -------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sync_binlog` | `1` | Every committed transaction is `fsync()`ed to the primary's binary log before returning success to the client. A kernel panic on the primary does **not** lose acknowledged transactions from the primary's own disk. | | `innodb_flush_log_at_trx_commit` | `2` | InnoDB redo log is written to the OS on every commit but flushed to disk only once per second. On a power loss (not just a process crash), up to \~1 s of redo can be lost — but the binlog is still authoritative, so replay from a backup + binlog is unaffected. | | `gtid_mode` | `ON` | Every committed transaction has a globally-unique GTID. This is what lets `status.sites[].divergentGtid` be a *set* rather than a timestamp — you get a precise count of lost transactions, not an estimate. | | `log_replica_updates` | `ON` | Replica writes its replicated transactions back into its own binlog so it can itself become a primary after failover without re-replaying. | | `max_binlog_size` | `100M` (configurable via `spec.backup.pitr.maxBinlogSize`) | Drives binlog **rotation cadence**, which is what PITR archival is bounded by. See [PITR and RPO](https://bloodraven.dev/#pitr-and-the-rpo-window) below. | **What this means for the primary-standalone case.** If the primary crashes but its PVC survives, MySQL restarts, the binlog tail is rotated and sealed, and replication to the replica catches up. No transactions are lost. The operator doesn't even need to fail over — it observes the primary return to `writable` and keeps it as primary. **What this means for async replication.** `sync_binlog=1` makes the *primary's* copy durable. It does not make the *replica's* copy durable. Between the moment the primary commits and the moment the replica's IO thread acknowledges receipt of those events, the bytes exist only on the primary. If the primary (and its PVC) vanish in that window, those transactions are unrecoverable from the replica. ## The lag signal: `seconds_behind_source` Replication lag near the moment of failure is a useful RPO estimate, not a strict bound. It is surfaced three ways: - `status.sites[].secondsBehindSource` on the `MysqlFailoverGroup` CR (populated for the replica site on every poll). - The `bloodraven_replication_lag_seconds` Prometheus gauge. - `SHOW REPLICA STATUS` column `Seconds_Behind_Source` on the replica. **Typical value under healthy operation: < 1 second.** Bloodraven's default base poll interval is 2 seconds (configurable with `spec.pollInterval`), and it asserts replication health on every poll. A replica that lags past `spec.replication.maxLagSeconds` (default `300` s) transitions the CR's `Degraded` condition to `True` with reason `ReplicationLagging`. Under a healthy topology, the last status sample is normally no more than one base interval plus probe time old. If the replica reported `secondsBehindSource=0` at `T−2s` and the primary dies at `T`, that observation is evidence of recent catch-up, but it does not prove that every commit after the sample reached the replica. MySQL's lag value can also be zero while a commit is still between the primary and the replica IO thread. After a site reaches `failureThreshold` and continues failing, Bloodraven backs the shared group poll loop off exponentially to a 30-second cap. During that outage, lag status for the surviving sites can therefore be up to one effective interval old, and a second fault can take up to about 90 seconds to reach the default three-poll failure threshold. This affects observation freshness and compound-fault recovery time; it does not itself change which transactions were replicated. See [Failover: Debouncing](https://bloodraven.dev/docs/operations/failover#state-transitions). ### Relay-log drain narrows the window further Step 3 of the [failover sequence](https://bloodraven.dev/docs/operations/failover#failover-sequence) is a bounded-wait relay-log drain on the candidate: before the candidate is promoted, the operator gives it up to 30 s to apply any relay-log events the IO thread had already fetched but the SQL thread hadn't processed. Transactions that made it *across the wire* to the replica before the primary died are therefore not lost even if `SQL_THREAD` was momentarily behind. The RPO window is specifically "transactions the primary committed but the replica's IO thread never received" — tighter than raw `seconds_behind_source` implies. ## The exact bound: `divergentGtid` `seconds_behind_source` is an estimate; GTIDs are authoritative. After an emergency failover Bloodraven computes the exact set of transactions that existed on the old primary but never made it to the new primary, and writes the result into two CR status fields: - `status.sites[].divergentGtid` — the GTID **set** of lost transactions, in MySQL's compact range notation (e.g. `a1b2c3d4-…:11-15`). - `status.sites[].divergentTransactionCount` — a simple count so you can alert on it without parsing the set. The same value drives the `RecoveryPending` condition with reason `DivergentTransactions` and the `bloodraven_divergent_transactions` Prometheus gauge. Zero means "the replica had everything at the moment the primary was fenced" — an RPO of 0 in this specific failover. Non-zero means "this many transactions are permanently lost from the replication stream unless you recover them out-of-band from the old primary's binlog." The report stays current: while recovery is blocked the operator re-verifies the divergence on the recovery retry cadence (\~30s), so a site that diverges further before you act (e.g. it respawned writable for a moment and was re-fenced) has its `divergentGtid` refreshed rather than under-reported, and divergence you resolve out-of-band (replaying the set onto the new primary) unblocks the automatic rejoin. Bloodraven also records `status.promotionGtidExecuted` — the GTID set on the new primary **at the moment of promotion**, before it began accepting writes. Together with `divergentGtid` this gives you a post-hoc audit of every failover: the promoted site started from `promotionGtidExecuted`, and any GTIDs the old primary had beyond that are enumerated in `divergentGtid`. ## Per-failure-mode RPO matrix | Failure mode | Replica state at failure | RPO | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Pod crash, **PVC survives**, MySQL restarts in place | Any | **0** — operator sees the primary return `writable` before debounce fires; no failover happens. | | Pod crash, **PVC lost**, operator triggers failover | Caught up (lag ≈ 0) | **0** — the exact moment-of-failure GTID lives on the replica; no transactions are unaccounted for. | | Pod crash, **PVC lost**, operator triggers failover | Lagging (`secondsBehindSource` > 0) | **≤ lag value in seconds worth of transactions.** Lost set is reported in `divergentGtid`. | | Cross-site partition, operator promotes the reachable side | Depends on lag at moment of partition | **≤ lag at partition start.** `divergentGtid` surfaces the exact set once both sites are observable again. | | Both sites unreachable (operator sees `TotalLoss`) | n/a | **No failover occurs.** RPO accumulates on the primary until at least one site comes back. See [Operator availability](https://bloodraven.dev/docs/architecture/operator-availability#operator-down--both-site-outage). | | Operator is down while the primary crashes | Depends on lag at crash | **Same as the equivalent PVC-lost case** — sidecars fence themselves, the replica is read-only, RPO is bounded by the lag at crash. Write-*availability* is also lost until the operator returns (see [Operator availability](https://bloodraven.dev/docs/architecture/operator-availability#operator-down--primary-failure)), but RPO is not made worse by the operator being down. | | Split-brain (both sites `writable`) | Both have been accepting conflicting writes | RPO is undefined until a human picks an authoritative side; the *losing* side's divergent GTIDs are lost when that side is fenced and recloned. See [Split-brain recovery](https://bloodraven.dev/docs/operations/operations#split-brain-recovery). | | [Planned failover](https://bloodraven.dev/docs/operations/planned-failover) (admin annotation) | Target caught up to fenced source GTID before promotion | **0 — by construction.** The zero-lag gate at `WaitingForLag` only advances once the target's `GTID_EXECUTED` ⊇ the source's fenced `GTID_EXECUTED`. Any scenario that would produce loss (lag never closes, source crashes mid-drain) routes to a `Failed` rollback instead of promoting a lagging replica. | ## PITR and the RPO window Point-in-Time Recovery, when enabled (`spec.backup.pitr.enabled=true`), archives each **sealed** binlog file to S3 or a shared PVC as the primary rotates. The archived binlogs are what a restore replays on top of a full dump to reach an arbitrary `stopDatetime`. See [Point-in-time recovery](https://bloodraven.dev/docs/backup-and-restore/backup-restore#point-in-time-recovery-pitr) for the end-to-end story. PITR **narrows** the worst-case RPO — but only inside a specific failure-mode envelope. The nuance: - **Archival is bounded by rotation cadence.** The archiver uploads every binlog *except* the currently-active one (the one MySQL is still writing to). Rotation happens on size (`max_binlog_size`, default `100M`), on `FLUSH BINARY LOGS`, or on restart. So at any moment on a healthy primary, the unarchived tail is at most one binlog file's worth of writes. - **Pod crash with PVC survival → PITR recovers the tail.** When MySQL restarts, it rotates the binlog, seals the previously-active one, and the archiver picks it up on the next scan. The previously-unarchived tail is now in storage. A PITR restore can replay up to the last committed transaction. - **PVC loss → PITR does not recover the tail.** The previously-active binlog lived on the destroyed PVC. It is gone forever, along with any transactions it contained. This is the case the wishlist flagged: "PITR narrows RPO to `max_binlog_size` rotation cadence **only if the primary's binlog tail survives**" — on PVC loss it doesn't. - **PITR doesn't recover what async replication already lost.** If you restore *from the surviving replica's dump + archived binlogs*, you are bounded by what the replica ever received. Transactions the old primary committed but never shipped are not in the replica's binlog stream and therefore not in PITR's replay material. Even a successful PITR restore cannot reach back beyond the async-replication cutoff. **Pragmatic takeaway.** PITR is a disaster-recovery tool for *rolling back time* (restore to 14:32 before the bad migration) or for recovering from backup-artifact loss. It is not a mechanism for bringing the async-replication RPO below zero seconds. For that you would need synchronous replication, which is not in Bloodraven's design. **Cross-cluster recovery.** PITR also powers the cross-cluster DR path: when an entire cluster is lost, the DR cluster can bootstrap a new `MysqlFailoverGroup` from the source's S3 dump archive with an optional `pointInTime` replay window. The unarchived binlog tail on the lost PVC is still unrecoverable, but the RPO is bounded to that tail — not the entire last-backup interval. See [Multi-cluster DR](https://bloodraven.dev/docs/architecture/multi-cluster-dr) for the end-to-end runbook. ## Tuning: what you control The relevant knobs for narrowing the async-replication RPO window, in order of impact: 1. **Keep `maxLagSeconds` tight.** The shipped default of `300` s is generous to accommodate catch-up after network blips; if your SLO is stricter, lower it. A tighter value means the `Degraded` condition fires sooner, so a human or an automation notices a lagging replica before the primary dies. :caution[`maxLagSeconds` is an **alerting threshold**, not a promotion gate. If the primary dies while the replica is beyond the threshold, Bloodraven still promotes the replica — the alternative of "no writable site at all" is almost always worse than a high-RPO failover. Treat a sustained `Degraded=True` with reason `ReplicationLagging` as a page.] 2. **Monitor `bloodraven_replication_lag_seconds{role=~"primary-candidate|dr-only"}` and alert on sustained non-zero values.** Sub-second lag is your *normal*; any drift upward is an early warning that the RPO on a future primary failure is growing. Do not page on `role="read-only"` reader lag — that is designed, and the reader is shed from the replicas Service. 3. **Tune `spec.backup.pitr.maxBinlogSize` if you enable PITR and care about the non-PVC-loss case.** Smaller rotation = smaller unarchived tail = tighter PITR RPO in the narrow case where the primary's PVC survives but you still want to do a point-in-time replay. The default `100M` is a reasonable balance; going below `10M` costs you archival overhead without a meaningful RPO benefit on most workloads. 4. **Use PVC storage that survives pod eviction.** The most common way to turn a pod-crash RPO-of-0 outage into a PVC-loss RPO-of-lag outage is storage that doesn't survive a node failure. Use a network-attached PV (EBS, Persistent Disk, Ceph RBD) rather than `hostPath` or `emptyDir`, and confirm your `StorageClass` has `reclaimPolicy: Retain` for the data PVC. What you **don't** control from Bloodraven: application-side write patterns. If your application is willing to accept "probably committed" on a non-fsynced write path, no amount of operator durability config can fix that. `sync_binlog=1` (what Bloodraven configures) is the strongest setting MySQL offers for primary-side durability. ## Related reading - [Failover → Old primary recovery](https://bloodraven.dev/docs/operations/failover#old-primary-recovery) — how the operator surfaces `divergentGtid` and how an admin runs the reclone interlock after an RPO-non-zero failover. - [Operator availability → Operator-down + primary failure](https://bloodraven.dev/docs/architecture/operator-availability#operator-down--primary-failure) — why operator downtime doesn't make RPO worse, but does make write-*availability* worse. - [Backup and restore → Point-in-time recovery](https://bloodraven.dev/docs/backup-and-restore/backup-restore#point-in-time-recovery-pitr) — the restore-side of the PITR story (the archiver side lives in [Architecture → Sidecar](https://bloodraven.dev/docs/architecture/architecture#sidecar-bloodraven-sidecar)). # Architecture Diagrams ![architecture diagrams infographic](https://bloodraven.dev/img/infographics/architecture-diagrams.webp) These diagrams are source-controlled as Mermaid so they can be edited with docs changes. ## Normal two-site topology ```mermaid flowchart LR App[Application Pods] --> DNS[orders.az.example.com] DNS --> PrimarySvc[Primary Service] PrimarySvc --> IAD[(MySQL iad primary)] PDX[(MySQL pdx replica)] --> IAD App --> DragonflySvc[orders-dragonfly Service] DragonflySvc --> DFIAD[(Dragonfly iad master)] DFPDX[(Dragonfly pdx replica)] --> DFIAD Operator[Bloodraven Operator] --> IAD Operator --> PDX Operator --> DFIAD Operator --> DFPDX Operator --> DNSEndpoint[DNSEndpoint] DNSEndpoint --> ExternalDNS[external-dns] Prom[Prometheus] --> Metrics[Operator metrics] ``` ## Emergency failover path ```mermaid sequenceDiagram participant O as Operator participant A as iad primary participant B as pdx replica participant DA as Dragonfly iad master participant DB as Dragonfly pdx replica participant D as DNS participant K as Kubernetes O->>A: poll health A--xO: unreachable O->>B: check replication and read-only state O->>B: promote to writable O->>DB: best-effort REPLTAKEOVER O->>DB: fallback REPLICAOF NO ONE if needed O->>D: update DNSEndpoint to pdx LB IP O->>K: taint old-site nodes O->>B: verify writable ``` ## Planned failover path ```mermaid sequenceDiagram participant Admin participant O as Operator participant A as Current primary participant B as Target replica participant DA as Current Dragonfly master participant DB as Target Dragonfly replica Admin->>O: annotate planned-failover=pdx O->>B: wait for lag to drain O->>A: fence writes O->>DB: wait for Dragonfly sync O->>DB: REPLTAKEOVER O->>B: promote O->>O: update status.activeSite ``` ## Dragonfly active Service ```mermaid flowchart LR App[Redis-compatible clients] --> ActiveSvc[orders-dragonfly Service] ActiveSvc --> Selector{role=master AND traffic=enabled} Selector --> Active[(Dragonfly active site)] Replica[(Dragonfly replica site)] --> Active Operator[Bloodraven Operator] --> Active Operator --> Replica Operator --> Labels[role/traffic labels] Labels --> Selector ``` ## Sidecar self-fencing ```mermaid flowchart TD S[Sidecar] --> O{Operator reachable?} O -- yes --> Keep[Keep current MySQL mode] O -- no --> P{Peer reachable?} P -- yes --> Keep P -- no --> Fence[Set super_read_only=ON] ``` ## DNS steering ```mermaid flowchart LR Operator --> DNSEndpoint DNSEndpoint --> ExternalDNS ExternalDNS --> Provider[DNS Provider] Provider --> ClientCache[Client resolver cache] ClientCache --> Apps ``` ## Backup and restore flow ```mermaid flowchart TD Cron[CronJob or MysqlBackup] --> Job[Backup Job] Job --> Source[(Replica preferred, primary fallback)] Job --> Store[(S3 or PVC)] Store --> Verify[MysqlBackupVerification] Store --> Restore[initFromBackup restore Job] Restore --> NewGroup[(Recovered failover group)] ``` ## PITR binlog archival ```mermaid flowchart LR MySQL[(MySQL binlogs)] --> Sidecar[Sidecar archiver] Sidecar --> Store[(Backup profile storage)] Store --> Restore[Restore Job] Restore --> Replay[mysqlbinlog replay to target time] ``` ## Monitoring and alerting flow ```mermaid flowchart LR Operator --> Metrics[Metrics endpoint] Operator --> Logs[Structured logs] Operator --> Events[Kubernetes Events] Metrics --> Prometheus Prometheus --> Alerts[Alerts] Prometheus --> Grafana Alerts --> Runbooks[Runbooks] ``` # Operations Overview ![operations overview infographic](https://bloodraven.dev/img/infographics/operations-overview.webp) Use this page during operational work to choose the safest runbook quickly. ## Decision table | Situation | Use | Do not start with | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | | Planned maintenance on active site | [Planned Failover](https://bloodraven.dev/docs/operations/planned-failover) | Emergency promotion | | Active site is unreachable | [Failover](https://bloodraven.dev/docs/operations/failover), then [Runbooks](https://bloodraven.dev/docs/operations/runbooks) | Reclone before confirming the new primary | | Both sites appear writable | [Network Partitions](https://bloodraven.dev/docs/operations/network-partitions) and [Runbooks](https://bloodraven.dev/docs/operations/runbooks#split-brain-recovery) | App restarts only | | Old primary has divergent GTIDs | [Runbooks](https://bloodraven.dev/docs/operations/runbooks#divergent-old-primary-recovery) | Auto-rejoin | | Backup failed | [Troubleshooting](https://bloodraven.dev/docs/operations/troubleshooting#backup-job-failing) | Delete all old backups | | Restore failed | [Troubleshooting](https://bloodraven.dev/docs/operations/troubleshooting#restore-job-failing) | Retrying without checking source and credentials | | Operator unavailable | [Operator Availability](https://bloodraven.dev/docs/architecture/operator-availability) | Disabling sidecar fencing | | DNS not moving | [Troubleshooting](https://bloodraven.dev/docs/operations/troubleshooting#dns-not-changing-after-failover) | Manual DB promotion | | Dragonfly degraded | [Runbooks](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded) and [Monitoring](https://bloodraven.dev/docs/observability/monitoring#available-metrics) | MySQL emergency promotion if MySQL is healthy | ## On-call path 1. Check the alert in [Alert to Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map). 2. Check `MysqlFailoverGroup` status and Kubernetes Events. 3. Run the matching runbook header checklist before taking action. 4. Verify active site, DNS, replication, application writes, and Dragonfly status when enabled after remediation. ```bash kubectl get mysqlfailovergroup orders -n orders -o wide kubectl describe mysqlfailovergroup orders -n orders kubectl get events -n orders --sort-by=.lastTimestamp ``` ## Related references - [Operations](https://bloodraven.dev/docs/operations/operations) - [Runbooks](https://bloodraven.dev/docs/operations/runbooks) - [Failure Mode Matrix](https://bloodraven.dev/docs/operations/failure-mode-matrix) - [Log Schema](https://bloodraven.dev/docs/observability/log-schema) - [Monitoring](https://bloodraven.dev/docs/observability/monitoring) ## Test strategy and Operator SDK Scorecard Bloodraven's test pyramid is unit tests (`internal/**/*_test.go`), cross-package component tests with fakes (`test/component/`), integration tests behind the Go `integration` build tag (run with `make test-integration`), envtest controller tests against a real API server (`test/envtest/`), and a Go-based chaos runner against a live k3d cluster (`cmd/playground-chaos`, exposed via `make chaos-list`, `make chaos-run SCENARIO=`, and `make chaos-run-all`). CI runs lint, build, unit, component, envtest, generate-check, and docs-build on every PR (`.github/workflows/ci.yml`). Integration tests exist in the repo but are not part of the default PR gate. The real-cluster end-to-end gate is tracked separately as WISHLIST #32. We evaluated the [Operator SDK Scorecard](https://sdk.operatorframework.io/docs/testing-operators/scorecard/){rel=""nofollow""} as an additional tier and **declined adoption** today. Scorecard is a containerized test runner that takes an OLM bundle as input and runs tests as Pods against a Kubernetes cluster. Its built-in OLM suite (five tests) all read a `ClusterServiceVersion`; its built-in basic suite is the single `basic-check-spec-test`; custom and kuttl scorecard paths both require a bundle plus a live-cluster gate. ### Why we declined today We chose decline only after every condition below held against the repository at the time of the decision. If any becomes false, reopen WISHLIST #34 and re-run this rubric. | ID | Condition | Today | | -- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----- | | R1 | No `bundle.Dockerfile`, no `ClusterServiceVersion`, no `bundle/` directory, no `config/scorecard/` kustomize templates exist. | True | | R2 | No `PROJECT` file at the repo root (no `operator-sdk init` retrofit has been done). | True | | R3 | No active plan to publish Bloodraven to OperatorHub or any OLM-distributed catalog (WISHLIST #30 remains open and explicitly conditional P3). | True | | R4 | No external consumer of `scapiv1alpha3.TestStatus` JSON in the Bloodraven release pipeline, sibling repos, or platform tooling. | True | | R5 | The existing pyramid (unit + component + envtest + the `cmd/playground-chaos` runner) already covers the failure modes Scorecard's basic and OLM suites would surface against a CSV-less, non-OLM operator, and emits richer forensic output (events, logs, raw `/metrics`) than `scapiv1alpha3.TestStatus`. | True | The basic `basic-check-spec-test` payoff is trivial: it would not surface a real defect against any custom resource sample shipped under `examples/`. The OLM tests cannot run because we ship no CSV. The custom and kuttl paths require both a bundle stub and a live-cluster gate (WISHLIST #32), and the resulting harness duplicates work the existing Go-based suites already do with richer output. ### When to reopen WISHLIST #34 Reopen if any of the following becomes true: - **T1.** Bloodraven publishes or commits to publish an OLM bundle — `bundle.Dockerfile`, CSV, or `config/scorecard/` kustomize templates land in the repo. (Falsifies R1.) - **T2.** A `PROJECT` file is introduced or `operator-sdk init` is run on the repository. (Falsifies R2.) - **T3.** Bloodraven adopts an external-distribution path that lists it on OperatorHub or an equivalent OLM catalog (e.g. WISHLIST #30 closes with that path chosen). (Falsifies R3.) - **T4.** A downstream tool (CI, `platform/`, sibling repo, certification flow) starts consuming `scapiv1alpha3.TestStatus` JSON. (Falsifies R4.) - **T5.** The real-cluster E2E gate (WISHLIST #32) ships and there is a documented argument that wrapping a subset of its assertions as scorecard custom tests is cheaper than maintaining them in Go. (R5 becomes worth re-evaluating.) - **T6.** The Operator SDK project ships a meaningful basic-suite expansion (beyond `basic-check-spec-test`) that delivers signal a non-OLM operator could consume without a CSV. When any trigger fires, the reopener should (a) flip the relevant rubric row in this table to False with a one-line citation, (b) reopen #34 in `WISHLIST.md`, and (c) cite this section as the prior art. # Upgrade and version-skew policy What MySQL and sidecar versions Bloodraven supports, how to move between them, and what mismatches the operator tolerates during a rollout. This page is deliberately narrow. It covers the MySQL image (`spec.image`) and the sidecar image (`spec.sidecarImage`). Operator (Helm chart) upgrades and CRD-version migrations are separate topics. ## Supported MySQL versions Bloodraven supports a single steady-state MySQL baseline at a time, pinned to the default `spec.image` tag: - **Current baseline**: `mysql:9.7`. This is the default for new deployments and the version exercised in CI, the playground, and backup / restore paths. The following are **not supported**: - MySQL 8.0 and 8.4. If you run these today, upgrade MySQL to 9.7 outside of Bloodraven before adopting the operator, or bootstrap a fresh failover group from a `mysqlsh`-produced dump using `initFromBackup`. - MySQL 9.8 and later Innovation releases. Bloodraven does not track Innovation releases unless and until the project explicitly adopts a later LTS baseline. - MySQL 5.7 and earlier (no GTID, no clone plugin). ::warning **Pin, don't float** Always pin `spec.image` to an immutable tag (e.g. `mysql:9.7`, not `mysql:9`). A floating tag can drift you onto an unsupported version between pod restarts. See the [Production hardening checklist](https://bloodraven.dev/docs/configuration/production-hardening#image-supply-chain). :: ### Implementation prerequisites These are satisfied by the supported tags and only matter if you point Bloodraven at something else: GTID-only replication (`SOURCE_AUTO_POSITION=1`), the MySQL clone plugin (`CLONE INSTANCE FROM`), and the `REPLICA`/`SOURCE` SQL vocabulary. The operator does not validate these at startup — an unsupported image surfaces as MySQL pod failures, not an operator admission error. ## Rolling image upgrades To roll a MySQL image change (e.g. patch-version bump): 1. Confirm `spec.updateStrategy: OrderedUpdate` is set (it is the default). 2. Bump `spec.image` to the new pinned tag. 3. `kubectl apply`. 4. Watch `status.updatePhase` and the `Updating` / `Bootstrapping` conditions on `status.conditions`, and confirm replication catches up before the second site rolls. 5. Run a backup-verification drill (see [Backup verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification)) against the new version. With `OrderedUpdate`, Bloodraven upgrades the **standby first**, then fails over to it, then upgrades the old active. This is the direction MySQL's rolling-upgrade contract requires: the replica's MySQL version must be ≥ the primary's during the rollout. See [Failover → Ordered updates](https://bloodraven.dev/docs/operations/failover#ordered-updates) for the full sequence. ::caution **Do not use `updateStrategy: Recreate` for a major bump** `Recreate` restarts both sites simultaneously. If one pod's in-place MySQL upgrade stalls or fails, you lose write availability with no healthy primary and no rollback path. Use `OrderedUpdate` for any MySQL image change. :: For MySQL-level upgrade semantics — data-dictionary migrations, deprecated-feature removals, schema changes — defer to Oracle's own release notes. Bloodraven drives the rollout; it does not duplicate the MySQL upgrade matrix. ## Cross-site version skew **Transient (during a rollout): allowed.** The `OrderedUpdate` sequence keeps the newer MySQL on the replica side of replication throughout the bump, which is MySQL-supported. The skew window lasts from the moment the standby finishes upgrading until the old active has been restarted on the new image — roughly pod-ready time plus the failover drain. No user action is required. **Steady-state (intentional per-site version split): not supported.**`spec.image` is a single field per `MysqlFailoverGroup`; `SiteSpec` has no per-site image override. The operator reconciles both site Deployments to the same image tag. If you edit a site's Deployment manifest out of band to pin a different tag, the next reconcile reverts it. ## Sidecar ↔ operator skew The sidecar binary and the operator binary are released together. The supported configuration is matching versions: `spec.sidecarImage` built from the same release as the running operator Deployment. Bloodraven tolerates a **one-minor** transient skew in either direction during a rolling upgrade. Both sides of the HTTP surface between the two are additive-only: - **Operator endpoints** (polled by sidecars): `/healthz`, `/status`, `/active-site`, `/pitr-cutoff`. - **Sidecar endpoints** (polled by the operator and peer sidecars): `/health`, `/status`, `/peer/ping`, `/peer/active-site`, `/archiver/status`. A briefly older-or-newer sidecar stays compatible with the operator while pods roll. Skew beyond one minor is not tested and not guaranteed. When a release bumps both `spec.image` and `spec.sidecarImage` together, `OrderedUpdate` restarts the sidecar as part of the pod restart it already performs — the replica-first ordering applies automatically, no separate sidecar sequencing is required. ## Restoring across versions `MysqlBackup.status.mysqlImage` records the image tag that produced each dump. The safe path is to restore onto the same major version that produced the backup. **Do not** restore a dump onto an older MySQL version — MySQL does not support version downgrade of data. Backup-restore throughput and sizing guidance live under a separate performance guide (WISHLIST #27). ## What Bloodraven does not do - **No version admission check.** Setting `spec.image` to an unsupported tag will not produce a CRD validation error. MySQL itself rejects unsupported upgrades at pod start. - **No skip-a-major guardrail.** There is no enforcement that blocks an `8.0 → 9.7` jump or a `9.7 → 9.8+` jump at the operator layer. - **No downgrade path.** MySQL does not support version downgrade, and Bloodraven inherits that limitation. - **No per-site MySQL version.** The CRD does not expose a per-site image, so mixed-version steady state is not expressible. # GitOps guidance ![gitops infographic](https://bloodraven.dev/img/infographics/gitops.webp) Use this page for Argo CD or Flux-managed Bloodraven installs. It covers resource ownership, sync order, custom health checks, and upgrade sequencing. ## CRD ownership Pick one CRD owner: | Owner | Notes | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bloodraven Helm release | Simple first install because Helm applies files in `charts/bloodraven/crds/`; CRD upgrades still need explicit review/application because Helm does not upgrade CRDs automatically. | | Platform CRD app | Better for centralized CRD review and ordering; install the operator chart after the CRD app syncs. | Do not manage the same CRDs in both places. ## Sync waves Recommended order: 1. external-dns `DNSEndpoint` CRD and controller. 2. cert-manager CRDs and issuers when TLS is enabled. 3. Bloodraven CRDs. 4. Bloodraven operator Helm release. 5. Tenant namespaces and Secrets. 6. `MysqlFailoverGroup` resources. 7. Monitoring dashboards and alert rules. ## Argo CD health checks Argo CD does not know how to evaluate `MysqlFailoverGroup` or `MysqlBackup` custom resources by default. Without custom health checks, the resources show **Unknown** in the Argo CD UI. Bloodraven ships Lua health scripts in `charts/bloodraven/argocd/` that map custom resource (CR) status to Argo CD health: | CR | Condition or field | Argo CD health | | -------------------- | ------------------------------------------------------------------------------------- | --------------- | | `MysqlFailoverGroup` | `Degraded=True`, `RecoveryPending=True`, or `Bootstrapping=True` with `reason=Failed` | **Degraded** | | `MysqlFailoverGroup` | `Bootstrapping=True` or `Updating=True` | **Progressing** | | `MysqlFailoverGroup` | `Ready=True` | **Healthy** | | `MysqlBackup` | `phase: Succeeded` | **Healthy** | | `MysqlBackup` | `phase: Failed` | **Degraded** | | `MysqlBackup` | `phase: Pending` or `Running` | **Progressing** | ### Patch argocd-cm directly Extract the chart and apply the patch file: ```bash helm pull bloodraven/bloodraven --untar kubectl patch configmap argocd-cm -n argocd \ --patch-file bloodraven/argocd/argocd-cm-patch.yaml kubectl rollout restart deployment argocd-application-controller -n argocd ``` ### Use Argo CD Helm chart values If you deploy Argo CD with its Helm chart, extract the Lua scripts from the Bloodraven chart: ```bash helm pull bloodraven/bloodraven --untar ``` Then copy the contents of `bloodraven/argocd/health-mysqlfailovergroup.lua` and `bloodraven/argocd/health-mysqlbackup.lua` into your Argo CD values: ```yaml configs: cm: resource.customizations.health.shipstream.io_MysqlFailoverGroup: | # paste contents of bloodraven/argocd/health-mysqlfailovergroup.lua resource.customizations.health.shipstream.io_MysqlBackup: | # paste contents of bloodraven/argocd/health-mysqlbackup.lua ``` ### Use a Kustomize overlay If you manage Argo CD with Kustomize, extract the patch from the chart and add it as a strategic merge patch: ```bash helm pull bloodraven/bloodraven --untar cp bloodraven/argocd/argocd-cm-patch.yaml . ``` ```yaml patchesStrategicMerge: - argocd-cm-patch.yaml ``` ### Verify Argo CD health checks Check that `argocd-cm` contains Bloodraven customizations: ```bash kubectl get configmap argocd-cm -n argocd -o yaml | grep shipstream ``` After Argo CD refreshes the application, `MysqlFailoverGroup` and `MysqlBackup` resources should show **Healthy**, **Progressing**, or **Degraded** instead of **Unknown**. ## Ignore status fields GitOps tools should not try to reconcile `.status`. ## Generated dashboards If Grafana dashboard ConfigMaps are rendered by the Bloodraven Helm release, keep them in the operator application. If your monitoring stack owns dashboards centrally, set `grafanaDashboards.enabled=false` and import `charts/bloodraven/dashboards/*.json` in that stack. ## Safe upgrade workflow 1. Read [Upgrade Policy](https://bloodraven.dev/docs/operations/upgrade-policy). 2. Apply CRD updates before controller image updates. 3. Roll the operator. 4. Watch reconciliation on a non-production failover group. 5. Roll tenant groups in maintenance windows if sidecar/MySQL image changes are involved. 6. Run `make generate && make manifests` before PRs that touch API fields. ## Related pages - [Production Install](https://bloodraven.dev/docs/get-started/install-production) - [Production Install Examples](https://bloodraven.dev/docs/get-started/production-install-examples) - [Production Hardening](https://bloodraven.dev/docs/configuration/production-hardening) # Operations ![operations infographic](https://bloodraven.dev/img/infographics/operations.webp) Day-2 operations guide for managing Bloodraven failover groups, including manual intervention, recovery procedures, and maintenance tasks. Before rolling Bloodraven out to production, also walk through the [Production hardening checklist](https://bloodraven.dev/docs/configuration/production-hardening) and keep the [Failure-mode matrix](https://bloodraven.dev/docs/operations/failure-mode-matrix) open during incidents. The [Log schema contract](https://bloodraven.dev/docs/observability/log-schema) lists the stable `msg` strings to filter on when correlating an incident in your log aggregator. ::tip **Incident entry point** Use [Operations Overview](https://bloodraven.dev/docs/operations/operations-overview), [Runbooks](https://bloodraven.dev/docs/operations/runbooks), and [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map) during active incidents. This page remains the detailed operations reference. :: ::tip **CLI shortcut** Most day-2 commands on this page have a `kubectl bloodraven ...` equivalent in the [kubectl plugin](https://bloodraven.dev/docs/operations/kubectl-plugin). The plugin validates inputs before posting, supports synchronous `--wait`, and keeps the annotation grammar honest. :: ## Operations decision table | Task | Use when | Do not use when | Verify with | | -------------------- | ------------------------------------------- | -------------------------------------- | ----------------------------------------- | | Planned failover | Healthy target and scheduled maintenance | Split-brain or target lagging | `status.plannedFailover`, DNS, app writes | | Manual promotion | Operator unavailable and old primary fenced | Operator is healthy | active site, GTIDs, DNS | | Split-brain recovery | Multiple writable sites | Only DNS is stale | writable state and GTID comparison | | Reclone old primary | Divergence accepted or no data to recover | Data owner has not reviewed divergence | replication running and lag low | | Backup restore | Production recovery or drill | Source artifact unverified | MySQL smoke test and app validation | ## Planned failover (graceful switchover) To promote a replica during planned maintenance, annotate the `MysqlFailoverGroup` with the target site name. The operator drains writes on the current primary, waits for the target to catch up, and promotes it atomically -- no `kubectl exec` required. ```bash kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/planned-failover=pdx ``` Optional per-request override (otherwise the `spec.plannedFailover` defaults apply): ```bash kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/planned-failover=pdx:maxLagWait=30s ``` Watch progress on the CR: ```bash kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.plannedFailover}{"\n"}' ``` The state machine advances through `Pending → Validating → Draining → WaitingForLag → Promoting → Resuming → Succeeded`, emitting Kubernetes Events (`PlannedFailoverStarted`, `PlannedFailoverDraining`, `PlannedFailoverLagOK`, `PlannedFailoverCompleted`) at each transition. See [Planned failover](https://bloodraven.dev/docs/operations/planned-failover) for the full lifecycle, rollback behaviour, and RPO guarantees. Unlike the legacy `kubectl exec` dance below, this path: - **Respects the anti-flap cooldown.** A planned failover cannot run within `spec.failoverCooldown` of the last failover (planned *or* emergency). - **Guarantees zero RPO on success.** Promotion only proceeds after `GTID_EXECUTED` on the target ⊇ the fenced source. - **Rolls back on timeout.** If the target does not catch up within `maxLagWait`, the operator clears the fence on the source and reports `status.plannedFailover.phase: Failed` with `reason: LagTimeout`. - **Leaves an audit trail.** Every attempt is recorded on the CR and in the `bloodraven_planned_failovers_total{result=...}` counter. ### Manual promotion (fallback, operator unreachable) If the operator is unreachable and you must promote a site by hand, the legacy three-step dance still works. Use this only as a break-glass procedure; prefer the annotation API above whenever the operator is healthy. **Step 1: Fence the current primary** ```bash SOURCE_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=iad \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n orders "$SOURCE_POD" -c mysql -- \ mysql -u root -p -e "SET GLOBAL super_read_only=ON;" ``` **Step 2: Stop and reset replication on the target** ```bash TARGET_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=pdx \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n orders "$TARGET_POD" -c mysql -- \ mysql -u root -p -e "STOP REPLICA; RESET REPLICA ALL;" ``` **Step 3: Promote the target** ```bash TARGET_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=pdx \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n orders "$TARGET_POD" -c mysql -- \ mysql -u root -p -e "SET GLOBAL read_only=0;" ``` When the operator returns it will observe the new topology on its next poll cycle and update Services, DNS, and taints accordingly. No operator restart is required. ::warning Manual promotion **bypasses** the anti-flap cooldown and the zero-lag gate. Ensure you understand the current replication state before promoting a site -- data loss is possible if the target has not finished applying relay logs. The planned-failover API above is safer; reach for this only when the operator is genuinely unavailable. :: ## Split-brain recovery If both sites report `writable` (split brain), the operator will **not** take automatic action. You must resolve this manually: 1. **Identify the authoritative site** -- determine which site has the most recent data by comparing GTID sets: ```bash IAD_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=iad \ -o jsonpath='{.items[0].metadata.name}') PDX_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=pdx \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n orders "$IAD_POD" -c mysql -- \ mysql -u root -p -e "SELECT @@gtid_executed;" kubectl exec -n orders "$PDX_POD" -c mysql -- \ mysql -u root -p -e "SELECT @@gtid_executed;" ``` 2. **Fence the non-authoritative site**: ```bash PDX_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=pdx \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n orders "$PDX_POD" -c mysql -- \ mysql -u root -p -e "SET GLOBAL super_read_only=ON;" ``` 3. **Re-establish replication** from the authoritative site to the fenced site: ```bash PDX_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=pdx \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n orders "$PDX_POD" -c mysql -- \ mysql -u root -p -e " STOP REPLICA; RESET REPLICA ALL; CHANGE REPLICATION SOURCE TO SOURCE_HOST='mysql-orders-iad-internal.orders.svc.cluster.local', SOURCE_USER='replicator', SOURCE_PASSWORD='', SOURCE_AUTO_POSITION=1; START REPLICA; " ``` :brRead the replicator password from the failover group's credentials Secret. If the group enforces TLS for replication, include the corresponding `SOURCE_SSL_*` options from your MySQL TLS setup as well. 4. The operator will detect the corrected topology and resume normal operation. ::caution Split brain means both sites accepted writes independently. You may need to reconcile conflicting data manually before re-establishing replication. Replication will break if conflicting transactions exist on both sides. :: ## Recovering a divergent old primary After an emergency failover, the old primary may come back with transactions that never replicated. If GTID sets match, the operator sets `RecoveryPending=True` with reason `RecoveryInProgress`, reconfigures the old primary as a replica, and clears the condition after replication is healthy. If the old primary has divergent transactions, the operator fences the site and sets `RecoveryPending=True` with reason `DivergentTransactions`. Check the divergence details: ```bash kubectl get mysqlfailovergroup orders -o jsonpath='{range .status.sites[*]}{.name}: recoveryState={.recoveryState} divergentTxns={.divergentTransactionCount} divergentGtid={.divergentGtid}{"\n"}{end}' ``` To recover the site, investigate the lost transactions and then trigger a reclone: **Step 1: Investigate the lost transactions** Review the divergent GTID set to understand what data was lost. These transactions were committed on the old primary but never reached the replica before failover. **Step 2: Trigger a reclone** Use the reclone annotation to have the operator run `CLONE INSTANCE` on the divergent site, replacing all its data with a fresh copy from the current primary. The annotation value must include a prefix (8 or more characters) of the observed `divergentGtid` — this **confirmation interlock** prevents a fat-fingered site name from destroying the wrong replica: ```bash # Read the divergent GTID first: DG=$(kubectl get mysqlfailovergroup orders -o jsonpath='{.status.sites[?(@.name=="iad")].divergentGtid}') echo "$DG" # a1b2c3d4-0000-0000-0000-000000000000:11-15 # Annotate with : kubectl annotate mysqlfailovergroup orders bloodraven.shipstream.io/reclone-site=iad:a1b2c3d4 ``` If the prefix doesn't match the observed `divergentGtid` (wrong site, stale copy of the GTID, typo), the operator emits a `RecloneRejected` Warning Event and clears the annotation — no clone is started. Re-read `status.sites[].divergentGtid` and try again. A `RecloneRequested` Normal Event marks the start. The site will temporarily appear unreachable during the MySQL restart that follows the clone. Progress is reported in the `Bootstrapping` condition. ### Recloning a read-only site A reader is a valid clone recipient but never a donor. If a reader has lost its PVC or source convergence is blocked and its data can be discarded, use the plugin's destructive cold-reclone confirmation: ```bash kubectl bloodraven reclone --cold ``` For example: ```bash kubectl bloodraven reclone orders reader --cold ``` The operator validates a uniquely writable `primary-candidate` donor before starting `CLONE INSTANCE`. The reader client Service has no healthy endpoint while clone, catch-up, or direct-source convergence is incomplete. Clone reads from the active primary at full speed and is not throttled. ### Troubleshooting blocked source convergence Inspect the generic source fields independently of old-primary recovery: ```bash kubectl get mysqlfailovergroup orders \ -o jsonpath='{range .status.sites[*]}{.name}: source={.sourceHost} state={.sourceConvergenceState} reason={.sourceConvergenceReason}{"\n"}{end}' ``` - `Pending/SourceMismatch` means the observed source is not the confirmed primary and convergence has not completed. - `Pending/ProbeFailed` or `Pending/MutationFailed` means a probe or bounded `STOP REPLICA` / `CHANGE REPLICATION SOURCE TO` / `START REPLICA` attempt failed. Check the four stable convergence log events and their `stage`. - `Blocked/GTIDDiverged` means the active primary does not contain the follower's executed GTID set. Bloodraven intentionally leaves the channel unchanged, or stopped if the second containment check failed after STOP. Do not reset replication metadata to force a rejoin. Review the GTIDs and preserve any required data before recloning. Convergence is also deliberately paused during bootstrap, ordered updates, restore, topology freeze, pending promotion, planned failover, and split brain. Resolve or allow that operation to finish before treating `Pending` as a source fault. ## PVC loss recovery runbook Use this when one site's MySQL data PVC is irrecoverable: cloud volume deleted, local disk lost, filesystem corrupted beyond repair, or a failed clone left the data directory unusable. ::caution **Data loss boundary** If the lost PVC belonged to a primary that accepted writes not yet replicated elsewhere, those transactions are unrecoverable from Bloodraven. PITR can only replay binlog files that survived long enough to be archived; a lost primary PVC takes its unarchived binlog tail with it. See [Durability and RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo#pitr-and-the-rpo-window). :: ### 1. Identify the failed site and current primary ```bash kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.activeSite}{"\n"}{range .status.sites[*]}{.name}: state={.state} recovery={.recoveryState} divergent={.divergentGtid}{"\n"}{end}' ``` Do not wipe the active primary unless you are deliberately restoring from backup. If the damaged site is the active primary, first let the operator fail over or run a planned failover to a healthy primary-candidate site. ### 2. Confirm the target has no recoverable divergent data For normal old-primary divergence, use the GTID-prefix reclone flow above. For PVC loss, the target has no usable data to preserve, so the operator accepts a cold reclone with an explicit destructive confirmation: ```bash kubectl bloodraven reclone orders iad --cold ``` The operator emits `RecloneRequested` and drives the same clone path used for automatic bootstrap. It will not reclone the current active primary. ### 3. If the PVC object itself is bad, delete it If Kubernetes is still binding the pod to a corrupted claim, delete the site pod and PVC so the operator can recreate storage from the CR spec: ```bash IAD_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=iad \ -o jsonpath='{.items[0].metadata.name}') kubectl delete pod -n orders "$IAD_POD" kubectl delete pvc -n orders mysql-orders-iad-data ``` If your cluster uses local-path storage, clear any `db-readonly` taint from the node before deleting the PVC; otherwise the provisioner's helper pod may be evicted and the replacement claim can stay `Pending`. ### 4. Watch bootstrap and clone progress ```bash kubectl describe mysqlfailovergroup orders kubectl get pods -l shipstream.io/failover-group=orders -w ``` Expected condition flow: | Phase | What you should see | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | Requested | `RecloneRequested` Event on the `MysqlFailoverGroup`. | | Bootstrapping | `Bootstrapping=True`; the target site pod may restart while `CLONE INSTANCE` replaces the data directory. | | Recovering | The cloned site comes back read-only and replication is configured from the active primary. | | Healthy | `Bootstrapping=False`, `RecoveryPending=False`, target `state=read-only`, `recoveryState` empty, replication running, and `divergentGtid` empty. | Clone time is proportional to dataset size and storage throughput. Small playground datasets finish in minutes; production datasets can take much longer. ### 5. Verify the site rejoined safely ```bash kubectl get mysqlfailovergroup orders \ -o jsonpath='{range .status.sites[*]}{.name}: state={.state} replicating={.replicating} lag={.secondsBehindSource} recovery={.recoveryState}{"\n"}{end}' ``` The recovered site should be read-only, replicating from the current primary, and below your `spec.replication.maxLagSeconds` threshold before you consider the incident closed. ::warning If the old primary came back with no divergence (GTID sets match), the operator automatically reconfigures it as a replica — no manual intervention is needed. The `RecoveryPending` condition only appears when divergent transactions exist. :: ## Total loss recovery If both sites are `unreachable`, the operator cannot take any action. Recovery steps: 1. Investigate infrastructure (nodes, network, storage) at both sites 2. Bring at least one MySQL instance back online 3. The operator will detect the recovered site and begin normal operation 4. Once the second site is back, re-establish replication if it does not resume automatically ## Scaling storage To resize PVCs, update the `spec.sites[].storage.size` field. This requires the StorageClass to support volume expansion (`allowVolumeExpansion: true`). ```yaml spec: sites: - name: iad storage: storageClassName: fast-ssd size: 200Gi # increased from 100Gi ``` The operator will update the PVC. The underlying volume expansion behavior depends on your storage provider -- some require a pod restart. ## Updating MySQL version Change `spec.image` to the new version: ```yaml spec: image: mysql:9.7 ``` If `updateStrategy: OrderedUpdate` is set, the rollout follows the [ordered update sequence](https://bloodraven.dev/docs/operations/failover#ordered-updates) with zero downtime. Otherwise, both sites are updated simultaneously. See the [Upgrade and version-skew policy](https://bloodraven.dev/docs/operations/upgrade-policy) for the supported MySQL baseline and the cross-site and sidecar ↔ operator skew contract. ## Updating the sidecar Change `spec.sidecarImage`: ```yaml spec: sidecarImage: ghcr.io/shipstream/bloodraven-sidecar:v2.0.0 ``` This follows the same update strategy as MySQL image changes. ## Rotating MySQL credentials ### Credentials mode (`spec.credentials`) When using per-role credential secrets, rotation is fully automated: 1. Update the `password` value in the relevant Kubernetes Secret (e.g. `mysql-operator-creds`) 2. The operator detects the change, connects to the primary, and runs `ALTER USER` to update the MySQL password 3. Pods are rolling-restarted to pick up the new credentials from the updated Secret No manual MySQL commands or operator restarts are needed. ### Legacy mode (`spec.secretName`) 1. Update the password on both MySQL instances directly 2. Update the Kubernetes Secret referenced by `spec.secretName` 3. Restart the operator to pick up the new credentials (or wait for the Secret to be reloaded, depending on your Secret mounting configuration) ## DNS provider configuration DNS provider credentials and settings are managed entirely by external-dns, not by Bloodraven. To change your DNS provider or rotate provider credentials, update your external-dns deployment configuration. Bloodraven only manages the `DNSEndpoint` CR; external-dns handles syncing it to the actual DNS provider. ## Maintenance mode To prevent automatic failovers during maintenance (e.g., network changes), you can temporarily increase the failover cooldown: ```bash kubectl patch mysqlfailovergroup orders --type merge -p ' {"spec": {"failoverCooldown": "24h"}}' ``` Restore it after maintenance: ```bash kubectl patch mysqlfailovergroup orders --type merge -p ' {"spec": {"failoverCooldown": "30m"}}' ``` ## Deleting a failover group The operator uses a finalizer to ensure graceful cleanup when a `MysqlFailoverGroup` is deleted: 1. Node taints are removed from all sites 2. The `DNSEndpoint` CR is automatically garbage-collected via its owner reference to the `MysqlFailoverGroup` 3. MySQL pods, Services, PVCs, and the PodDisruptionBudget are deleted 4. The finalizer is removed and the CR is deleted To delete: ```bash kubectl delete mysqlfailovergroup orders ``` ::warning PVCs are deleted as part of cleanup. If you need to preserve data, back up the MySQL instances before deleting the failover group. :: # kubectl bloodraven plugin `kubectl-bloodraven` is the operator's day-2 sidekick: a single static binary that wraps the annotation grammars and on-demand CR shapes the operator already understands. It is the recommended way to drive a planned failover, kick off an ad-hoc backup or verification, or peek at the current health of a `MysqlFailoverGroup` without reaching for raw `kubectl annotate` / `kubectl apply` incantations. The plugin only touches API objects — it never talks to MySQL directly. Any operator-side validation (cooldown, site role, divergent GTID prefix match, profile existence) still applies, so the safety properties documented under [Failover](https://bloodraven.dev/docs/operations/failover), [Planned failover](https://bloodraven.dev/docs/operations/planned-failover), and the backup pages are preserved. ## Install The plugin follows the [`kubectl` plugin convention](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/){rel=""nofollow""}: any executable named `kubectl-` on `$PATH` is callable as `kubectl `. ```bash make build-kubectl-plugin # produces bin/kubectl-bloodraven make install-kubectl-plugin # copies it into ~/.local/bin (preferred) # or sudo install -m 0755 bin/kubectl-bloodraven /usr/local/bin/ kubectl bloodraven version ``` Pinning a release tag at build time: ```bash make build-kubectl-plugin KUBECTL_PLUGIN_VERSION=v0.2.0 ``` The plugin reuses your existing kubeconfig (`$KUBECONFIG` / `~/.kube/config`), honours `--context` / `--namespace` (`-n`), and inherits the RBAC of the user invoking it — exactly the same surface you'd hit with raw `kubectl get/annotate/create`. ## Commands | Command | What it does | Equivalent without the plugin | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `status [group]` | One-shot health view for one group or every group in the namespace | `kubectl get mysqlfailovergroup -o wide` + `kubectl describe` | | `promote ` | Apply the `planned-failover` annotation with optional `--max-lag-wait` override | `kubectl annotate mysqlfailovergroup bloodraven.shipstream.io/planned-failover=:maxLagWait=...` | | `reclone ` | Re-clone a divergent follower, or cold-reclone a candidate, DR site, or reader with `--cold`; auto-fills the GTID prefix when present | `kubectl annotate mysqlfailovergroup bloodraven.shipstream.io/reclone-site=:` | | `backup --profile ` | Create a `MysqlBackup` CR; optional `--source-site`, `--wait` | `kubectl create -f mysqlbackup.yaml` | | `verify-backup --profile ` | Create a `MysqlBackupVerification` CR; optional `--backup`, `--wait` | `kubectl create -f mysqlbackupverification.yaml` | Global flags accepted by every command: ```text --kubeconfig string Path to kubeconfig (defaults to $KUBECONFIG or ~/.kube/config) --context string Kubeconfig context to use --namespace, -n Namespace (defaults to the kubeconfig context's namespace, or "default") --output, -o string "table" (default), "wide", "json", "yaml" — for commands that print data ``` ### status Per-group health, including site state, replication, recovery progress, planned-failover phase, in-flight restores, backup schedules, and PITR window. ```bash # Detailed report for one group. kubectl bloodraven status orders -n orders # Whole-namespace table. kubectl bloodraven status -n orders # Wide table with planned-failover and recovery columns. kubectl bloodraven status -n orders -o wide # All namespaces. kubectl bloodraven status --all-namespaces # JSON/YAML for piping into other tools. kubectl bloodraven status orders -o json | jq '.status.activeSite' ``` ### promote (planned failover) ```bash # Same effect as `kubectl annotate ... planned-failover=pdx`. kubectl bloodraven promote orders pdx -n orders # With per-request override and synchronous wait. kubectl bloodraven promote orders pdx \ --max-lag-wait 30s \ --wait --timeout 5m ``` The only per-request annotation override the operator recognises is `maxLagWait`. `drainTimeout` is read from `spec.plannedFailover.drainTimeout` on the CR — set it there if you need a non-default value. `--wait` polls `.status.plannedFailover.phase` and prints one line per phase transition until the state machine reaches `Succeeded`, `Failed`, or `Deferred`. **Exit code is 0 only on `Succeeded`.** `Failed`, `Deferred`, and wait timeout all return non-zero. `Deferred` means the operator has accepted the request but the anti-flap cooldown is still ticking — the operator will retry once the cooldown expires, but the promotion has *not* taken effect yet, so scripts that chain commands on success will get the right answer. See [Planned failover](https://bloodraven.dev/docs/operations/planned-failover) for the full lifecycle, rollback rules, and what happens when the target is unhealthy / the anti-flap cooldown is active. ### reclone ```bash # Divergent-GTID case: plugin auto-fills the prefix from # status.sites[iad].divergentGtid. kubectl bloodraven reclone orders iad -n orders # Cold reclone (no divergent GTID recorded). CLONE INSTANCE wipes the # datadir, so the operator requires an explicit confirm token — # `--cold` generates it for you from the group name. kubectl bloodraven reclone orders iad -n orders --cold # Explicit override (matching the documented 8-char minimum). kubectl bloodraven reclone orders iad -n orders --gtid-prefix=a1b2c3d4 ``` Readers use the same destructive cold-reclone workflow. The exact generic form is: ```bash kubectl bloodraven reclone --cold ``` `--cold` is mandatory when no divergent GTID confirmation is available. It generates the group-name confirmation token, and the operator still rejects the active primary or an invalid target. A successful request wipes the reader datadir, clones from the uniquely confirmed active primary, and then waits for direct replication to converge. Clone traffic is unthrottled and runs against the primary, so plan capacity before rebuilding a large reader. ### backup ```bash # Ad-hoc backup against the configured "nightly" profile. kubectl bloodraven backup orders --profile nightly -n orders # Pin the source site, e.g. for taking a dump from the new replica # right after a planned failover. kubectl bloodraven backup orders --profile ondemand --source-site iad -n orders # Block until done. kubectl bloodraven backup orders --profile nightly --wait --timeout 1h -n orders ``` The plugin validates that the profile exists in `spec.backup.profiles` and that any `--source-site` is in `spec.sites` before posting the CR — so a typo fails immediately instead of being deferred to the operator's reconcile loop. The operator additionally rejects `read-only` sites as backup sources; readers are ineligible both for automatic selection and explicit overrides. ### verify-backup ```bash # Verify the latest Succeeded MysqlBackup for the nightly profile. kubectl bloodraven verify-backup orders --profile nightly -n orders # Verify a specific backup with a blocking wait. kubectl bloodraven verify-backup orders --profile nightly \ --backup orders-nightly-abcde --wait --timeout 30m -n orders ``` When the profile carries a `verification` block with PITR / sanity-check options, those settings are copied onto the manual CR verbatim, so a manual run reproduces what the scheduled CronJob would do. ## Why a plugin instead of raw `kubectl`? Three motivations: 1. **Discoverability.** `kubectl bloodraven --help` lists every Bloodraven capability without an admin having to grep CRD YAML. 2. **Safety nets.** Pre-flight validation refuses obviously-broken inputs (wrong site, wrong profile, missing CR) so the operator's event log isn't polluted with `Failed{UnknownSite}` from typos. 3. **Streaming `--wait`.** Long-running operations (planned failover, backup, verification) emit one progress line per phase transition, which makes CI/CD pipelines and runbooks dramatically easier than polling `kubectl get -w` and parsing JSONPath. The plugin is intentionally thin: it never replaces the operator's authority over MySQL, never bypasses the cooldown, and never holds state of its own. Treat it as a typing aid that prints a useful summary at the end. ## Troubleshooting - `error: MysqlFailoverGroup ... not found` — check `--namespace` (or `--context`) and run `kubectl get mysqlfailovergroups -A` to confirm the CR exists. - `error: site "..." is not defined in spec.sites` — list the sites with `kubectl bloodraven status `; spelling is case-sensitive. - `Planned failover failed: reason=CooldownActive` — the anti-flap cooldown is still ticking. Wait for the indicated `retryAfter`, or set `spec.plannedFailover.onCooldown: defer` to queue the request. - `timed out after Xs waiting for ...` — the operation is still in progress on the server side; re-run `kubectl bloodraven status` to inspect the current phase. `--wait --timeout` is the polling budget, not the operator's budget. - A site with `sourceConvergenceState: Blocked` and `sourceConvergenceReason: GTIDDiverged` cannot be safely repointed. Compare the follower and active-primary GTID sets, preserve any data that needs review, then use the divergent-prefix reclone flow or the explicit reader cold-reclone command above. # Runbooks ![runbooks infographic](https://bloodraven.dev/img/infographics/runbooks.webp) Every runbook starts with the same incident header. Fill it in before making changes. | Field | Value | | -------------------- | -------------------------------------------------------------------- | | Severity | Set from paging policy | | Expected impact | Read/write outage, degraded failover, backup risk, or no user impact | | Data loss risk | None, possible async lag loss, or unknown | | Prerequisites | `kubectl`, namespace access, MySQL credentials if SQL is needed | | Required permissions | Read CRs/Events/logs; write CR annotations only when runbook says so | | Estimated time | Record expected and actual time | | Rollback path | Document before action | | Verification command | Always include active site, DNS, and app write check when relevant | If `spec.dragonfly.enabled=true`, include `status.dragonfly.phase`, active Dragonfly Service endpoints, and a Redis-compatible cache/session read/write check in the same incident record. ## Planned failover Use [Planned Failover](https://bloodraven.dev/docs/operations/planned-failover) for controlled maintenance. Do not use during split-brain or when the target replica is unhealthy. ```bash kubectl annotate mysqlfailovergroup orders -n orders \ bloodraven.shipstream.io/planned-failover=pdx --overwrite kubectl get mysqlfailovergroup orders -n orders -o wide ``` Verify DNS and application writes before starting maintenance on the old active site. If Dragonfly is enabled, also verify `status.plannedFailover.dragonfly.sessionsPreserved` and confirm the active Dragonfly Service points at the new site. ```bash kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.status.plannedFailover.dragonfly}{"\n"}' kubectl get endpoints orders-dragonfly -n orders ``` ## Dragonfly degraded Use when `status.dragonfly.phase=Degraded`, `bloodraven_dragonfly_site_up==0`, Dragonfly promotion failed, `status.dragonfly.replTakeoverSupported=false`, or you see `DragonflySessionsLost` / `DragonflyReplTakeoverUnsupported`. MySQL is still authoritative; this runbook protects cache/session continuity. 1. Confirm MySQL `status.activeSite` and application writes are healthy. 2. Inspect `status.dragonfly.replTakeoverSupported` and `replTakeoverProbeMessage`. A false value means the running image does not advertise `REPLTAKEOVER` — pin `v1.38.0+` (or a private build that has the command) and roll the StatefulSets. 3. Inspect `status.dragonfly.sites[]` for unreachable, stale-master, syncing, or link-down sites. 4. Check the active Dragonfly Service endpoints and pod role/traffic labels. 5. Restart or reschedule only the affected Dragonfly pod if the stale site has not accepted writes. 6. If a stale master may have accepted writes, keep it out of the active Service and decide whether cache/session data can be discarded. ```bash kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.status.dragonfly}{"\n"}' kubectl get pod -n orders -l app.kubernetes.io/name=dragonfly --show-labels kubectl get endpoints orders-dragonfly -n orders ``` ## Emergency manual promotion Use only when automation cannot complete and you have confirmed the old primary cannot accept writes. 1. Preserve Events and logs. 2. Confirm old primary is unreachable or fenced. 3. Confirm candidate has the best GTID/replication position available. 4. Promote using your approved MySQL operational procedure. 5. Update or unblock Bloodraven reconciliation only after state is consistent. Verification: ```bash kubectl get mysqlfailovergroup orders -n orders -o yaml dig orders.az.example.com ``` ## Split-brain recovery Use when more than one site can accept writes. 1. Stop application writes if possible. 2. Identify all writable sites. 3. Choose the authoritative site using business ownership and GTID evidence. 4. Fence losing sites by ensuring `super_read_only=ON` and blocking application traffic. 5. Preserve divergent data for manual reconciliation. 6. Reclone losing sites only after data owners approve. Do not skip GTIDs to make replication look green. ## Divergent old primary recovery Use when the old primary contains transactions absent from the promoted site. 1. Export evidence from the old primary. 2. Ask the data owner whether divergent rows must be recovered manually. 3. Keep the old primary fenced. 4. Reclone from the active primary when approved. 5. Verify replication returns to healthy. ```bash kubectl describe mysqlfailovergroup orders -n orders kubectl get events -n orders --sort-by=.lastTimestamp | grep -i divergent ``` ## PVC loss recovery Use when a MySQL data PVC is deleted or corrupted. 1. Confirm whether the affected site is active. 2. If active, stop application writes and follow emergency recovery policy. 3. If standby, reclone from the active primary. 4. If both sites are lost, restore from backup. See [Backup And Restore](https://bloodraven.dev/docs/backup-and-restore/backup-restore) for restore workflows. ## Total site loss Use when all nodes/storage/network for a site are gone. 1. Confirm surviving site is writable and healthy. 2. Confirm DNS points to the surviving site. 3. Disable app scheduling to the lost site. 4. Rebuild site infrastructure. 5. Rejoin via clone or restore. Expected impact is bounded by failover detection, relay-log drain, DNS TTL, and application reconnect behavior. ## Keyring not sealed Use when `bloodraven_keyring_phase{phase="sealed"}` is 0 for a site, or `status.conditions[type=EncryptionAtRestReady]` is `False`. A site that is not sealed is running with a **writable keyring**. Its data is still encrypted, but the operator cannot yet prove the key is durably escrowed. 1. Find out what it is waiting for — `message` carries the reason: ```bash kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}' ``` 2. `Unsealed` with `unsealReason: Clone` or `Bootstrap` is normal and transient. Wait for the clone to finish. 3. `Unsealed` with "waiting for the sidecar to escrow" — check the sidecar can reach the operator: ```bash kubectl exec -n orders deploy/mysql-orders-iad -c sidecar -- \ wget -qO- localhost:8080/keyring/status ``` :br`lastError` on that payload names the failure. Common causes: the operator's auxiliary Service is unreachable, or the escrow token Secret was deleted (the operator re-mints it on the next reconcile, but the pod needs to roll to pick it up). 4. `Escrowed` that never becomes `Sealed` — the pod has not rolled onto the sealed rendering. Check for a stuck ordered update (`status.updatePhase`). 5. `Failed` — read the message and follow [Keyring escrow lost](https://bloodraven.dev/#keyring-escrow-lost) if it names a missing or corrupt Secret. Full background: [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest). ## Keyring escrow lost Use on a `KeyringEscrowMissing` or `KeyringEscrowCorrupt` Event. ::caution **Do not restart the affected pod** A sealed site's escrow Secret is its **only** durable copy of the keys. If the Secret is gone but the pod is still running, the keyring is alive on the pod's tmpfs. Deleting, evicting, or draining that pod destroys it and makes that site's data unrecoverable. :: 1. Confirm the pod is still running: ```bash kubectl get pod -n orders -l shipstream.io/site=iad ``` 2. If it is running, recreate the exact missing Secret from the keyring still mounted in the running pod. Do not force an unseal or rollout: that would destroy the only surviving in-memory copy before it was durable again. ```bash set -o pipefail POD=$(kubectl get pod -n orders \ -l shipstream.io/failover-group=orders,shipstream.io/site=iad \ -o jsonpath='{.items[0].metadata.name}') SECRET=$(kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringSecret}') VERSION=$(kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringVersion}') DIGEST=$(kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringDigest}') KEYRING_DIR=$(kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.spec.encryptionAtRest.keyring.dataFileDir}') KEYRING_DIR=${KEYRING_DIR:-/run/mysql-keyring} FG_UID=$(kubectl get mysqlfailovergroup orders -n orders -o jsonpath='{.metadata.uid}') kubectl exec -n orders "$POD" -c mysql -- cat "$KEYRING_DIR/keyring" | \ kubectl create secret generic "$SECRET" -n orders --from-file=keyring=/dev/stdin ACTUAL_DIGEST=$(kubectl get secret "$SECRET" -n orders -o jsonpath='{.data.keyring}' | \ base64 -d | sha256sum | cut -d' ' -f1) if [[ "sha256:${ACTUAL_DIGEST}" != "$DIGEST" ]]; then kubectl delete secret "$SECRET" -n orders echo "restored keyring digest does not match status; refusing to continue" >&2 exit 1 fi kubectl label secret "$SECRET" -n orders \ app.kubernetes.io/name=mysql-keyring \ app.kubernetes.io/managed-by=bloodraven \ shipstream.io/failover-group=orders shipstream.io/site=iad \ shipstream.io/keyring-version="$VERSION" kubectl annotate secret "$SECRET" -n orders shipstream.io/keyring-digest="$DIGEST" kubectl patch secret "$SECRET" -n orders --type=merge -p="{ \"metadata\":{\"ownerReferences\":[{ \"apiVersion\":\"shipstream.io/v1alpha1\",\"kind\":\"MysqlFailoverGroup\", \"name\":\"orders\",\"uid\":\"${FG_UID}\",\"controller\":true,\"blockOwnerDeletion\":true }]},\"immutable\":true }" SITE_INDEX=$(kubectl get mysqlfailovergroup orders -n orders -o json | \ jq -er '.status.encryptionAtRest.sites | map(.name) | index("iad")') kubectl patch mysqlfailovergroup orders -n orders --subresource=status \ --type=json -p="[ {\"op\":\"test\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/name\",\"value\":\"iad\"}, {\"op\":\"replace\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/phase\",\"value\":\"Sealed\"} ]" ``` :brVerify that the site returns to `Sealed` without restarting the pod. 3. If the pod is already gone, that site's data is unrecoverable. Delete the unreadable PVC, then re-clone it from a healthy peer. The confirmation token is mandatory because this permanently discards the target site's local data: ```bash kubectl delete pvc mysql-orders-iad-data -n orders kubectl annotate mysqlfailovergroup orders -n orders \ bloodraven.shipstream.io/reclone-site='iad:confirm=orders' ``` 4. If **every** site lost its escrow, restore from backup. Backup artifacts are logical dumps protected by a separate key, so they do not depend on any site's keyring. ## Network partition diagnosis Use [Network Partitions](https://bloodraven.dev/docs/operations/network-partitions). Prioritize preventing two writable sites. ```bash kubectl get mysqlfailovergroup orders -n orders -o yaml kubectl get endpoints,networkpolicy -n orders kubectl get events -n orders --sort-by=.lastTimestamp ``` ## Failed backup 1. Keep the last successful backup. 2. Inspect `MysqlBackup` status and Job logs. 3. Fix credentials, storage, or MySQL source health. 4. Trigger a manual backup. 5. Verify the new backup. ```bash kubectl get mysqlbackup -n orders kubectl describe mysqlbackup -n orders ``` ## Failed restore 1. Confirm this is a recovery environment or approved destructive action. 2. Inspect restore Job logs. 3. Validate source path and decryption Secret. 4. Recreate the recovery failover group if bootstrap partially initialized MySQL. ## Operator unavailable Use [Operator Availability](https://bloodraven.dev/docs/architecture/operator-availability). Sidecar self-fencing is part of the safety model; do not disable it during diagnosis. ```bash kubectl rollout status deployment/bloodraven -n bloodraven kubectl logs -n bloodraven deploy/bloodraven kubectl get lease -A | grep bloodraven ``` ## DNS failover stuck 1. Confirm `status.activeSite` changed. 2. Inspect `DNSEndpoint` and external-dns logs. 3. Confirm provider record and TTL. 4. If applications still write old site, fence the old site and restart affected clients. ```bash kubectl get dnsendpoint -A | grep orders kubectl logs -n external-dns deploy/external-dns dig orders.az.example.com ``` # Troubleshooting ![troubleshooting infographic](https://bloodraven.dev/img/infographics/troubleshooting.webp) Use this page by symptom. If an alert fired, start with [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map). ## MysqlFailoverGroup not becoming Ready **Symptoms:** `Ready=False`, `Bootstrapping=True`, MySQL pods exist but the group never settles. **Likely causes:** node labels do not match `taintNodeSelector`, missing Secrets, StorageClass failure, failed clone, DNS CRD missing. **Inspect:** ```bash kubectl describe mysqlfailovergroup orders -n orders kubectl get pods,pvc,events -n orders --sort-by=.lastTimestamp kubectl logs -n bloodraven deploy/bloodraven ``` **Safe remediation:** fix the missing dependency first, then let reconciliation continue. Do not delete PVCs unless this is a disposable bootstrap. **See also:** [Getting Started](https://bloodraven.dev/docs/get-started/getting-started), [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract), [CRD Reference](https://bloodraven.dev/docs/configuration/crd-reference). ## MySQL pod stuck Pending **Symptoms:** MySQL pod has no node, PVC unbound, or scheduler reports no matching nodes. **Likely causes:** missing site labels, taints/tolerations mismatch, unavailable StorageClass, insufficient CPU or memory. **Inspect:** ```bash kubectl describe pod -n orders -l app.kubernetes.io/instance=orders kubectl get nodes --show-labels kubectl get pvc -n orders ``` **Safe remediation:** add required node labels, fix StorageClass, or lower resource requests after confirming capacity. **See also:** [Placement Contract](https://bloodraven.dev/docs/configuration/placement-contract). ## MySQL pod CrashLoopBackOff **Symptoms:** pod restarts repeatedly; MySQL or sidecar logs show bootstrap, TLS, or credential errors. **Likely causes:** invalid MySQL config, missing TLS files, wrong Secret keys, incompatible image, failed clone restart. **Inspect:** ```bash kubectl logs -n orders pod/ -c mysql --previous kubectl logs -n orders pod/ -c sidecar --previous kubectl describe pod -n orders ``` **Safe remediation:** restore the last known-good image/config or fix the Secret/TLS reference. If clone completed and MySQL exited for a container restart, Kubernetes may recover it automatically. **See also:** [Credentials And TLS](https://bloodraven.dev/docs/configuration/credentials-and-tls), [Failure Mode Matrix](https://bloodraven.dev/docs/operations/failure-mode-matrix). ## Site marked unreachable **Symptoms:** `status.sites[].reachable=false`, failover may start after threshold. **Likely causes:** MySQL down, network policy block, node outage, Service endpoint missing, TLS or credential failure. **Inspect:** ```bash kubectl get mysqlfailovergroup orders -n orders -o yaml kubectl get endpoints -n orders kubectl get networkpolicy -n orders ``` **Safe remediation:** restore connectivity or let failover complete. Avoid manual promotion until you know which site accepted the last writes. **See also:** [Failover](https://bloodraven.dev/docs/operations/failover), [Network Partitions](https://bloodraven.dev/docs/operations/network-partitions). ## DNS not changing after failover **Symptoms:** `status.activeSite` changed but `orders.az.example.com` still resolves to the old IP. **Likely causes:** `DNSEndpoint` CRD missing, external-dns not watching the namespace, provider credentials failure, TTL/cache delay. **Inspect:** ```bash kubectl get dnsendpoint -A | grep orders kubectl logs -n external-dns deploy/external-dns dig orders.az.example.com ``` **Safe remediation:** fix external-dns and wait for TTL expiry. If apps are writing to the old site, prioritize fencing and app routing. **See also:** [Runbooks](https://bloodraven.dev/docs/operations/runbooks#dns-failover-stuck), [App Integration](https://bloodraven.dev/docs/configuration/app-integration). ## Dragonfly degraded or active Service empty **Symptoms:** `status.dragonfly.phase=Degraded`, `bloodraven_dragonfly_site_up==0`, `orders-dragonfly` has no endpoints, or Events mention `DragonflyPromotionFailed`. **Likely causes:** Dragonfly pod down, NetworkPolicy blocking port `6379` or admin port `9999`, auth Secret mismatch, target replica syncing/loading, stale master kept out of the active Service, or snapshot restore in progress. **Inspect:** ```bash kubectl get mysqlfailovergroup orders -n orders \ -o jsonpath='{.status.dragonfly}{"\n"}' kubectl get pod -n orders -l app.kubernetes.io/name=dragonfly --show-labels kubectl get endpoints orders-dragonfly -n orders kubectl get events -n orders --sort-by=.lastTimestamp | grep -i dragonfly ``` **Safe remediation:** restore the failed Dragonfly pod or network path first. Do not force a stale master back into the active Service unless you have decided its cache/session contents are authoritative. MySQL remains authoritative for durable data. **See also:** [Runbooks](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded), [App Integration](https://bloodraven.dev/docs/configuration/app-integration), [Monitoring](https://bloodraven.dev/docs/observability/monitoring#available-metrics). ## App still writing to old site **Symptoms:** application errors or writes appear on the old primary after failover. **Likely causes:** DNS cache too long, connection pool did not reconnect, app pinned a site Service, old site not fenced. **Inspect:** ```bash kubectl get mysqlfailovergroup orders -n orders -o jsonpath='{.status.activeSite}{"\n"}' dig orders.az.example.com kubectl get pods -n orders -o wide ``` **Safe remediation:** restart or drain affected app pods, lower DNS/client cache TTL, confirm old site is read-only. **See also:** [App Integration](https://bloodraven.dev/docs/configuration/app-integration), [Network Partitions](https://bloodraven.dev/docs/operations/network-partitions). ## Replication lag high **Symptoms:** `secondsBehindSource` exceeds policy, alerts fire, replica may not be chosen for backups. **Likely causes:** under-sized replica, slow network/storage, long transaction, backup load, SQL thread stopped. **Inspect:** ```bash kubectl get mysqlfailovergroup orders -n orders -o yaml kubectl exec -n orders -c mysql -- mysql -e 'SHOW REPLICA STATUS\G' ``` **Safe remediation:** reduce write load, investigate storage/network, restart replication only after recording the error. **See also:** [Monitoring](https://bloodraven.dev/docs/observability/monitoring), [Durability And RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo). ## Replication stopped **Symptoms:** replica IO or SQL thread down, `replicating=false`. **Likely causes:** credential drift, network failure, missing binlog, divergent transaction, MySQL error. **Inspect:** ```bash kubectl exec -n orders -c mysql -- mysql -e 'SHOW REPLICA STATUS\G' kubectl describe mysqlfailovergroup orders -n orders ``` **Safe remediation:** fix the underlying error. If GTIDs diverged, use the divergent old primary recovery runbook rather than skipping transactions. **See also:** [Runbooks](https://bloodraven.dev/docs/operations/runbooks#divergent-old-primary-recovery). ## Divergent transactions detected **Symptoms:** old primary cannot rejoin; Events or logs mention divergent GTIDs. **Likely causes:** old primary accepted writes that never reached the promoted site. **Inspect:** ```bash kubectl describe mysqlfailovergroup orders -n orders kubectl get events -n orders --sort-by=.lastTimestamp | grep -i divergent ``` **Safe remediation:** preserve evidence, decide whether to extract missing data manually, then reclone the divergent site from the active primary. **See also:** [Runbooks](https://bloodraven.dev/docs/operations/runbooks#divergent-old-primary-recovery), [Durability And RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo). ## Backup Job failing **Symptoms:** `MysqlBackup.status.phase=Failed`, backup Job failed. **Likely causes:** S3 credentials, bucket policy, PVC full, MySQL credentials, staging volume eviction, source replica lag. **Inspect:** ```bash kubectl describe mysqlbackup -n orders kubectl logs -n orders job/ kubectl get events -n orders --sort-by=.lastTimestamp ``` **Safe remediation:** fix credentials/storage first, then create a new manual backup. Do not lower retention until at least one newer backup succeeds. **See also:** [S3 Backups](https://bloodraven.dev/docs/backup-and-restore/backup-s3), [PVC Backups](https://bloodraven.dev/docs/backup-and-restore/backup-pvc). ## Restore Job failing **Symptoms:** restore status is failed, restore Job exits non-zero. **Likely causes:** wrong artifact path, missing decryption passphrase, S3/PVC access, incompatible MySQL shell/server version. **Inspect:** ```bash kubectl describe mysqlfailovergroup orders-restore -n orders kubectl logs -n orders job/ kubectl get pods,pvc -n orders ``` **Safe remediation:** verify source artifact and credentials, then recreate the recovery group if bootstrap state is partial. Preserve failed restore logs. **See also:** [Backup And Restore](https://bloodraven.dev/docs/backup-and-restore/backup-restore), [Backup Encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption). ## Grafana dashboards show no data **Symptoms:** dashboards import but panels are empty. **Likely causes:** Prometheus is not scraping Bloodraven, wrong datasource variable, dashboard ConfigMaps in an unwatched namespace. **Inspect:** ```bash kubectl get servicemonitor -n bloodraven kubectl port-forward -n bloodraven deploy/bloodraven 8080:8080 curl http://localhost:8080/metrics | grep '^bloodraven_' ``` **Safe remediation:** fix scraping first, then dashboard datasource selection. **See also:** [Prometheus Setup](https://bloodraven.dev/docs/observability/monitoring-prometheus), [Grafana Dashboards](https://bloodraven.dev/docs/observability/monitoring-grafana). ## Operator logs show permission denied **Symptoms:** reconcile errors mention forbidden Kubernetes API operations. **Likely causes:** RBAC drift, namespace-scoped install missing permissions, Helm chart RBAC not updated with generated RBAC. **Inspect:** ```bash kubectl auth can-i list mysqlfailovergroups.shipstream.io --as=system:serviceaccount:bloodraven:bloodraven kubectl describe clusterrole bloodraven kubectl logs -n bloodraven deploy/bloodraven ``` **Safe remediation:** restore chart RBAC from the current release. Do not grant wildcard permissions unless your platform policy explicitly accepts it. **See also:** [Security Model](https://bloodraven.dev/docs/configuration/security-model), [Production Install](https://bloodraven.dev/docs/get-started/install-production). # Failover ![failover infographic](https://bloodraven.dev/img/infographics/failover.webp) This page covers the state machine that drives MySQL failover decisions, the exact sequence of operations during a failover, Dragonfly follow-along behavior when enabled, the anti-flap cooldown, and ordered updates for zero-downtime rollouts. For the bounded-RPO contract and the exact set of transactions that can be lost on emergency failover, see [Durability and RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo). For what happens when the operator itself is unavailable during a failure, see [Operator availability](https://bloodraven.dev/docs/architecture/operator-availability). ## State machine Each site in a failover group is tracked independently. The possible states are: | State | Meaning | | ------------- | ----------------------------------------------------------- | | `unknown` | Initial state before the first successful poll | | `writable` | MySQL is reachable and `read_only=0` | | `read-only` | MySQL is reachable and `read_only=1` | | `unreachable` | MySQL has failed the configured number of consecutive polls | ### State transitions ```mermaid stateDiagram-v2 [*] --> unknown unknown --> writable: poll OK + read_only=0\n(after recovery threshold) unknown --> read_only: poll OK + read_only=1 unknown --> unreachable: poll error\n(after failure threshold) writable --> read_only: poll OK + read_only=1 writable --> unreachable: poll error\n(after failure threshold) read_only --> writable: poll OK + read_only=0\n(after recovery threshold) read_only --> unreachable: poll error\n(after failure threshold) unreachable --> writable: poll OK + read_only=0\n(after recovery threshold) unreachable --> read_only: poll OK + read_only=1 ``` **Debouncing:** - A site transitions to `unreachable` only after **failureThreshold** consecutive failed polls (default: 3). For the first fault in an otherwise healthy group, the default 2-second base interval means about 6 seconds of downtime before the operator considers the site unreachable. - A site transitions to `writable` only after **recoveryThreshold** consecutive successful polls showing `read_only=0` (default: 2). This prevents premature promotion on transient successes. - Transitions to `read-only` are immediate (single poll) since this is a safe, non-destructive state. `spec.pollInterval` is a base interval, not a permanently flat cadence. Once any site has reached `failureThreshold` and continues to fail, the shared group poll loop backs off exponentially: with the defaults, later intervals progress from 2 seconds to 4, 8, 16, and then the 30-second cap. A successful probe resets that site's failure count, and the next interval returns to the base when no other site remains in backoff. This does not delay detection of the first isolated failure: the backoff starts only after the site is already `unreachable`. It does affect compound outages. If one site has been down long enough for polling to reach the 30-second cap, a second site that fails can take three capped polls, or up to about 90 seconds, to reach the default `failureThreshold`. Use that bound for alerting and compound-failure recovery estimates. ### Cross-site evaluation After updating individual site states, the operator evaluates the pair together: | Site A | Site B | Action | | ------------- | ------------- | ----------------------------------------------------------------------------------------------- | | `writable` | `read-only` | **Healthy** -- no action needed | | `unreachable` | `read-only` | **Promote Site B** -- the `Degraded` condition reason is `Degraded` until topology converges | | `read-only` | `unreachable` | **Promote Site A** -- the `Degraded` condition reason is `Degraded` until topology converges | | `writable` | `writable` | **Split brain** -- see [Split-brain resolution](https://bloodraven.dev/#split-brain-resolution) | | `read-only` | `read-only` | **No primary** -- alert, no automatic action (with one exception, below) | | `unreachable` | `unreachable` | **Total loss** -- alert, no automatic action | The operator only takes automatic action for the failover case (and, opt-in, for split brain). All other anomalous states require human investigation. `Failover` is not a `Degraded` condition reason. Alert on `Degraded=True, reason=Degraded` for this topology state, on the `FailoverExecuted` Kubernetes Event for a completed promotion, or on an increase in `bloodraven_failovers_total`. A condition alert that matches `reason=Failover` never fires because the operator does not emit that value. The table is the two-candidate core reduced to two columns. In an N-site group, only `primary-candidate` sites can be the active site, promotion target, or split-brain winner. `dr-only` and `read-only` sites are non-promotable. A writable non-promotable site is an anomaly and is fenced on every poll; a reader is fenced even when it is the sole writable site. Reader outages and replication failures remain visible per site but are excluded from group readiness and degradation calculations. Readers never trigger node taint changes or become active DNS targets. **Exception — re-asserting a fenced promoted primary.** A freshly promoted primary can be fenced back to read-only by its own sidecar: the sidecar's fencing lease may still be stale when the promotion lands (for example, the operator restarted after a full-site outage and promoted before its auxiliary Service endpoint became Ready, so the sidecar's operator probes kept failing while the operator was already driving MySQL). That leaves every site reachable and read-only — a state the table above refuses to touch, because without history it is indistinguishable from a fresh-start condition that needs human input. The operator, however, does have history: `status.lastFailoverTarget` names the site it made authoritative. When **all** of the following hold, the operator restores writability on that site instead of alerting and waiting: - a prior failover is recorded and its target is reachable, read-only, and a `primary-candidate`; - every other site is also reachable and read-only (an unreachable site hands the decision back to the normal failover row above); - the target contains every other site's `GTID_EXECUTED` and the recorded `status.promotionGtidExecuted` — restoring it cannot lose transactions or create a second primary. The re-assert is rate-limited to once per `failoverCooldown`, logs `re-asserting fenced promoted primary` (see the [log schema](https://bloodraven.dev/docs/observability/log-schema)), and increments `bloodraven_primary_reassert_total`. GTID divergence between the sites still blocks it — that genuinely needs a human. ## Failover sequence When the operator decides to fail over to a candidate site, it executes these steps in order: 1. **Best-effort fence the old primary** with `SET GLOBAL super_read_only=ON`. If no old-primary connection is available, the sequence continues because the failed site is already isolated from the operator. 2. **Best-effort evict application connections from the old primary** so clients reconnect through the primary Service or DNS. A failure is logged and does not block promotion. 3. **Drain relay logs on the candidate**, bounded by 30 seconds. A timeout or drain error is logged and promotion continues; the drain narrows the normal asynchronous-replication RPO but is not a zero-RPO gate. 4. **Stop replication on the candidate** with `STOP REPLICA`. Failure stops the promotion attempt. 5. **Remove the candidate's replication configuration** with `RESET REPLICA ALL`. Failure stops the promotion attempt. 6. **Capture the promotion GTID** with `SELECT @@global.gtid_executed` before the candidate accepts writes. A read failure is logged and leaves the recorded value empty; it does not stop promotion. 7. **Clear `super_read_only` on the candidate**. This is required because a sidecar or earlier fence may have set it. Failure stops the promotion. 8. **Clear `read_only` on the candidate**, making it writable. Failure stops the promotion. 9. **Confirm writability immediately** with a bounded `read_only` probe. This is part of the same promotion attempt, not the next topology poll. If the probe fails, DNS and failover bookkeeping are not advanced, although the preceding write may already have made MySQL writable. 10. **Record and publish the promotion**. The operator records anti-flap and promotion-GTID state and increments `bloodraven_failovers_total` before it best-effort updates the `DNSEndpoint`. A DNS write failure does not undo a successful MySQL promotion; poll-driven DNS reconciliation keeps retrying. Dragonfly follow-along, when enabled, starts after this MySQL path. Node taints, the `-primary` Service selector, and follower source convergence are poll-driven consequences rather than steps inside `FailoverController.Execute`. The old site's transition to `unreachable` applies its taint before promotion. Later successful polls debounce the new primary to `writable` and remove its taint; the status update then lets the resource reconciler move pod role labels and Service endpoints. Direct-source convergence separately verifies and repairs each remaining follower after replica status collection. ## Direct-source convergence Source convergence runs after replica status collection and outside the promotion sequence. It is therefore able to repair a healthy wrong source after an operator restart even when no failover history exists. Mutation is allowed only when there is exactly one writable `primary-candidate`, no second writable site, and no bootstrap, update, restore, topology freeze, pending promotion, planned failover, or split brain in flight. For a wrong non-empty source, Bloodraven: 1. Verifies that the active primary's `GTID_EXECUTED` contains the follower's executed set. 2. Runs `STOP REPLICA` and repeats both GTID reads and the containment check, closing the race where the SQL applier advances after the first check. 3. Runs `CHANGE REPLICATION SOURCE TO` with the configured credentials and TLS settings, without `RESET REPLICA ALL`. 4. Runs `START REPLICA` and boundedly verifies the direct canonical hostname and both replication threads. If containment fails, status becomes `Blocked/GTIDDiverged` and no unsafe repoint occurs. A post-STOP containment failure leaves replication stopped. Other bounded failures remain `Pending/MutationFailed` for a later safe retry. This generic state is recorded in `sourceHost`, `sourceConvergenceState`, and `sourceConvergenceReason`; it does not replace old-primary `recoveryState`/`divergentGtid` reporting. ## Dragonfly during failover When `spec.dragonfly.enabled=true`, Dragonfly follows the MySQL failover group but remains best-effort cache/session state, not durable data. During **planned failover**, Bloodraven inserts two Dragonfly phases before MySQL promotion: 1. `WaitingForDragonflySync` captures the source Dragonfly replication offset and waits for the target Dragonfly replica to catch up, bounded by `spec.dragonfly.plannedFailover.maxSyncWait` (default `30s`). 2. `PromotingDragonfly` removes the source pod's `shipstream.io/dragonfly-traffic` label, promotes the target with `REPLTAKEOVER`, stamps the target as `shipstream.io/dragonfly-role=master`, and best-effort kills old-master clients so they reconnect through the active Dragonfly Service. If sync or `REPLTAKEOVER` fails, `spec.dragonfly.plannedFailover.onSyncTimeout` controls the outcome. The default `proceed` continues MySQL promotion and records `status.plannedFailover.dragonfly.sessionsPreserved=false`. `fail` rolls back before MySQL promotion and leaves the original MySQL primary active. During **emergency failover**, MySQL promotion is the priority. After MySQL promotion succeeds, the operator attempts to promote Dragonfly on the new MySQL active site within a bounded budget. It first tries `REPLTAKEOVER` to preserve sessions; if that fails, it falls back to `REPLICAOF NO ONE`, which restores a writable Dragonfly master but does not guarantee session continuity. That fallback emits Warning Events `DragonflyPromotionCompleted` and `DragonflySessionsLost` and increments `bloodraven_dragonfly_promotions_total{result="sessions_lost"}`. It never returns an error to the MySQL path. If Dragonfly is unreachable, MySQL recovery still completes. The Dragonfly manager also handles Dragonfly-only failures. If the active Dragonfly master dies while MySQL remains healthy, the manager can promote the single healthy Dragonfly replica and leave `status.activeSite` for MySQL unchanged. ## Old primary recovery After an emergency failover, the old primary may come back online. The operator automatically detects this and takes action based on whether the old primary's data has diverged from the new primary. ### Detection On each poll cycle, if a site is read-only with no active replication (the signature of a former primary) while another site is the directly confirmed writable primary, the operator initiates recovery. Recovery is deliberately **not** gated on a recorded failover: a primary can change hands without one (a replica respawns writable and is adopted while the old primary respawns fenced, or the failover record was lost to a status-write outage plus an operator restart), and the orphaned ex-primary still needs to rejoin. Only a **genuinely fresh datadir** is skipped and left to bootstrap/auto-clone. Freshness is decided from GTID history, not from the absence of user schemas: a site whose `GTID_EXECUTED` UUIDs share nothing with the new primary (server-init transactions under a brand-new `server_uuid`) may still be treated as empty when it also has no user schemas. A returning cluster member always carries the cluster's shared UUIDs, so a schemaless but previously participating site still runs the divergence comparison — never a silent clone-over. If the new primary cannot be probed for this check, the operator fails safe toward recovery rather than toward the fresh-datadir path. The sequence: 1. **Fence** the returning site with `SET GLOBAL super_read_only=ON` (defensive — the sidecar may have already fenced it) 2. **Drain application sessions** until a pass finds none or `spec.connectionDrainTimeout` elapses (default `30s`). Each topology poll performs at most one bounded eviction pass, so recovery waits without blocking failure detection or failover progress. This runs after promotion, so it cannot kill the operator's promotion session. A timeout does not block recovery because the fence prevents writes; any survivor is limited to stale reads. 3. **Query `@@global.gtid_executed`** on both the old and new primary 4. **Compare GTID sets** to determine if the old primary has any transactions not on the new primary If the old primary returns **writable** (e.g., power was cut before the sidecar could self-fence), the operator first detects this as a split-brain condition and fences it immediately. The fence is retried on **every** poll cycle while the split-brain persists — a transient error on the first attempt (a network blip during the site's recovery turbulence) does not leave the returning site writable. Recovery proceeds once the site transitions to read-only. ### No divergence (automatic rejoin) If the new primary's GTID set contains all transactions from the old primary, there is no data loss. The operator automatically reconfigures the old primary as a replica: 1. `SET GLOBAL super_read_only=ON` 2. `STOP REPLICA` 3. `RESET REPLICA ALL` 4. `CHANGE REPLICATION SOURCE TO ... SOURCE_AUTO_POSITION=1` 5. `START REPLICA` While the sequence runs, `status.sites[].recoveryState` is `RecoveryInProgress` and the `RecoveryPending` condition is `True` with reason `RecoveryInProgress`. The operator keeps that state until MySQL reports healthy replication and the bounded application-connection drain has completed, then writes `replicating=true` and `gtidExecuted` for the read-only site and clears recovery state. ### Divergence detected (manual intervention required) If the old primary has committed transactions that never replicated to the new primary, the operator: - Keeps the site **fenced** (`super_read_only=ON`) - Records the divergent GTID set and transaction count in `status.sites[].divergentGtid` and `status.sites[].divergentTransactionCount` - Sets `status.sites[].recoveryState` to `RecoveryBlocked` - Sets the `RecoveryPending` condition to `True` with reason `DivergentTransactions` - Emits the `bloodraven_divergent_transactions` Prometheus metric While a site is `RecoveryBlocked`, the operator re-verifies the divergence roughly every 30 seconds rather than freezing the first report. If the site diverges *further* before you resolve it (for example its pod respawned writable, accepted a few writes, and was re-fenced), `divergentGtid` and the count refresh to the full current set. A writable observation does not erase that evidence unless the site is both the recorded failover target and the unique, directly confirmed writable primary; split-brain remains blocked. If the divergence is resolved externally — you replay the missing transactions onto the new primary so its GTID set comes to contain the old primary's — the re-check notices containment and automatically rejoins the site as a replica. ::caution Divergent transactions mean the old primary accepted writes that the new primary never received. These transactions are effectively lost from the replication stream. There are exactly two sanctioned ways out: **re-clone** the site from the current primary (discarding the divergent transactions after you have extracted what you need), or **replay the divergent transactions onto the new primary** so its GTID set comes to contain the old primary's — the periodic re-verification then rejoins the site automatically. Do not manually reconfigure replication on the divergent site — conflicting GTID sets will cause replication errors. :: To recover a divergent site: 1. Investigate the divergent transactions to understand what data was lost (check `status.sites[].divergentGtid`) 2. Trigger a reclone using the annotation, including the first 8+ characters of the observed `divergentGtid` as a confirmation token: ```bash # Read the divergent GTID first: kubectl get mysqlfailovergroup -o jsonpath='{.status.sites[?(@.name=="")].divergentGtid}' # Then annotate with :: kubectl annotate mysqlfailovergroup bloodraven.shipstream.io/reclone-site=: ``` 3. The operator validates that the prefix matches the observed `divergentGtid` — a mismatch is rejected with a `RecloneRejected` Warning Event, so a fat-fingered site name can't destroy the wrong replica. When a site has **no** `divergentGtid` (cold reclone: PVC loss, manual rebuild), use `kubectl bloodraven reclone --cold` to provide the required destructive confirmation. 4. The operator runs `CLONE INSTANCE` on the target site, replacing all data with a fresh copy from the current primary. A `RecloneRequested` Event marks the start. See [Recovering a divergent old primary](https://bloodraven.dev/docs/operations/operations#recovering-a-divergent-old-primary) for the full procedure. ### Prerequisites Old primary recovery requires replication credentials (`MYSQL_REPLICATION_USER` and `MYSQL_REPLICATION_PASSWORD`) in the Secret referenced by `spec.secretName`. Without these, recovery is skipped and the site remains fenced. ## Split-brain resolution When the state machine observes both sites as `writable` simultaneously, the operator's response is tiered: 1. **After a prior operator-initiated failover** -- The operator already knows which site it promoted (`status.lastFailoverTarget`). The other site being writable means the old primary returned. The operator fences it immediately (`SET GLOBAL super_read_only=ON`) and recovery proceeds on the next poll. This runs regardless of `spec.splitBrainPolicy`. 2. **No prior failover history, `spec.splitBrainPolicy.sitePriorities` is non-empty** -- The first listed site that is currently writable wins. The operator fences every other writable site and re-promotes the winner through the standard failover path. The anti-flap cooldown still applies to this promotion. 3. **No prior failover history, no priorities configured** -- The operator alerts only (`SPLIT BRAIN: both sites are writable`) and takes no automated action. This is the default. ### When `sitePriorities` applies `sitePriorities` is an ordered tiebreaker for states the operator cannot resolve from its own history. The two common triggers: - **Fresh deploy with existing data.** Both sites come up writable and `lastFailoverTarget` is empty because this operator instance has never failed anything over. Without `sitePriorities`, the operator alerts and waits for an admin. - **Operator restart amnesia.** In-memory `lastFailoverTarget` is repopulated at startup from the newer of `status.lastFailoverTarget` and the out-of-band annotation (see [Where the cooldown is stored](https://bloodraven.dev/#where-the-cooldown-is-stored)), so a failure on either durable path no longer costs the operator its history. But if a split brain occurred during the restart window — for example, an old primary came back *while* the operator was restarting — the operator may never have had an opportunity to record the most recent failover at all. `sitePriorities` provides a deterministic answer in this case. When history *is* available (case 1 above), the operator trusts it and does **not** consult `sitePriorities`. This preserves the invariant that the site most recently promoted keeps its writes. ### Configuration ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: sites: - name: iad # ... - name: pdx # ... splitBrainPolicy: sitePriorities: [iad, pdx] # iad wins when writable; pdx is next ``` Every entry must match a `primary-candidate` in `spec.sites`; the CRD's CEL validation rejects other names and non-promotable sites at admission time. An empty or omitted list leaves unresolvable split brain in alert-only mode. ### Data-loss implications ::caution `sitePriorities` is a **policy** decision, not a safety feature. When the operator fences a losing site to resolve a split brain: - Any transactions committed on the losing site that did *not* replicate to the winner are isolated. - Those transactions are *not* automatically replayed, merged, or preserved. They remain on the losing site's PVC but are outside the replication stream. - When the fenced site attempts to rejoin, Bloodraven's existing divergent-GTID detection compares `executed_gtid_set` on both sides. If the loser has GTIDs the winner never saw, rejoin is **blocked** and the site must be recloned to recover. The divergent GTID set and transaction count are recorded in `status.sites[].divergentGtid` and `status.sites[].divergentTransactionCount`. - In other words, `sitePriorities` makes split-brain resolution fast and deterministic by selecting one history branch. The loser's unreplicated writes are isolated and surfaced through the `RecoveryBlocked` condition and `bloodraven_divergent_transactions` gauge, but they are not merged into the winner. :: Configure `sitePriorities` only when your operational model has a clear authority order -- for example, a primary region that should win whenever it is writable, followed by explicit fallback regions. ### Observability - Metric: `bloodraven_split_brain_auto_resolve_total{prefer_site=""}` -- counter, incremented for a successful losing-site fence; the retained label name identifies the winning site. - Log event: `split-brain auto-resolve: fencing non-preferred site per spec.splitBrainPolicy.sitePriorities` at `WARN`, with `winner` and `fencedSite` fields. - The standard failover log, metric (`bloodraven_failovers_total`), and DNS-flip metric (`bloodraven_dns_flips_total`) also fire, since split-brain resolution runs through the same promotion path. ## Anti-flap cooldown To prevent rapid failover oscillation (e.g., a flapping network link), the operator enforces a cooldown period between automatic failovers. The default is 5 minutes, configurable via `spec.failoverCooldown`. During the cooldown: - The operator continues to monitor both sites and update status - Automatic failovers are suppressed - Manual intervention can still be performed (see [Operations](https://bloodraven.dev/docs/operations/operations)) The cooldown timer resets after each failover. ### Where the cooldown is stored The last failover time and target are written to two places on every promotion, so an operator restart cannot silently reset the cooldown: | Location | Written by | | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------- | | `status.lastFailover`, `status.lastFailoverTarget` | the per-poll CR status update | | `bloodraven.shipstream.io/last-failover`, `bloodraven.shipstream.io/last-failover-target` annotations | a merge patch issued inline with the promotion | The two travel different API paths — the status subresource has its own RBAC rule and admission chain — so an outage on one does not take the other with it. Both writes retry every poll until accepted, and a restarting operator rehydrates from whichever copy carries the later timestamp. A restart that had to fall back to the annotations logs `restored lastFailover from out-of-band annotations` at WARN. That means this group's status writes were failing when it last promoted, and is worth alerting on. Losing the cooldown across a restart requires **both** paths to be rejecting writes at once; see [Known limitations](https://bloodraven.dev/docs/get-started/known-limitations) → Operator availability. The annotations are operator-owned bookkeeping. Editing or removing them by hand changes what a restart believes about the cooldown; the running process keeps its in-memory value either way and rewrites both copies on its next promotion. ## Ordered updates When `spec.updateStrategy` is set to `OrderedUpdate`, spec changes (such as a new `image`, effective per-site MySQL config, or resource adjustments) are rolled out with zero downtime: ```mermaid sequenceDiagram participant Op as Operator participant Standby as Standby Site participant Active as Active Site Op->>Standby: 1. Update deployment (new image/resources) Op->>Standby: 2. Wait for pod ready Op->>Standby: 3. Wait for replication to catch up Op->>Standby: 4. Failover to standby (now updated) Note over Op,Active: Standby becomes new active Op->>Active: 5. Update old active site deployment Op->>Active: 6. Wait for pod ready Op->>Active: 7. Replication resumes Note over Op,Active: Both sites now updated ``` This sequence ensures: - The active primary is never restarted while serving traffic - Replication is healthy before each transition - At most one site is unavailable at any time For groups with more than two sites, every drifted non-active follower is updated sequentially and must be read-only with a direct healthy source before and after restart. If the active site has no drift, the rollout completes without failover; this is the normal reader-only configuration path. If the active site is drifted, only a healthy `primary-candidate` standby may receive the handoff. A reader or `dr-only` follower is never promoted to facilitate an update, and failed or unprocessed drift remains queued for a later reconcile. Without `OrderedUpdate`, both sites are updated simultaneously, which may cause brief downtime if both pods restart at the same time. For MySQL image changes, see the [Upgrade and version-skew policy](https://bloodraven.dev/docs/operations/upgrade-policy). The replica-first ordering above is the MySQL-required direction for a rolling version upgrade. # Planned failover ![planned failover infographic](https://bloodraven.dev/img/infographics/planned-failover.webp) Planned failover is the admin-triggered graceful switchover path. It is the operational-hygiene counterpart to [emergency failover](https://bloodraven.dev/docs/operations/failover): use it for maintenance windows, rolling kernel upgrades on the active site's nodes, or any time you want to move the primary role to a specific replica **without** relying on the operator's automatic detection. ## TL;DR ```bash kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/planned-failover=pdx ``` The operator fences the current primary, waits for `pdx` to catch up on the fenced source's GTID set, promotes it, flips DNS, and clears the fence. When `spec.dragonfly.enabled=true`, the same request also waits for the target Dragonfly replica to catch up and promotes it before MySQL promotion. Status lands on `.status.plannedFailover.phase: Succeeded` with `transactionsLost: 0`. Only `primary-candidate` sites are valid targets. `dr-only` and `read-only` followers are never promoted, used for active DNS, or involved in node-taint handoff. Readers remain direct followers and application read pools after the switchover. ## Why not `kubectl exec`? Manual promotion via three `kubectl exec ... SET GLOBAL ...` commands (see [Operations](https://bloodraven.dev/docs/operations/operations#manual-promotion-fallback-operator-unreachable)) works but has four real problems: 1. **No atomicity** — a dropped connection or typo mid-sequence leaves the cluster fenced-on-both-sides or promoting a replica that has not drained its relay logs. 2. **No lag gate** — nothing mechanical checks that the target is caught up before you flip `read_only=0`. 3. **No audit trail** — `bloodraven_failovers_total` does not increment, no Event fires, no CR status reflects what you did. 4. **Bypasses the anti-flap cooldown** — a switchover immediately before a real emergency exposes the cluster to a window where the operator refuses to take automatic action. The annotation API closes all four. ## Lifecycle ```text Pending ──► Validating ──► Draining ──► WaitingForLag ──► WaitingForDragonflySync ──► PromotingDragonfly ──► Promoting ──► Resuming ──► Succeeded │ │ │ │ │ │ │ │ │ └──► Deferred (cooldown active, onCooldown=defer) ──► re-enters Validating │ └─ reject ────┴─ rollback ────┴─ rollback ────┴─ rollback/proceed ───────┴─ rollback/proceed ───┴─ fail ──────┴──► Failed ``` `WaitingForDragonflySync` and `PromotingDragonfly` are skipped when Dragonfly is disabled. | Phase | What the operator is doing | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Pending` | Annotation observed; status block initialised. | | `Validating` | Checking target is primary-candidate, replicating, read-only, no concurrent restore/update, cooldown not active. | | `Deferred` | `spec.plannedFailover.onCooldown: defer` is set and cooldown is still active. Annotation is retained; reconciler re-validates at `status.plannedFailover.retryAfter`. | | `Draining` | Two steps, one per reconcile: (1) `SET GLOBAL super_read_only=ON` on source + record `sourceGtidAtFence`, strip `role=primary` pod label so the `-primary` Service sheds endpoints; (2) `KillAppConnections` in a loop until idle or `drainTimeout` elapses. | | `WaitingForLag` | Polling the target's `GTID_EXECUTED` until it contains the source's fenced GTID set. | | `WaitingForDragonflySync` | When Dragonfly is enabled, captures the source Dragonfly replication offset and polls the target until it catches up or `spec.dragonfly.plannedFailover.maxSyncWait` expires. | | `PromotingDragonfly` | When Dragonfly is enabled, strips the source Dragonfly traffic label, promotes the target with `REPLTAKEOVER`, updates role/traffic labels, and best-effort kills old-master clients so they reconnect through the active Service. | | `Promoting` | Running the same promotion sequence as emergency failover: kill app connections, relay-log drain, `STOP REPLICA`, `RESET REPLICA ALL`, capture promotion GTID, clear `super_read_only`, `SET GLOBAL read_only=0`, verify the target is writable, then flip DNS. | | `Resuming` | Persisting `status.activeSite`, `status.lastFailover`, `status.promotionGtidExecuted`; releasing the topology-manager guard. | | `Succeeded` | Terminal success. Status block retained until the next annotation replaces it. | | `Failed` | Terminal failure. See rollback behaviour below. | One phase transition per reconcile means operator restarts always land on a well-defined observable state — if the operator crashes during `WaitingForLag`, the next reconcile resumes the wait from the persisted `sourceGtidAtFence`. ## Spec-level defaults Cluster-wide knobs live on the CR so common overrides don't have to be spelled into every `kubectl annotate`: ```yaml spec: plannedFailover: maxLagWait: 5m # max time to wait for the target to catch up before rolling back drainTimeout: 30s # upper bound on KillAppConnections polling after super_read_only=ON onCooldown: reject # one of: "reject" (default) or "defer" dragonfly: plannedFailover: maxSyncWait: 30s # Dragonfly target sync / REPLTAKEOVER budget onSyncTimeout: proceed # one of: "proceed" (default) or "fail" ``` All fields are optional; omitting `plannedFailover` is equivalent to `{maxLagWait: 5m, drainTimeout: 30s, onCooldown: reject}`. Omitting `dragonfly.plannedFailover` is equivalent to `{maxSyncWait: 30s, onSyncTimeout: proceed}` when Dragonfly is enabled. - **`maxLagWait`** — how long `WaitingForLag` polls the target before rolling back. Set shorter than the default when you know the cluster is caught up and want to fail fast. - **`drainTimeout`** — upper bound on the `Draining` phase's `KillAppConnections` loop. After `super_read_only=ON`, the reconciler kills stragglers every second until the count reaches zero or the budget expires; advancing to `WaitingForLag` happens either way so a stuck client cannot block the switchover. - **`onCooldown`** — what happens when `Validating`rejects the request because the anti-flap cooldown is active: - `reject` (default): stamp `Failed{CooldownActive}` and clear the annotation. The admin must re-annotate after the cooldown expires. - `defer`: stamp `Deferred`, keep the annotation in place, and re-try validation automatically at `status.plannedFailover.retryAfter`. Useful when you want to queue a switchover immediately after an emergency event without manually waiting out the cooldown. - **`dragonfly.plannedFailover.maxSyncWait`** — how long `WaitingForDragonflySync` waits for the target Dragonfly replica to reach the source offset. The same duration is used as the `REPLTAKEOVER` timeout. - **`dragonfly.plannedFailover.onSyncTimeout`**— what happens when Dragonfly sync or promotion cannot prove session preservation: - `proceed` (default): continue MySQL promotion and stamp `status.plannedFailover.dragonfly.sessionsPreserved=false`. - `fail`: roll back before MySQL promotion, release the MySQL fence, and leave `status.activeSite` unchanged. ### Per-request override The annotation value may carry a `maxLagWait` override, mirroring the reclone-annotation key=value grammar: ```bash kubectl annotate mysqlfailovergroup orders \ bloodraven.shipstream.io/planned-failover=pdx:maxLagWait=30s ``` Useful when you know the target is caught up and want to fail fast rather than wait five minutes for the default timeout. ## Status Inspect progress: ```bash kubectl get mysqlfailovergroup orders \ -o jsonpath='{.status.plannedFailover}{"\n"}' ``` Example `Succeeded` block: ```yaml status: plannedFailover: phase: Succeeded target: pdx sourcePrimary: iad sourceGtidAtFence: "abc-...:1-9182731" targetGtidAtPromotion: "abc-...:1-9182731" startTime: "2026-04-20T14:32:00Z" completionTime: "2026-04-20T14:32:47Z" durationSeconds: 47 transactionsLost: 0 dragonfly: enabled: true sessionsPreserved: true promotionMethod: REPLTAKEOVER syncWaitSeconds: 3 message: "promoted pdx, 0 transactions lost" ``` ## Rollback and failure modes Every path out of `Draining`, `WaitingForLag`, or `Promoting` either completes a promotion or restores the old primary to writable. Below, **rollback** means the operator unfenced the source and left the cluster unchanged. | Failure | Observable | Rollback? | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | Target is unknown / dr-only / read-only / not replicating | `phase: Failed`, `reason: TargetUnhealthy` or `UnknownSite`, event `PlannedFailoverRejected` | no fence applied | | Anti-flap cooldown active + `onCooldown: reject` | `phase: Failed`, `reason: CooldownActive`, message includes `retry after ...` | no fence applied | | Anti-flap cooldown active + `onCooldown: defer` | `phase: Deferred`, `reason: CooldownActive`, `retryAfter` populated; annotation retained | no fence applied; reconciler retries automatically | | Admin removes the annotation while `Deferred` | `phase: Failed`, `reason: Cancelled` | no fence applied | | Concurrent in-place restore or ordered update | `phase: Failed`, `reason: ConcurrentOperation` | no fence applied | | Target is mid-keyring-rotation (`UnsealReason=Rotation`) | `phase: Failed`, `reason: KeyringRotation`, event `PlannedFailoverRejected` | no fence applied; wait for `Sealed` and re-annotate | | Zero-lag wait times out | `phase: Failed`, `reason: LagTimeout`, source unfenced, `role=primary` label restored, no DNS flip | yes | | `drainTimeout` elapses with clients still connected | `Draining` advances to `WaitingForLag` anyway with a message noting remaining connections | N/A — source is already fenced, clients get `ER_OPTION_PREVENTS_STATEMENT` on their next write | | Source crashes during `Draining` | `phase: Failed`, `reason: SourceCrashed` | hand-off to emergency failover path | | Dragonfly target fails to catch up and `onSyncTimeout=proceed` | `phase` advances through `PromotingDragonfly`, `status.plannedFailover.dragonfly.sessionsPreserved=false` | no MySQL rollback; MySQL promotion continues | | Dragonfly target fails to catch up and `onSyncTimeout=fail` | `phase: Failed`, `reason: DragonflySyncTimeout`, source unfenced, no DNS flip | yes | | Dragonfly `REPLTAKEOVER` fails and `onSyncTimeout=proceed` | `status.plannedFailover.dragonfly.sessionsPreserved=false`, `reason: DragonflyPromotionFailed`; MySQL promotion continues | no MySQL rollback | | Dragonfly `REPLTAKEOVER` fails and `onSyncTimeout=fail` | `phase: Failed`, `reason: DragonflyPromotionFailed`, source unfenced, no DNS flip | yes | | `FailoverController.Execute` fails mid-promotion | `phase: Failed`, `reason: ExecuteFailed` | **no** — same failure mode as emergency; manual recovery required | | DNS flip fails | `phase: Succeeded` with a warning event | N/A — the Service label swap carries writes to the new primary while external DNS catches up | The topology-manager's automatic cross-site evaluation is **paused** while a planned failover is in flight (via the `plannedFailoverActive` guard). When the planned path stamps `Succeeded` or `Failed`, the guard is released and the automatic path resumes. ## Follower convergence after success Once the target is confirmed writable and planned-failover mutation is no longer in flight, Bloodraven converges every other candidate, `dr-only`, and `read-only` follower directly to the new primary. A chained or stale source is not considered healthy merely because its replication threads are running. Before `STOP REPLICA`, the new primary's `GTID_EXECUTED` must contain the follower's executed set. Bloodraven repeats this check after STOP to close the SQL-applier race, then changes the source and starts replication without resetting replication metadata. A failure is retried boundedly on subsequent polls. GTID non-containment is reported per site as `sourceConvergenceState: Blocked` and `sourceConvergenceReason: GTIDDiverged`; the operator does not discard the follower's extra transactions or silently repoint it. Reader client endpoints remain absent until MySQL is read-only, both threads are healthy, direct-source convergence is `Converged`, and known lag is within the effective reader threshold. A blocked or lagging reader does not change the successful planned-failover result or the group's shared Ready/Degraded conditions. ## Anti-flap cooldown The planned path writes `status.lastFailover` on success, exactly like the emergency path. This means: - A planned failover within `spec.failoverCooldown` of any prior failover is **rejected** at `Validating`. - An emergency failover within `spec.failoverCooldown` of a planned failover is **blocked** by the same cooldown check in the topology manager. The default cooldown is 5 minutes; production deployments should keep this, not lower it. ## Observability ### Metrics | Metric | Type | Labels | | ---------------------------------------------- | --------- | ---------------------------------------------------------------------------------- | | `bloodraven_planned_failovers_total` | counter | `target_site`, `result` (`success`, `rejected`, `failed_timeout`, `failed_other`) | | `bloodraven_planned_failover_duration_seconds` | histogram | `target_site` | | `bloodraven_planned_failover_lag_wait_seconds` | histogram | `target_site` | | `bloodraven_dragonfly_promotions_total` | counter | `group`, `target_site`, `result` (`success`, `failed`, `skipped`, `sessions_lost`) | `bloodraven_failovers_total` (automatic only) and `bloodraven_dns_flips_total` retain their existing semantics. Dashboards keyed on `bloodraven_failovers_total` see no change. ### Events `PlannedFailoverStarted`, `PlannedFailoverSkipped` (already-active target), `PlannedFailoverRejected`, `PlannedFailoverDeferred` (cooldown + defer policy), `PlannedFailoverDraining`, `PlannedFailoverLagOK`, `PlannedFailoverCompleted`, `PlannedFailoverFailed`, `DragonflyPromotionStarted`, `DragonflyPromotionCompleted`, `DragonflyPromotionFailed`, and `DragonflySyncTimeout`. ## RPO Planned failover is the **one** Bloodraven operation with an RPO of zero by construction. The zero-lag gate at `WaitingForLag` guarantees the target has replicated every committed transaction on the fenced source before promotion. Any scenario that would produce loss (target falls behind, source crashes mid-drain, catch-up times out) routes to the `Failed` rollback path instead of promoting a lagging replica. See [Durability and RPO](https://bloodraven.dev/docs/architecture/durability-and-rpo). # Network partitions ![network partitions infographic](https://bloodraven.dev/img/infographics/network-partitions.webp) This runbook expands the network rows in the [failure-mode matrix](https://bloodraven.dev/docs/operations/failure-mode-matrix). It focuses on what the operator can observe, which action it takes, which metrics/events move, and what an operator should do after connectivity returns. Timing assumes the defaults: `pollInterval=2s`, `failureThreshold=3`, `sidecar.leaseTimeout=20s`, and a 30-second relay-log drain timeout. `pollInterval` is the base cadence. Continued failures after a site is marked `unreachable` back the group loop off exponentially to a 30-second cap. A first isolated partition still debounces in about 6 seconds; a second fault during a sustained partition can take up to about 90 seconds to reach the three-poll threshold. Sidecar lease timing is independent of this operator poll backoff. ## First principles - Bloodraven promotes only when the current primary is unreachable and a promotable `primary-candidate` replica is reachable. - A replica-side network problem does not trigger failover while the primary remains writable. - The sidecar self-fences a writable MySQL when it cannot reach the operator and cannot reach any peer for longer than `leaseTimeout`. - Cross-site replication lag is an alert, not a failover trigger, when the primary is otherwise healthy. - Once a partition heals, GTID comparison decides whether the returning site can rejoin automatically or must be recloned. ## Scenario A: operator cannot reach site A, site B reachable Example: the active primary is `iad`; the operator and `pdx` are on the surviving side of a site partition; all operator polls to `iad` time out. | Aspect | Expected behavior | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Observable signal | `iad` transitions to `unreachable`; `pdx` remains `read-only` and replicating until the link fully breaks. | | Operator action | After debounce, promotes `pdx` via the normal emergency failover path, updates Services/DNS, and taints old-site nodes. | | Sidecar action | If `iad` is still running but isolated from both operator and peers, its sidecar sets `super_read_only=ON` at roughly T+20s. | | Metrics | `bloodraven_site_state{site="iad",state="unreachable"}=1` (also labelled `namespace`, `group`, `role`), `bloodraven_failovers_total{target_site="pdx"}` increments on successful promotion, `bloodraven_dns_flips_total{site="pdx"}` increments when DNS is updated, `bloodraven_taint_operations_total{site="iad",action="taint"}` increments. | | Events | Expect `FailoverExecuted` after the promotion and, if the old primary later returns diverged, `DataLossDetected`. | | RPO | Bounded by what had replicated to `pdx` before the partition. Any writes accepted only on `iad` after the partition are divergent when `iad` returns. | Recovery after heal: 1. Watch `status.sites[?(@.name=="iad")].recoveryState`. 2. If there is no divergence, the operator reconfigures `iad` as a replica automatically. 3. If `divergentGtid` is set, review lost transactions and trigger the reclone flow in [Operations](https://bloodraven.dev/docs/operations/operations#recovering-a-divergent-old-primary). ## Scenario B: replica site isolated, primary reachable Example: active primary `iad` remains reachable by the operator; replica `pdx` cannot receive replication traffic or cannot be polled. | Aspect | Expected behavior | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Observable signal | `pdx` becomes `unreachable`, or replication IO/SQL stops and `secondsBehindSource` climbs. `iad` remains `writable`. | | Operator action | No failover. The primary is healthy, so promoting away would reduce availability and risk data loss unnecessarily. | | Sidecar action | Replica sidecar does not self-fence a read-only MySQL; replicas are already not accepting writes. | | Metrics | `bloodraven_site_state{site="pdx",state="unreachable"}=1` or `bloodraven_replication_lag_seconds{role="primary-candidate"}` rises; `bloodraven_replication_running{site="pdx",thread="io"}` may become `0`. | | Events | Degraded/alert events for unreachable or lagging replication; no failover-complete event. | | RPO | Emergency failover is not available until a primary-candidate replica is reachable and reasonably current. | Recovery after heal: 1. Confirm replication restarts and lag falls below `spec.replication.maxLagSeconds`. 2. If replication does not restart, inspect MySQL replica status and operator logs for `old primary recovery failed` or replication errors. 3. If the replica has been manually written to and now diverges, reclone it from the current primary. ## Scenario C: MySQL-to-MySQL link broken, operator reaches both Example: the operator can poll both `iad` and `pdx`, but `pdx` cannot pull binlog events from `iad`. | Aspect | Expected behavior | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Observable signal | Primary remains `writable`; replica remains `read-only`; replica IO thread stops or lag increases. | | Operator action | No automatic failover. From the operator's perspective this looks like replication lag or IO pressure, not primary failure. | | Metrics | `bloodraven_replication_running{thread="io"}=0` and/or `bloodraven_replication_lag_seconds` exceeds the configured threshold. | | Events | Replication-lagging / degraded events; no DNS flip and no taint change. | | Human action | Decide whether the link outage is temporary. If the primary is healthy, keep serving writes there. If you need to move writes, use planned failover only after the replica catches up. | This is the scenario most likely to page a human without automatic action. Bloodraven intentionally refuses to guess that lag means the primary should be abandoned. ## Scenario D: asymmetric peer reachability Example: operator reaches both sites, `iad` can reach `pdx`, but `pdx` cannot reach `iad`; or only one sidecar can reach its peer. | Aspect | Expected behavior | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Observable signal | Poll results can remain healthy while replication or peer checks show one-way failures. | | Operator action | Follows the data-plane state it can poll: no failover while the active primary is reachable and writable. | | Sidecar action | A writable primary self-fences only when both the operator and every peer are unreachable beyond `leaseTimeout`. One reachable peer is enough to avoid self-fencing. | | Metrics | Usually replication lag/running metrics, plus possible state transitions if MySQL polling is affected. | | Events | Degraded/replication events unless the asymmetry also makes the primary unreachable to the operator. | Asymmetric partitions are exactly why GTID reconciliation after heal matters. If either side accepted writes that the final primary did not receive, the returning site is fenced and marked divergent. MySQL replication is replica-initiated, so the two orientations are not equivalent. A cut from replica to primary stops IO fetch; a cut from primary to replica does not. Sidecar peer checks are likewise client-initiated: a writable primary self-fences on the lease rule only when it cannot reach the operator **and** cannot reach any peer. ## Scenario E: both sites unreachable to the operator | Aspect | Expected behavior | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Observable signal | All sites become `unreachable`; `Degraded=True` with total-loss semantics. | | Operator action | No promotion. There is no reachable candidate to promote. | | Sidecar action | Any still-running writable site self-fences after `leaseTimeout` if it cannot reach operator or peers. | | Metrics | `bloodraven_site_state{state="unreachable"}=1` for all sites; no failover counter increment. | | Events | Total-loss / degraded events. | | Human action | Restore at least one site or the operator network path. Once one site is reachable, the operator resumes normal reconciliation. | ## Testing partitions Use pod-level `NetworkPolicy` or CNI-native fault injection. Host-level iptables rules in a k3d node are not reliable for Kubernetes Service traffic because kube-proxy DNAT and pod networking happen in different paths. Playground examples live in `playground/chaos-scenarios.md` and use NetworkPolicy-based partitions. Always run `./playground/chaos.sh recover` or delete the NetworkPolicy manually after a test. Coverage of the five shapes: | Shape | What broke | Live chaos | DST | | -------------------------------- | ---------------------------------------------- | ------------------------------------ | ---------------------------------- | | A — operator ↛ site | operator cannot poll one site | 09, 06 | `partitionOperatorSite` | | B — replica isolated | replica unreachable; primary healthy | 17 (negative assertion) | — | | C — MySQL ↛ MySQL | both directed peer cuts; operator reaches both | none | `partitionPair` (two one-way cuts) | | D — asymmetric peer reachability | one directed peer cut | none | `partitionOneWay` | | E — all sites lost | operator cannot reach any site | 11 (scale-to-0, not a network fault) | both sites `partitionOperatorSite` | Shape D is the lease-rule case: one reachable peer suppresses self-fence even when the operator is silent. The reverse orientation (primary cannot reach the replica) is what lets the lease expire. Both orientations are hand-built DST trials; the campaign also schedules `partitionOneWay`. ## On-call checklist 1. Identify the current active site: `kubectl get mysqlfailovergroup orders -o jsonpath='{.status.activeSite}'`. 2. Print per-site state: `kubectl get mysqlfailovergroup orders -o jsonpath='{range .status.sites[*]}{.name}: {.state} lag={.secondsBehindSource} recovery={.recoveryState}{"\n"}{end}'`. 3. Check whether `bloodraven_failovers_total` increased. If not, the operator likely chose to alert rather than promote. 4. If a site returns with `divergentGtid`, do not manually attach it as a replica. Follow the reclone flow. 5. After recovery, verify replication lag is below your RPO threshold and that DNS/external-dns has converged if clients use external DNS. # Failure-mode matrix ![failure mode matrix infographic](https://bloodraven.dev/img/infographics/failure-mode-matrix.webp) A single-table reference for "what happens when *X* breaks" — mapping common failure modes to the operator's detection signal, the action it takes, roughly when it acts, and what it will **not** do. Use this to cross-check an incident in progress against the operator's expected behavior, and to decide which alerts to carry. Timing assumes the shipped defaults: `spec.pollInterval=2s`, `spec.failureThreshold=3`, `spec.failoverCooldown=5m`, and sidecar `spec.sidecar.leaseTimeout=20s`. The 2-second value is the base cadence. After a confirmed site outage, continued failures back the shared group poll loop off exponentially to a 30-second cap. The first isolated failure still reaches the threshold in about 6 seconds, but a second fault during a sustained outage can take up to about 90 seconds to collect three failed polls. See [Failover: Debouncing](https://bloodraven.dev/docs/operations/failover#state-transitions). ## Quick reference | Failure | Observable signal | Operator action | Operator time-to-act | Operator limitations | | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Primary pod killed** | MySQL poll fails on primary; pod `NotReady`. | Debounces to `unreachable` → promotes replica via [failover sequence](https://bloodraven.dev/docs/operations/failover#failover-sequence) (fence old primary, drain relay, promote, flip DNS). | First isolated fault: ≈ 6 s detection + up to 30 s relay drain, or ≈ **37 s total**. During an existing sustained site outage: up to ≈ 90 s detection + drain. | If replica is lagging, committed transactions that never reached it can be lost. In a healthy group, a primary that respawns inside the ≈ 6 s debounce window is not failed over. | | **Replica pod killed** | Poll fails on replica; replica becomes `unreachable` and `Degraded=True` with reason `Degraded`. | Marks replica `unreachable`; narrows the `-replicas` Service selector to exclude it. **No failover** — the primary is still writable. | First isolated fault: ≈ 6 s. During an existing sustained site outage: up to ≈ 90 s. | Read-only workloads hitting the `-replicas` Service lose endpoints until the replica returns. Backups fall back to the primary (controlled by `maxLagSecondsForSource`). | | **Primary node lost** | Primary pod goes `NotReady` + no node heartbeat. | Same as *primary pod killed* — fails over to the replica. Kubernetes separately reschedules the pod onto a healthy node. | ≈ 37 s for the first isolated fault; compound-fault detection can add up to ≈ 90 s before the relay drain. Pod reschedule is Kubernetes's job and depends on eviction policy (30 s – 5 min). | If the PVC is zone-bound (e.g. `topology.kubernetes.io/zone`-pinned EBS) and the lost node was the zone's only node, the pod cannot reschedule until capacity returns. The operator has already failed over; the zone-bound PVC problem is a Kubernetes concern. | | **Replica node lost** | Poll fails on replica. | Marks replica `unreachable`; narrows `-replicas` selector. No failover. | ≈ 6 s as a first fault; up to ≈ 90 s during a sustained outage elsewhere in the group. | Same as replica-pod-killed. Kubernetes reschedules the pod to another node with access to the PVC. | | **Primary PVC lost** | Primary pod restarts into an empty MySQL or stuck bootstrap. GTID-empty detection fires on a successful poll. | Fails over (if the primary was active) → after recovery, auto-clones the empty site from the current primary via `CLONE INSTANCE`. | First-fault failover: ≈ 37 s. An already-backed-off group can take up to 30 s to observe the successful empty-site probe. Clone duration depends on dataset size (minutes to hours). | Committed-but-unreplicated transactions on the lost PVC are unrecoverable. Auto-clone requires `MYSQL_REPLICATION_USER`/`PASSWORD` in the operator Secret (see [Failover → Prerequisites](https://bloodraven.dev/docs/operations/failover#prerequisites)). | | **AZ / site partition** (operator can still reach the surviving side) | All poll targets at the isolated side time out; one side's pods go `NotReady`. | Fails over to the reachable site. Sidecar on the partitioned primary self-fences at T + `leaseTimeout` = **T + 20 s**, closing the split-brain window. | First isolated fault: ≈ 37 s to promotion. During a sustained outage elsewhere: up to ≈ 90 s detection + relay drain. Partitioned-side self-fencing remains driven by the 20 s sidecar lease. | If the partitioned side had unreplicated transactions, they are "stranded" until the partition heals — at which point the operator detects `divergentGtid` and blocks recovery until an admin runs the reclone flow. | | **Cross-site partition** (link between MySQL pods dies; operator still reaches both) | Replica's IO thread stops receiving events; `Seconds_Behind_Source` climbs. Eventually `Degraded=True ReplicationLagging`. | **None automatic.** The primary is still writable, the replica is still read-only. The operator alerts but does not fail over. | Alert at `maxLagSeconds` (default 300 s). | This mode is indistinguishable, from the operator's point of view, from "replica fell behind because of I/O pressure". Human judgement decides whether to force a failover. | | **Operator pod crash** | `up{job="bloodraven"} == 0`. Liveness probe fails. CR status stops updating. | Kubernetes restarts the pod. On return, leader election resumes, polling resumes. | Pod restart ≈ 5–10 s. | No failover decisions happen during downtime. If the primary fails *while* the operator is down, writes are unavailable until the operator restart + a full failover sequence completes. Sidecars self-fence to preserve correctness. | | **Operator down + one site fails** | Combined: operator liveness fails AND one MySQL site becomes `unreachable`. | Sidecars self-fence at T + `leaseTimeout`; no promotion happens until the operator returns, then it runs the normal failover sequence. | Operator-down duration + ≈ 37 s. | Write availability is lost for the operator-down window plus the post-restart failover. **Correctness is preserved** (no split-brain), but availability takes a direct hit. RPO is unchanged vs. the operator-up case. | | **Both sites unreachable (TotalLoss)** | All polls to both sites fail. `Degraded=True` with reason `TotalLoss`. | **None.** Emits `TotalLossDetected` Event and alerts. No automatic recovery is possible — nothing to fail over *to*. | ≈ 6 s when both fail from a healthy baseline. If the second failure begins after polling has backed off for the first, it can take up to ≈ 90 s. | Requires human intervention. Once at least one site is back, the operator resumes; if both return, split-brain handling takes over. See [Operations → Total loss recovery](https://bloodraven.dev/docs/operations/operations#total-loss-recovery). | | **Split brain** (`writable/writable`) | Both sites poll as `writable` simultaneously. | With no recorded failover history and empty `spec.splitBrainPolicy.sitePriorities`, alert via `SplitBrainDetected` and require human action. With priorities, fence every writable loser and promote the first listed writable candidate. After an operator-initiated failover, fence the returning stale primary regardless of priorities. | One effective poll interval: normally ≈ 2 s, but up to 30 s while the group is backed off for another outage. | No automatic data merge. If a fenced side accepted writes that the winner lacks, those transactions surface as `divergentGtid` and require replay or a reclone. See [Failover → Split-brain resolution](https://bloodraven.dev/docs/operations/failover#split-brain-resolution). | | **Old primary returns diverged** | After a failover, the returning site has committed transactions the new primary never received. | Fences the returning site, records `status.sites[].divergentGtid` + count, sets `RecoveryPending=True` with reason `DivergentTransactions`, emits `DataLossDetected` Event. | The first successful observation can take one effective poll interval: normally ≈ 2 s, but up to 30 s during poll backoff. Connection drain and bounded recovery probes add time before divergence status is final. | Cannot auto-rejoin — divergent GTIDs would corrupt replication. Admin must replay the missing transactions or run the reclone flow; the operator enforces the confirmation interlock on the annotation. See [Failover → Divergence detected](https://bloodraven.dev/docs/operations/failover#divergence-detected-manual-intervention-required). | | **DNS provider down / slow** | `DNSEndpoint` CR updates succeed, but application DNS lookups still return the old A-record. | Operator writes the new `DNSEndpoint` on failover and moves on — DNS propagation is **external-dns's** job, not the operator's. | CR update happens within the failover sequence (a few seconds). External-dns reconcile time and your DNS provider's TTL decide the app-visible delay. | The operator cannot accelerate DNS propagation. Applications should use short TTLs (`spec.dns.ttl`, default 60 s) and not cache MySQL connection addresses longer than the TTL. Monitor external-dns independently; a stuck external-dns is an outage for writes even after the operator has "finished". | | **S3 (or backup storage) unreachable** | Backup Jobs fail; `bloodraven_archiver_upload_failures` climbs; `bloodraven_archiver_backlog_files` > 0; PITR `lastUploadAt` stops advancing. | Backup reconciler retries per `spec.backup.retry`. PITR archiver retries on every scan (`archivePollInterval`, default 60 s). | Retry backoffs are bounded by the retry spec; next archival attempt is one poll away. | **No data-plane impact** — MySQL keeps serving writes. PITR RPO silently drifts as unarchived binlogs accumulate. Alert on `bloodraven_archiver_backlog_files > 0` sustained. See [Monitoring](https://bloodraven.dev/docs/observability/monitoring#available-metrics). | | **Anti-flap cooldown blocks needed failover** | Second failover condition arises within `failoverCooldown` of the previous one. | Operator logs `failover blocked by anti-flap cooldown` and does **not** fail over. | No automatic action until the cooldown expires (default 5 min). | The [planned-failover API](https://bloodraven.dev/docs/operations/planned-failover) also honours this cooldown and will be rejected with `reason: CooldownActive`. If you genuinely need a second failover inside the cooldown window, perform a [manual promotion](https://bloodraven.dev/docs/operations/operations#manual-promotion-fallback-operator-unreachable) — that path bypasses the cooldown explicitly. | | **Primary dies while a replica is mid-keyring-rotation** | `status.encryptionAtRest.sites[].unsealReason=Rotation` on the would-be target. `Degraded=True` reason `NoPrimary` with a message naming `UnsealReason=Rotation`. Warning Event `KeyringPromotionRefused` (or `KeyringPromotionSkipped` if another sealed candidate was promoted). Counter `bloodraven_keyring_promotions_blocked_total{outcome="refused|skipped"}`. | **Refuses to promote the rotating site.** If another sealed primary-candidate exists, that site is promoted instead. If the rotating site is the only remaining candidate, the group stays without a writable primary until the site returns to `Sealed`. Planned failover to that target is rejected with `reason: KeyringRotation`. | Immediate on the failover decision (same debounce as any primary death). Healing after the replica seals is the next poll. | Deliberate availability cost: promoting a mid-rotation replica would make it the sole authoritative copy while its new master key is not yet sealed, so a pod loss in that window is unrecoverable. There is no cancel-rotation command. Finish the rotation (repair escrow/sidecar if it is stalled), then retry. See [Rotating keys](https://bloodraven.dev/docs/configuration/encryption-at-rest#rotating-keys). | | **Planned failover target fails to catch up** | Admin annotated `bloodraven.shipstream.io/planned-failover=`; target's `GTID_EXECUTED` does not cover source's fenced GTID within `maxLagWait`. | Operator unfences the source, stamps `status.plannedFailover.phase: Failed` with `reason: LagTimeout`, emits `PlannedFailoverFailed` Event. `role=primary` label is restored; no DNS flip occurred. | `maxLagWait` (default 5 min). | No data loss; cluster returns to pre-annotation state. Retry after replication has caught up, or use emergency failover if the source is truly gone. See [Planned failover → Rollback](https://bloodraven.dev/docs/operations/planned-failover#rollback-and-failure-modes). | | **Dragonfly master killed** | `spec.dragonfly.enabled=true`; `status.dragonfly.activeSite` is unreachable, and one replica is reachable and not syncing/loading. | Promotes the replica with `REPLTAKEOVER` or fallback promotion, updates Dragonfly role/traffic labels, and leaves MySQL `status.activeSite` unchanged. | Next Dragonfly manager poll plus promotion budget. | Cache/session continuity is best-effort. If no healthy Dragonfly replica exists, the active Dragonfly Service may have no endpoints until recovery. MySQL availability is unaffected. | | **Dragonfly unavailable during MySQL emergency failover** | MySQL primary is failed, and Dragonfly target/source cannot be reached or promoted. | Completes MySQL failover first. Dragonfly promotion is attempted best-effort and may record `bloodraven_dragonfly_promotions_total{result="failed"}` or `{result="sessions_lost"}` on the `REPLICAOF NO ONE` fallback. | MySQL path remains ≈ 37 s; Dragonfly attempt is bounded separately. | Sessions/cache may be lost or unavailable. Dragonfly never blocks emergency MySQL recovery. | | **Planned Dragonfly sync timeout** | Planned failover enters `WaitingForDragonflySync`, but the target Dragonfly replica does not reach the source offset within `maxSyncWait`. | With `onSyncTimeout=proceed` (default), promotes MySQL and stamps `sessionsPreserved=false`. With `onSyncTimeout=fail`, rolls back before MySQL promotion. | `spec.dragonfly.plannedFailover.maxSyncWait` (default 30 s). | `proceed` preserves MySQL availability at the cost of cache/session continuity. `fail` preserves the old active site and avoids claiming session preservation. | | **Replication credentials missing / wrong** | Auto-clone fails; `Bootstrapping` condition stays in `Failed`. | Operator reports the failure via the `Bootstrapping` condition and events; does not keep retrying indefinitely. | Immediate (on the first bootstrap attempt). | Without `MYSQL_REPLICATION_USER` / `MYSQL_REPLICATION_PASSWORD`, old-primary recovery and auto-clone are both disabled. The rest of the operator (polling, failover, DNS) keeps working. | ## Backup encryption failure modes These rows apply only when a backup profile has `spec.backup.profiles[].encryption` set. See [Backup encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption) for the full threat model. | Failure | Detection path | Operator action | Typical time-to-act | Notes & limits | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Passphrase Secret missing at restore time** | Restore Job init container can't mount `spec.initFromBackup.decryption.passphraseSecret`; `kubectl describe pod` shows `CreateContainerConfigError`. | Reconciler records `DecryptionSecretMissing` on the restore CR; stops creating retry Jobs after `backoffLimit`. | Seconds (Job pod fails to start). | Recovery: recreate the Secret with the original passphrase value, or delete & re-apply the restore CR. Ciphertext is unrecoverable without the exact passphrase used at backup time — treat the Secret as critical recovery material. | | **Passphrase rotated between backup and restore** | `decrypt-download` init container exits non-zero with a chunk-auth failure from `backupcrypto`. | Job pod's logs show `backupcrypto: chunk 0: cipher: message authentication failed`; reconciler reports `DecryptionFailed`. | Immediate on first read. | AES-GCM catches mismatched keys cleanly — there's no silent garbage load. Recovery requires producing the old passphrase; there is no re-wrap path today. | | **Ciphertext truncated by S3 multipart abort** | `decrypt-download` returns `backupcrypto: ciphertext truncated before final chunk` (`ErrTruncated`). | Job pod fails fast; reconciler reports `BackupCiphertextTruncated`. | Immediate. | `ErrTruncated` distinguishes "tail missing" from a generic auth failure. Recover by re-running the source backup and restoring from the replacement artifact. | | **Object has no BRV1 magic (tamper / downgrade)** | `decrypt-download` returns `sidecar: object missing BRV1 magic while encryption is required` (`ErrTamperedOrDowngrade`). | Job pod fails fast; reconciler reports `BackupDecryptionRejected`. | Immediate. | Defends against an attacker with bucket write access overwriting ciphertext with attacker-chosen plaintext. Legacy mixed-encryption deployments migrating to encryption can opt back into the old passthrough behavior via `BLOODRAVEN_ALLOW_PLAINTEXT_FALLBACK=1` (sidecar / decrypt-download init container) — time-bounded migration only. | ## How to read this table for your on-call Two common uses: 1. **"What's the operator doing right now?"** Find the row matching the symptom, read the *Operator action* and *time-to-act* cells to set your clock for when the situation should resolve on its own. If the expected time has passed and nothing has changed, check the *Operator limitations* column. 2. **"Do I need to page a human?"** Alert on the rows whose *Operator action* is "none automatic" (cross-site partition, total loss, split brain, DNS provider, S3). These are the cases where the operator has deliberately deferred to a human. ## Related reading - [Failover → State machine](https://bloodraven.dev/docs/operations/failover#state-machine) — the per-site and cross-site truth tables that drive every row's *Operator action* cell. - [Failover → Failover sequence](https://bloodraven.dev/docs/operations/failover#failover-sequence) — the step-by-step sequence behind the first-isolated-fault "≈ 37 s" number. - [Operations](https://bloodraven.dev/docs/operations/operations) — runbooks for the rows where the operator action is "none automatic". - [Monitoring](https://bloodraven.dev/docs/observability/monitoring) — the metrics and events each row's *Observable signal* column refers to. - [Production hardening](https://bloodraven.dev/docs/configuration/production-hardening) — which rows here the hardening checklist is specifically guarding against. # Backup And Restore Overview ![backup overview infographic](https://bloodraven.dev/img/infographics/backup-overview.webp) Bloodraven backs up MySQL with mysqlsh `util.dumpInstance()` and restores with `util.loadDump()`. Backups are configured on `MysqlFailoverGroup.spec.backup`; one-off backup runs use `MysqlBackup`. ## Choose a storage backend | Backend | Best for | Avoid when | | ------- | ----------------------------------------------------------------- | ---------------------------------------------------------- | | S3 | Off-cluster durable backups, cross-cluster restore, production DR | You cannot grant object-store credentials safely | | PVC | Local labs, isolated environments, short-term staging | You need disaster recovery after cluster or storage loss | | PITR | Lower RPO between full dumps | You cannot operate binlog archive storage and verification | ## Workflow 1. Configure a backup profile on the failover group. 2. Add a schedule for recurring backups. 3. Trigger one manual `MysqlBackup` before go-live. 4. Verify the backup with `MysqlBackupVerification`. 5. Document the restore path and test it on a non-production group. ## Backup source eligibility Bloodraven prefers a healthy replica within the configured backup lag threshold and falls back to the confirmed active primary. A healthy `dr-only` follower remains eligible. A `read-only` site is never selected as an automatic backup source, and setting `sourceSiteOverride` or `kubectl bloodraven backup --source-site` to a reader is rejected. This keeps application read pools and backup load separate; reclassify the site only if you intend to change its operational role. ## Start here - [S3 Backups](https://bloodraven.dev/docs/backup-and-restore/backup-s3) for object storage. - [PVC Backups](https://bloodraven.dev/docs/backup-and-restore/backup-pvc) for local persistent volumes. - [Backup And Restore Reference](https://bloodraven.dev/docs/backup-and-restore/backup-restore) for full behavior, retention, PITR, and restore safety. - [Backup Encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption) for application-level encryption. - [Backup Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) for proving recoverability. ::warning **Restore safety** `spec.initFromBackup` initializes a new failover group from a backup. In-place restore controls are destructive. Confirm target namespace, failover group name, source backup, and DNS exposure before applying restore manifests. :: # Backup and restore ![backup restore infographic](https://bloodraven.dev/img/infographics/backup-restore.webp) Bloodraven drives scheduled and on-demand MySQL backups using the `mysqlsh util.dumpInstance()` / `util.loadDump()` utilities against a configurable S3 or PVC destination. Restores are bootstrap-only in v1: populate a brand-new `MysqlFailoverGroup` from a previous dump before cross-site replication starts. ::tip **Quick setup guides** Use [S3 Backups](https://bloodraven.dev/docs/backup-and-restore/backup-s3) or [PVC Backups](https://bloodraven.dev/docs/backup-and-restore/backup-pvc) for copy-paste setup. This page is the deep behavior reference for source selection, retention, verification, PITR, restore semantics, and failure modes. :: ## Backup decision table | Need | Use | | --------------------------------------------- | ----------------------------------- | | Durable off-cluster recovery | S3 profile | | Local lab or short-lived staging copy | PVC profile | | Lower RPO than full backup cadence | PITR binlog archival | | Proof that backups load | `MysqlBackupVerification` | | Compliance-grade application-level encryption | `spec.backup.profiles[].encryption` | ## Restore workflow checklist - Confirm whether this is a new recovery group or a destructive in-place restore. - Confirm source artifact, namespace, bucket/PVC path, and decryption Secret. - Disable production traffic to the recovery DNS name until validation completes. - Verify MySQL starts, expected schemas exist, and application smoke tests pass. - Only then move traffic or update application configuration. ## Recovery time and sizing considerations Restore time depends on dump size, compression, object-store or PVC throughput, `loadOptions.threads`, MySQL startup time, and any PITR replay window. Size staging volumes and verification PVCs for at least the compressed artifact plus restore working space; production teams should measure this with a full-size backup before go-live. ## Concepts - **Backup profile** — a named, reusable configuration living on `MysqlFailoverGroup.spec.backup.profiles[]`. Selects a storage target (S3 or PVC), dump options, and retention. - **Backup schedule** — a cron expression on `MysqlFailoverGroup.spec.backup.schedules[]` that references a profile by name. Each schedule becomes a Kubernetes `CronJob` owned by the operator. - **`MysqlBackup` CRD** — one CR per backup run. Created ad-hoc via `kubectl create` or by the schedule CronJob. Tracks phase, start / completion times, dump location, size, GTID, and binlog coordinates. - **Restore** — `spec.initFromBackup` on the failover group. Gates initial bootstrap on a one-shot `util.loadDump()` into a dynamically- selected target site. - **Verification** — a parallel `MysqlBackupVerification` CRD and `spec.backup.profiles[].verification` block that periodically restores the latest Succeeded backup into a throwaway MySQL instance to prove it's actually loadable. See [Backup verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification). - **Encryption at rest** — opt-in, per-profile `spec.backup.profiles[].encryption` block that turns on client-side envelope encryption (AES-256-GCM) for every dump artifact and archived PITR binlog. The passphrase lives in a Kubernetes Secret the operator controls, so keys and storage credentials can be kept on separate blast radii. See [Backup encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption). ## The backup image The default image is pinned to `container-registry.oracle.com/mysql/community-server:9.7`. This bundles the `mysqlsh` binary; the `community-shell` repository that appears in some docs **does not exist** in the Oracle registry — a common stumbling block. Production deployments should always pin this explicitly via `spec.backup.image` and avoid floating tags like `:9` or `:latest`, since `mysqlsh` dump/load compatibility across versions is not guaranteed. ## Backup source selection The reconciler prefers the *replica* site as the dump source so the primary is not loaded by long-running backups. It falls back to the primary when the replica is unreachable, not replicating, or lagging beyond `spec.backup.maxLagSecondsForSource` (default `300`). Override per-backup with `spec.sourceSiteOverride` on a `MysqlBackup` CR. If no healthy source is available before a Job is created, the backup stays `Pending` with reason `NoHealthySource` and retries for up to 30 minutes; a normal failover window does not consume the one-shot backup as a terminal failure. The reconciler emits a warning Event on the transition into `Pending`. If no healthy source appears within that window the CR fails terminally (still with reason `NoHealthySource`) so retention pruning can reclaim it. Once the Job is created, its selected source is pinned. Later reconciles observe that Job without re-running source selection against a topology that may be changing. When `ActiveSite` changes *while* a backup is in flight, the backup reconciler emits an `InFlightFailover` warning event on the CR. This is a soft signal: the artifact is still a valid point-in-time snapshot of the original source, but operators should know a failover happened mid-dump so they can correlate it with any replication-gap alerts. ## Security context defaults Every backup, restore, and cleanup Job pod runs with a hardened pod- and container-level `SecurityContext` matching the Restricted Pod Security Standard: | Level | Field | Default | | --------- | -------------------------- | ---------------- | | pod | `runAsNonRoot` | `true` | | pod | `runAsUser` / `runAsGroup` | `27` (mysql) | | pod | `fsGroup` | `27` | | pod | `seccompProfile.type` | `RuntimeDefault` | | container | `allowPrivilegeEscalation` | `false` | | container | `readOnlyRootFilesystem` | `true` | | container | `runAsNonRoot` | `true` | | container | `capabilities.drop` | `[ALL]` | | container | `seccompProfile.type` | `RuntimeDefault` | Because `readOnlyRootFilesystem=true`, the backup Job pod also mounts two ephemeral `emptyDir` volumes: `mysqlsh-home` at `/home/mysqlsh` (mysqlsh needs a writable home for `~/.mysqlsh`) and `tmp` at `/tmp`. `HOME` is set to `/home/mysqlsh` so mysqlsh discovers them cleanly. Users can override individual fields via `spec.backup.podSecurityContext` and `spec.backup.containerSecurityContext`. Overrides are merged on top of the defaults — any unset field stays at the default, and any set field wins. This lets you tighten further (e.g. set a specific `seLinuxOptions`) without reintroducing capabilities or root access. ## Credential layout MySQL credentials are never injected into the backup pod via `envFrom`. Instead the reconciler derives a Secret per backup run (with `MYSQL_USER` / `MYSQL_PASSWORD` keys) and mounts it **as files** at `/run/bloodraven/mysql-creds/` with mode `0400`. The embedded Python dump script reads the files via the `BLOODRAVEN_MYSQL_CREDS_DIR` env var. This keeps plaintext passwords out of `/proc/$PID/environ` on the pod and makes accidental env-var leaks (e.g. in panic stack traces) much less dangerous. In **credentials mode** (`spec.credentials`), the derived Secret reads `username`/`password` directly from `spec.credentials.backupSecret` (or falls back to `operatorSecret` if no dedicated backup secret is configured). In **legacy mode** (`spec.secretName`), the credentials are parsed from the DSN in the referenced Secret. For S3 profiles the same pattern applies to AWS credentials: the profile's `credentialsSecret` is mounted at `/run/bloodraven/aws-creds/` (mode `0400`), pointed at by `BLOODRAVEN_AWS_CREDS_DIR`, and the dump script assembles a standard `~/.aws/credentials` file on the fly so the AWS SDK inside `mysqlsh` picks it up natively. ## S3 backup profile ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: image: mysql:9.7 secretName: mysql-credentials # ... sites, dns, sidecar omitted ... backup: image: container-registry.oracle.com/mysql/community-server:9.7 maxLagSecondsForSource: 120 retry: maxAttempts: 3 initialBackoffSeconds: 60 maxBackoffSeconds: 1800 profiles: - name: nightly-s3 retentionPolicy: count: 14 maxAgeDays: 30 minKeep: 1 maxFailedKeep: 10 storage: type: S3 s3: bucket: shipstream-backups prefix: orders region: us-east-1 credentialsSecret: s3-backup-creds dump: threads: 8 bytesPerChunk: "128M" compression: zstd consistent: true schedules: - name: nightly profileName: nightly-s3 schedule: "0 2 * * *" timeZone: "America/Los_Angeles" ``` The referenced `s3-backup-creds` Secret must contain `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `AWS_SESSION_TOKEN` / `AWS_REGION`. The keys are each mounted as a separate file under `/run/bloodraven/aws-creds/` and the dump script stitches them into a standard credentials file at startup. ### TimeZone `schedules[].timeZone` is an IANA zone name (default `Etc/UTC`) that is passed to the `CronJob` spec's `.timeZone` field. This matters because kube-controller-manager's local timezone is environment- dependent and unreliable — two clusters running the same manifest can fire the same cron at different wall-clock times. Pinning the TZ per-schedule makes backup scheduling reproducible no matter where the control plane runs. ### MinIO or other S3-compatible stores Set `storage.s3.endpointURL` to the target endpoint (e.g. `https://minio.internal:9000`). The reconciler maps this onto mysqlsh's `s3EndpointOverride` option. ## PVC-backed profile ```yaml backup: profiles: - name: daily-local retention: 7 storage: type: PVC pvc: storageClassName: fast size: 50Gi dump: compression: zstd ``` If `pvc.claimName` is empty the operator provisions a PVC named `mysql--backup-`. Otherwise the user-managed claim is mounted read-write into the backup Job at `/backups`. ## Structured retention Each profile can use the structured `retentionPolicy` field instead of the shorthand `retention: N` int: | Field | Meaning | Default | | --------------- | --------------------------------------------------------------- | ------- | | `count` | Max successful CRs to keep. 0 disables count-based pruning. | 0 | | `maxAgeDays` | Max age of a successful CR before it's eligible for pruning. | 0 | | `minKeep` | Safety floor — this many newest successful CRs are always kept. | 1 | | `maxFailedKeep` | Max Failed CRs to keep per profile. | 10 | A successful CR is kept if **any** of the enabled checks say "keep": inside the `count` window, inside the `maxAgeDays` window, or within the `minKeep` floor. `minKeep` is the critical safety knob: if every recent attempt after a long outage has failed, it prevents a retention sweep on the next successful run from wiping the last good backup. The legacy shorthand `retention: 7` is still supported and maps to `(count=7, minKeep=1, maxFailedKeep=10)`. ## Artifact cleanup When a `MysqlBackup` CR is deleted (manually or by retention) the reconciler runs a short-lived cleanup `Job` via the `shipstream.io/mysqlbackup` finalizer. The cleanup Job uses the same `mysqlsh` image and credentials layout as the backup Job, but runs `cleanup.py` instead of `dump.py`. It dispatches on the `BLOODRAVEN_STORAGE_TYPE` env var: - **S3** — invokes `util.rmdump(prefix, {s3BucketName: ...})` to recursively delete the dump prefix. "Not found" / "no such key" responses are treated as success. - **PVC** — resolves the dump subdirectory under the mount and `rmtree`s it. Refuses to delete anything outside the mount root. Events: - `ArtifactCleanupStarted` — cleanup Job created. - `ArtifactCleanupSucceeded` — artifact removed or already gone. - `ArtifactCleanupFailed` — cleanup Job failed. The finalizer blocks the CR deletion until either the next attempt succeeds or an operator force-deletes by removing the `shipstream.io/mysqlbackup` finalizer by hand. - `ArtifactCleanupSkipped` — the referenced failover group or profile is gone, so cleanup cannot run. ## Retries for scheduled backups `spec.backup.retry` configures operator-level retries for scheduled CRs that land in `Failed`. This is independent of the Job-level `backoffLimit`: Job backoff retries the container inside a single CR, whereas this retries the whole CR and produces a fresh `MysqlBackup` object with its own Job and attempt counter. ```yaml backup: retry: maxAttempts: 3 # total attempts including the original initialBackoffSeconds: 60 maxBackoffSeconds: 1800 ``` Backoff is exponential: the Nth retry waits `initialBackoffSeconds * 2^(N-1)` seconds from the failed CR's completion time, capped at `maxBackoffSeconds`. `spec.backup.retry` is ignored for ad-hoc CRs — those are operator-initiated and do not participate in automatic retries. ## One-off backups ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlBackup metadata: name: orders-preupgrade namespace: orders spec: failoverGroupRef: name: orders profileName: nightly-s3 triggeredBy: manual ``` Inspect with `kubectl get mysqlbackups`. The `Phase`, `Location`, and `Size` columns reflect the current state. Failed runs leave a condition on `.status.conditions` explaining the reason. ## Restore at bootstrap To recover a lost failover group from a previous backup, create a **new** `MysqlFailoverGroup` that references an existing `MysqlBackup` CR in the same namespace: ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: image: mysql:9.7 secretName: mysql-credentials # ... sites etc. ... backup: profiles: - name: nightly-s3 storage: type: S3 s3: bucket: shipstream-backups prefix: orders credentialsSecret: s3-backup-creds initFromBackup: source: mysqlBackupRef: name: orders-preupgrade ``` ### Dynamic target-site resolution The restore reconciler does not hard-code `spec.sites[0]` as the restore target. Instead it resolves dynamically via `restoreTargetSite`: 1. If `status.activeSite` is set and that site is observed writable (or has no observed state yet on fresh deploys), use it. 2. If `status.activeSite` is set but that site is observed in any other state (read-only, unreachable), **refuse** — the reconciler emits a `RestoreTargetUnavailable` warning event and parks the restore in `Pending`. This prevents accidentally overwriting a recovering standby with a stale dump. 3. Otherwise (true fresh deploy) fall back to `spec.sites[0].Name`. ### Rolled-out Deployment gate Before spawning the load Job, the reconciler waits until the target site's Deployment is fully rolled out: - `generation <= 1` → accept as long as `ReadyReplicas >= 1` (fresh deploy; `ObservedGeneration` may briefly lag). - Otherwise, require `ObservedGeneration >= Generation`, `UpdatedReplicas >= 1`, and `ReadyReplicas >= 1`. This guards against firing a load against a Deployment mid-rolling- update, which would race the ready MySQL container against the terminating one. ### Direct S3 / PVC sources `initFromBackup.source` also accepts a direct S3 URL or a read-only PVC, which is useful when the source backup CR has been garbage- collected: ```yaml initFromBackup: source: s3: bucket: shipstream-backups prefix: orders/orders-preupgrade credentialsSecret: s3-backup-creds ``` Restores are one-shot. If the Job fails, inspect the logs, delete the Job, and the reconciler will rebuild it on the next pass. Restoring into a non-empty data directory is not supported — `util.loadDump()` fails fast with a clear error. ### Per-schema bootstrap (tenant migration pattern) `loadOptions.includeSchemas` and `loadOptions.excludeSchemas` are forwarded to `util.loadDump()` and let you carve a subset of a full-instance dump into a brand-new failover group. This is useful for migrating a single tenant off a shared cluster onto its own dedicated one: ```yaml spec: initFromBackup: source: mysqlBackupRef: name: shared-nightly loadOptions: includeSchemas: ["tenant_42"] ``` Operationally, the migration pattern is: put the tenant into application-level maintenance mode, take a backup of the shared cluster, deploy a new `MysqlFailoverGroup` with `includeSchemas` pointing at the tenant's schema, then flip the app's connection string to the new group. System users are not part of a `util.dumpInstance` dump, so pair the restore with whatever user-provisioning process the new group uses. ## In-place restore (`spec.restoreInPlace`) `initFromBackup` is one-shot and runs before the failover group is considered ready. For rolling an existing, live cluster back to a previous dump (or replaying to a point-in-time) without teardown or rename, use `spec.restoreInPlace`. It is re-triggerable and operates directly against the active primary. Two modes are supported, selected by `loadOptions.includeSchemas`: ### Full-instance in-place restore When `includeSchemas` is empty, the restore wipes every user schema on the primary and reloads from the dump. Choreography: 1. **Preflight** — validates that `status.activeSite` is writable and the `confirm` timestamp is newer than the last consumed one. 2. **Fencing** — the operator strips the `primary` role label on the active pod(s), so the `-primary` Service sheds endpoints and apps see connection errors. The topology manager is frozen: no promotion, auto-clone, or recovery actions fire during the restore. 3. **Restoring** — a Job runs `util.loadDump()` after a `STOP REPLICA; RESET REPLICA ALL; DROP DATABASE ` preflight. `skipBinlog=true` on the load (the replica is about to be re-cloned, so there is no point burning binlog space). Optional PITR replay runs after the dump load, against the same primary. 4. **Resuming** — the primary role label is restored, the topology manager is unfrozen, and the `reclone-site` annotation is set for the peer. The existing reclone machinery then CLONEs the peer from the freshly-restored primary in the background. ```yaml spec: restoreInPlace: confirm: "2026-04-17T14:32:00Z" source: mysqlBackupRef: name: orders-nightly-20260416 pointInTime: stopDatetime: "2026-04-17T14:30:00Z" ``` ### Per-schema in-place restore When `includeSchemas` contains exactly one entry, the restore drops and reloads only that schema. The primary Service stays up — other tenants keep writing — and replication carries the DROP + load through the primary's binlog to the peer. No reclone is scheduled. ```yaml spec: restoreInPlace: confirm: "2026-04-17T14:32:00Z" source: mysqlBackupRef: name: shared-nightly loadOptions: includeSchemas: ["tenant_42"] ``` The caller is responsible for putting the affected tenant into application-level maintenance mode. The operator forces `skipBinlog=false` on the load so the propagation to the peer works. ::warning **Per-schema PITR has a caveat** PITR binlog replay on a per-schema restore adds `mysqlbinlog --database=` to the pipeline. That filter matches on the session's default database at log time, **not** on the schemas a statement actually touches. For well-isolated multi-tenant schemas this is fine. For apps that issue cross-schema statements (`INSERT INTO a.t SELECT ... FROM b.t`) use a full-instance restore instead — the filter can silently drop or misapply events. :: ### The `confirm` timestamp `spec.restoreInPlace.confirm` must be an RFC 3339 timestamp (e.g. `2026-04-17T14:32:00Z`). It is the anti-fat-finger gate: the operator refuses to run the restore unless `confirm` parses and is strictly greater than `status.restoreInPlace.confirmTokenUsed`. On every new run, bump `confirm` to a newer timestamp. A common idiom in automation is to set `confirm: $(date -u +%FT%TZ)` at the moment the user authorizes the restore. This gives programmatic callers a simple "just send now()" pattern while also protecting against replay: an older manifest applied accidentally does not re-trigger a restore. Changing `confirm` while a restore Job is **already running** does not interrupt it. The running Job may be mid-`DROP` or mid-load against the live primary, so the operator lets it finish and records its outcome against the `confirm` it was accepted under (visible as `status.restoreInPlace.confirmTokenUsed`); the new `confirm` then starts its own run once the first reaches a terminal phase. Until then the CR says so: ```text status.restoreInPlace.message: in-flight restore Job mysql-orders-pdx-inplace-restore is running under confirm=2026-04-17T14:32:00Z; confirm=2026-04-17T15:05:00Z starts a new run once it finishes ``` ### Observing the state machine ```bash kubectl get mysqlfailovergroup orders -o jsonpath='{.status.restoreInPlace}' | jq . ``` Phase sequence: `Preflight` → `Fencing` → `Restoring` → `Resuming` → `Succeeded` (or `Failed` at any point). `Failed` is terminal; inspect the Job logs, fix the underlying issue, then bump `confirm` to re-arm. ## Required MySQL privileges Consistent dumps (`dump.consistent: true`, the default) need `BACKUP_ADMIN` on the backup user in addition to standard replication and dump grants: ```sql GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT, SELECT, SHOW VIEW, EVENT, TRIGGER, BACKUP_ADMIN ON *.* TO 'backup'@'%'; ``` Without `BACKUP_ADMIN`, set `dump.consistent: false` on the profile. ### `local_infile` and the restore user Loading a dump uses `LOAD DATA LOCAL INFILE`, which the server rejects unless `@@GLOBAL.local_infile` is `ON`. MySQL 8 ships it `OFF`, so the restore Job turns it on for the duration of the load and puts the prior value back afterwards — the hardened posture outside the load window is unchanged. That requires `SYSTEM_VARIABLES_ADMIN` on the restore user: ```sql GRANT SYSTEM_VARIABLES_ADMIN ON *.* TO 'restore'@'%'; ``` This is checked **before** anything destructive runs. If `local_infile` is `OFF` and the operator cannot enable it, the Job exits with `BLOODRAVEN_LOCAL_INFILE_UNAVAILABLE` and **no schema is dropped** — the load could not have succeeded, so the restore fails with the data still intact rather than dropping the target and then failing to reload it. (A server already running `local_infile=ON` needs no extra grant.) ## Point-in-time recovery (PITR) Bloodraven can continuously archive MySQL binary logs so restores can target an arbitrary `--stop-datetime` on top of any retained full dump, not just the dump's own capture point. For the interaction between PITR and emergency-failover RPO — specifically, when PITR does and does not narrow the data-loss window — see [Durability and RPO → PITR and the RPO window](https://bloodraven.dev/docs/architecture/durability-and-rpo#pitr-and-the-rpo-window). ### How it works Binlog archival runs **inside the existing sidecar** on every MySQL pod. The archiver goroutine: 1. Uses **inotify** on `mysql-bin.index` in the data PVC to detect rotations. A poll-tick every `archivePollInterval` (default 60s) is a belt-and-suspenders safety net in case an inotify event is missed. 2. Gates on `@@read_only` — only the primary uploads. After a failover the new primary takes over within one scan cycle; GTID dedup at restore time makes any brief overlap harmless. 3. For each newly sealed binlog file, extracts timestamp + GTID metadata, uploads the file to the referenced backup profile's storage, and appends an entry to a per-site manifest. Storage layout under the profile's prefix: ```text /binlogs/ ├── manifest-.json ├── manifest-.json # post-failover (or different primary) ├── /mysql-bin.000042 ├── /mysql-bin.000043 └── /mysql-bin.000001 ``` Per-site manifests prevent races between the current primary and a post-failover primary writing to the same object. The restore side reads every manifest in the prefix and merges their entries. ### Enabling PITR Set `spec.backup.pitr` on the failover group and point it at an existing backup profile: ```yaml spec: backup: profiles: - name: primary storage: type: S3 s3: bucket: lion-backups prefix: prod region: us-east-1 credentialsSecret: aws-creds retentionPolicy: count: 14 minKeep: 3 pitr: enabled: true profileName: primary maxBinlogSize: "100M" # MySQL max_binlog_size; controls # rotation cadence and therefore RPO ``` When enabled, the operator: - Injects `max_binlog_size=` into the generated `my.cnf` (default `100M`). Smaller values shorten the RPO gap (unarchived tail on a crashed primary) at the cost of more objects. - Mounts the MySQL data PVC **read-only** into the sidecar container so the archiver can read sealed binlog files. - Mounts the profile's storage credentials (for S3) or the backup PVC (for PVC) into the sidecar. - Rolls the MySQL pods to pick up the new config (the spec hash includes effective PITR values, with defaults normalized, so changing the default in a future release also rolls pods). ### Retention Archived binlogs are pruned to track full-backup retention: once the oldest retained `MysqlBackup` for a profile moves forward, binlogs with `lastEventTime` before that cutoff are no longer needed and get deleted. The mechanism is pull-driven: the sidecar archiver polls the operator's auxiliary HTTP server at `/pitr-cutoff?namespace=X&group=Y&profile=Z` (default once per hour), which returns the minimum `CompletionTime` across retained backups. The archiver then prunes its own manifest and deletes the corresponding objects. The operator pod doesn't need storage credentials — the archiver already has them. ### Restore to a target timestamp Add a `pointInTime` block to `spec.initFromBackup`: ```yaml spec: initFromBackup: source: mysqlBackupRef: name: nightly-2026-04-14 pointInTime: stopDatetime: "2026-04-15T09:30:00Z" # excludeGtids: "server-uuid:42" # optional, skip a bad txn ``` `stopDatetime` accepts RFC 3339 (with or without trailing `Z` or timezone offset) or MySQL's native `YYYY-MM-DD HH:MM:SS` form. The restore Job runs in two phases, one per container: 1. **`pitr-download` init container** (`bloodraven pitr-download` subcommand) — downloads every archived binlog whose `firstEventTime ≤ stopDatetime` from the PITR archive storage into a shared `emptyDir` (`/pitr-binlogs//...`). Uses the AWS SDK v2 for S3 (with paginator), so it doesn't depend on the `aws` CLI existing in the MySQL image. 2. **`mysqlsh` main container** — runs the existing `util.loadDump()` as before, then feeds the downloaded binlogs through `mysqlbinlog --stop-datetime= | mysql --binary-mode`. Server-side GTID dedup skips transactions already applied by the dump load. ### Observability - **Archiver status**: `GET /archiver/status` on the sidecar (`mysql--:8080`) returns `{enabled, primary, lastScanAt, filesArchived, lastError, storageType, manifestPrefix, site}` — handy for `kubectl exec` poking or a Prometheus scraper. - **Job logs**: `BLOODRAVEN_PITR_START`, `BLOODRAVEN_PITR_COMPLETE` / `BLOODRAVEN_PITR_FAILED` / `BLOODRAVEN_PITR_NOOP` from the restore container; one JSON line per archived file from the init container. ### Known limitations - A GTID-set-based file filter would let the restore download fewer binlogs; today it filters on `firstEventTime` only and relies on GTID dedup to cover over-inclusion. ## Metrics The operator emits five Prometheus metrics for backups. All are labelled by `{group, profile}` unless noted. | Metric | Type | Labels | What it measures | | -------------------------------------------------- | --------- | ------------------------ | --------------------------------------------------------------- | | `bloodraven_backup_runs_total` | counter | `group, profile, result` | Terminal backups labelled `success` or `failure`. | | `bloodraven_backup_duration_seconds` | histogram | `group, profile` | Wall-clock duration from Job StartTime to CompletionTime. | | `bloodraven_backup_last_success_timestamp_seconds` | gauge | `group, profile` | Unix timestamp of the most recent successful backup. | | `bloodraven_backup_last_attempt_timestamp_seconds` | gauge | `group, profile` | Unix timestamp of the most recent terminal attempt, any result. | | `bloodraven_backup_last_size_bytes` | gauge | `group, profile` | Size in bytes of the last successful backup artifact. | All five are emitted **exactly once** per terminal reconcile, via a semantic-equality check on the computed next status plus a stable completion timestamp derived from the Job's terminal condition. This means re-reconciling an already-terminal CR does not produce duplicate counter increments or churny gauge updates. Typical alerts: - **Stale backup**: `time() - bloodraven_backup_last_success_timestamp_seconds > 86400` - **Repeated failures**: `increase(bloodraven_backup_runs_total{result="failure"}[24h]) > 3` - **Runaway duration**: `histogram_quantile(0.95, rate(bloodraven_backup_duration_seconds_bucket[1h])) > 3600` ## Monitoring - `kubectl get mysqlbackups -A` — phase, location, size. - `kubectl describe mysqlfailovergroup ` — rollup under `status.backupSchedules[]` (including `lastSuccessfulBackupName`, `lastRetryAttempt`, `nextRetryTime`) and `status.lastBackupTime`. - `kubectl logs job/mysqlbackup-` — live dump progress, plus the final `BLOODRAVEN_DUMP_COMPLETE` sentinel line with the structured `location`, `sizeBytes`, `gtidExecuted`, `binlogFile`, `binlogPos` fields. - Prometheus metrics above. - Events on the `MysqlBackup` CR: `BackupStarted`, `BackupSucceeded`, `BackupFailed`, `InFlightFailover`, `ArtifactCleanup*`. - Events on the `MysqlFailoverGroup` CR: `BackupRetryScheduled`, `BackupPITRNotImplemented`, `RestoreTargetUnavailable`. # S3 Backups ![backup s3 infographic](https://bloodraven.dev/img/infographics/backup-s3.webp) Use S3 or an S3-compatible object store for production backups that must survive cluster or site loss. ## Prerequisites - A bucket, prefix, and region or endpoint URL. - Credentials that can put, get, list, and delete objects under the backup prefix. - `spec.credentials.backupSecret` or operator credentials with backup privileges. - Network egress from backup Jobs to the S3 endpoint. ## Credentials Secret ```bash kubectl create secret generic orders-s3-backup -n orders \ --from-literal=AWS_ACCESS_KEY_ID='replace-with-access-key' \ --from-literal=AWS_SECRET_ACCESS_KEY='replace-with-secret-key' \ --from-literal=AWS_REGION='us-east-1' ``` ## Profile and schedule ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: backup: image: container-registry.oracle.com/mysql/community-server:9.7 maxLagSecondsForSource: 300 profiles: - name: s3-daily storage: type: S3 s3: bucket: orders-mysql-backups prefix: prod/orders region: us-east-1 credentialsSecret: orders-s3-backup dump: threads: 4 compression: zstd retentionPolicy: count: 14 maxAgeDays: 35 minKeep: 2 verification: enabled: true schedule: "30 8 * * *" schedules: - name: nightly profileName: s3-daily schedule: "0 6 * * *" timeZone: Etc/UTC concurrencyPolicy: Forbid ``` The snippet shows only `spec.backup`; keep your existing `sites`, `credentials`, `tls`, and `dns` fields in the same failover group. ## On-demand backup ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlBackup metadata: name: orders-manual-20260427 namespace: orders spec: failoverGroupRef: name: orders profileName: s3-daily triggeredBy: manual ``` Apply and check status: ```bash kubectl apply -f orders-manual-backup.yaml kubectl get mysqlbackup orders-manual-20260427 -n orders -o wide kubectl describe mysqlbackup orders-manual-20260427 -n orders ``` ## List artifacts ```bash aws s3 ls s3://orders-mysql-backups/prod/orders/ --recursive ``` For MinIO or another compatible store: ```bash AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123 \ aws --endpoint-url https://minio.example.com \ s3 ls s3://orders-mysql-backups/prod/orders/ --recursive ``` ## S3-compatible endpoint ```yaml storage: type: S3 s3: bucket: orders-mysql-backups prefix: lab/orders region: us-east-1 endpointURL: https://minio.minio.svc.cluster.local:9000 credentialsSecret: orders-s3-backup ``` ## Restore from S3 ::warning Restore into a new failover group or a deliberately prepared recovery environment first. Confirm the target DNS name cannot receive production writes until validation is complete. :: ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders-restore namespace: orders spec: initFromBackup: source: s3: bucket: orders-mysql-backups prefix: prod/orders/orders-manual-20260427 region: us-east-1 credentialsSecret: orders-s3-backup ``` Include the normal `sites`, `credentials`, and `dns` fields in the recovery failover group. ## Common failures | Failure | Check | | -------------------------------- | ----------------------------------------------------------- | | Bad credentials | `kubectl describe job -n orders`, S3 access denied messages | | Bucket permissions | IAM policy for prefix, list permissions on bucket | | Region mismatch | Secret `AWS_REGION`, profile `region`, bucket region | | Endpoint TLS issues | CA trust for S3-compatible endpoint, endpoint URL scheme | | PVC or node eviction during dump | `spec.backup.stagingVolumeSizeLimit`, pod events | ## Next steps - [Backup Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) - [Backup Encryption](https://bloodraven.dev/docs/backup-and-restore/backup-encryption) - [Backup And Restore Reference](https://bloodraven.dev/docs/backup-and-restore/backup-restore) # PVC Backups ![backup pvc infographic](https://bloodraven.dev/img/infographics/backup-pvc.webp) PVC-backed backups are useful for labs, isolated clusters, and short-retention local copies. They are not a disaster-recovery substitute for off-cluster storage. ## When to use PVC backups Use PVC backups when: - You are testing backup and restore behavior locally. - Your cluster has a durable shared storage system. - You need a short-lived staging copy before exporting elsewhere. Avoid PVC backups when: - Cluster loss must not lose backups. - The StorageClass is node-local. - The PVC cannot be mounted by backup, restore, and cleanup Jobs. ## Operator-managed PVC ```yaml backup: profiles: - name: pvc-local storage: type: PVC pvc: storageClassName: fast-ssd size: 500Gi subPath: orders retentionPolicy: count: 7 minKeep: 2 schedules: - name: nightly profileName: pvc-local schedule: "0 6 * * *" timeZone: Etc/UTC concurrencyPolicy: Forbid ``` ## User-managed PVC ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: orders-backup-store namespace: orders spec: accessModes: - ReadWriteOnce storageClassName: fast-ssd resources: requests: storage: 500Gi --- apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders namespace: orders spec: backup: profiles: - name: pvc-local storage: type: PVC pvc: claimName: orders-backup-store subPath: orders ``` ## On-demand backup ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlBackup metadata: name: orders-pvc-manual-20260427 namespace: orders spec: failoverGroupRef: name: orders profileName: pvc-local triggeredBy: manual ``` ```bash kubectl apply -f orders-pvc-backup.yaml kubectl get mysqlbackup orders-pvc-manual-20260427 -n orders -o wide ``` ## Restore from PVC ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders-restore namespace: orders spec: initFromBackup: source: pvc: claimName: orders-backup-store subPath: orders/orders-pvc-manual-20260427 ``` Include normal `sites`, `credentials`, and `dns` fields in the recovery failover group. ## Sizing and retention - Size the PVC for peak dump size, compression ratio uncertainty, and retention overlap. - Keep at least two successful backups with `retentionPolicy.minKeep`. - Watch PVC usage; a full PVC usually causes backup Jobs to fail after doing expensive work. ## Common failures | Failure | Check | | ------------------ | ---------------------------------------------------------------- | | PVC full | `kubectl describe pvc orders-backup-store -n orders`, Job logs | | Wrong access mode | Whether the backup Job can mount the claim on its scheduled node | | Node-local storage | Restore Job scheduled to a different node cannot see data | | Reclaim policy | Deleting a claim might delete the only backup copy | ## Related pages - [Backup Overview](https://bloodraven.dev/docs/backup-and-restore/backup-overview) - [Backup Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) - [Backup And Restore Reference](https://bloodraven.dev/docs/backup-and-restore/backup-restore) # Backup encryption at rest ![backup encryption infographic](https://bloodraven.dev/img/infographics/backup-encryption.webp) Bloodraven can encrypt every backup artifact — full dump objects written by backup Jobs, archived binlog files uploaded by the sidecar, and their accompanying manifests — using a per-profile passphrase Secret. Encryption is opt-in, enabled by a single new field on a backup profile: ```yaml spec: backup: profiles: - name: nightly-s3 storage: type: S3 s3: bucket: shipstream-backups prefix: orders region: us-east-1 credentialsSecret: s3-backup-creds encryption: algorithm: AES-256-GCM # default; reserved for future algos passphraseSecret: name: orders-backup-passphrase key: passphrase # defaults to "passphrase" ``` Operators who prefer storage-layer protection (AWS SSE-KMS, LUKS on PVCs, bucket default encryption) can leave the `encryption:` block unset and rely on that. For compliance-sensitive workloads — or any setup where the operator team wants to own the keys end-to-end — turn on client-side encryption. ## Why client-side envelope encryption Bucket default SSE-KMS already encrypts objects at rest in AWS, but the data-encryption keys live in the AWS account that hosts the backups. With client-side envelope encryption: - The passphrase never leaves the Kubernetes control plane. Reads happen inside the operator-controlled pods (backup Job, restore Job, sidecar archiver, verification Job, `pitr-download` init container), which mount the Secret as a 0400 file — not an env var. - The bucket / PVC credentials are divorced from the key material. A leaked S3 access key grants access to encrypted objects only; an attacker still needs the passphrase to recover plaintext. - The `mysqlsh` image never handles encryption. Dumps are staged in a local emptyDir by the mysqlsh init container and then encrypted and uploaded by a separate `bloodraven encrypt-upload` main container; on restore the mirror `bloodraven decrypt-download` init container decrypts into the emptyDir before mysqlsh runs `util.loadDump()`. Every encryption / decryption step runs in a container that carries the operator-authored `bloodraven` binary, so MySQL-tooling CVEs can't exfiltrate plaintext if the mysqlsh container is subsequently compromised. ## Wire format Every encrypted object is a self-describing byte stream in the **BRV1** format: ```text Header (32 bytes): [0..3] magic "BRV1" [4] version 0x01 [5] algorithm 0x01 (AES-256-GCM) [6..7] chunk_log2 uint16 BE. Plaintext chunk = 1 << chunk_log2. [8..23] salt 16 bytes random. Fed to HKDF-SHA256. [24..31] nonce_prefix first 8 bytes of each 12-byte GCM nonce. Body: repeated chunks of AES-256-GCM(ciphertext || tag). Nonce layout: nonce[0..7] = nonce_prefix nonce[8..10] = counter (24-bit big-endian, starts at 0) nonce[11] = final_flag (0x00 except on the last chunk: 0x01) ``` - **Key derivation.** `dk = HKDF-SHA256(secret=passphrase, salt, info="bloodraven-backup-encryption-v1", len=32)`. A fresh random salt per file means two encryptions of the same plaintext do not share a data-encryption key. - **Tamper detection.** AES-GCM authenticates every chunk; the final chunk carries a distinct AAD byte so a truncated stream is rejected rather than silently decoded as a shorter plaintext. A post-final trailer of extra bytes is also rejected. - **File size ceiling.** 2^24 chunks × 64 KiB default chunk size = 1 TiB per file. MySQL dump objects almost never hit this (a consistent dump is split across thousands of smaller chunk files by mysqlsh), and single sealed binlog files are capped by MySQL's `max_binlog_size` which users tune to hundreds of MiB. If you ever need bigger single files, raise `chunk_log2` in a future release. Object names (S3 keys, PVC filenames) remain plaintext — the format doesn't try to hide them. Manifest files (`manifest-.json`) are wrapped in the same BRV1 envelope so per-site GTID ranges and file lists do not leak either. The format is implemented by `internal/backupcrypto`. The package test suite covers round-trip, wrong-passphrase rejection, truncation detection, empty plaintext, and a 4-MiB random payload torture test. ## Wiring it up ### 1. Create the passphrase Secret Bloodraven does not generate the passphrase itself — this keeps the Secret's lifecycle and backup policy firmly in the operator's hands. Use a long (≥32 bytes) random string: ```bash head -c 48 /dev/urandom | base64 > passphrase.txt kubectl create secret generic orders-backup-passphrase \ --namespace orders \ --from-file=passphrase=passphrase.txt shred -u passphrase.txt ``` ::warning **Treat the Secret as critical recovery material** The passphrase is the only way to read an encrypted backup. If the Secret is deleted or the value is rotated without re-encrypting the existing artifacts, those artifacts become unrecoverable. Back the Secret up out of band (e.g. 1Password, AWS Secrets Manager, sealed Secrets) with an audit trail; never rely solely on the live Kubernetes Secret. :: ### 2. Reference it from the profile Add `encryption` to the backup profile in the MysqlFailoverGroup spec, as shown at the top of this page. That single field enables: - Client-side encryption of every full dump produced by Jobs from this profile. - Client-side encryption of every sealed binlog the sidecar archiver uploads under this profile's storage (when `spec.backup.pitr` points at the same profile). - Decryption on the matching restore, in-place restore, and verification paths — the operator stamps `MysqlBackup.status.encrypted: true` and the restore builder automatically pulls the same passphrase from the profile. ### 3. Apply; the operator validates On apply the operator checks that the referenced Secret exists and carries the configured key (defaulting to `passphrase`). A misconfiguration emits a warning event: ```text Warning BackupEncryptionInvalid profile "nightly-s3": encryption.passphraseSecret Secret "orders-backup-passphrase" not found in namespace "orders" ``` Subsequent backups still attempt to run — Kubernetes surfaces the missing-Secret error on the Job pod itself — but the event makes the root cause visible in `kubectl describe mysqlfailovergroup`. ## Observing encryption **On each MysqlBackup CR:** ```bash kubectl get mysqlbackup orders-nightly-20260423 -o jsonpath='{.status}' | jq . ``` ```json { "phase": "Succeeded", "location": "s3://shipstream-backups/orders/orders-nightly-20260423/", "sizeBytes": 18874368, "encrypted": true, "encryptionAlgorithm": "AES-256-GCM", ... } ``` **In Job logs** the `bloodraven encrypt-upload` main container emits a BLOODRAVEN\_DUMP\_COMPLETE sentinel identical in shape to the unencrypted flow, plus `encrypted=true` and `algorithm=AES-256-GCM`: ```text BLOODRAVEN_DUMP_COMPLETE location=s3://... sizeBytes=18874368 size=18.0_MiB \ gtidExecuted=abc:1-100 binlogFile=mysql-bin.000042 binlogPos=118 \ ciphertextBytes=19006 files=42 encrypted=true algorithm=AES-256-GCM ``` The operator log-tail parser rehydrates the `encrypted` and `algorithm` tokens into `.status` so dashboards can filter on them. **Object-level verification** — fetch the first 32 bytes of any artifact: ```bash aws s3 cp --range bytes=0-31 s3://shipstream-backups/orders/orders-nightly-20260423/@.json head xxd head | head -1 # 00000000: 4252 5631 0101 0010 ... -- "BRV1" magic + version + algorithm ``` ## Restore paths Encryption is transparent to the restore caller: when the source backup's `.status.encrypted` is true the operator automatically adds a `decrypt-download` init container to the restore Job. The mysqlsh main container reads plaintext from a shared emptyDir and runs `util.loadDump()` as before. The restore Job picks the passphrase Secret in this order: 1. `spec.initFromBackup.decryption.passphraseSecret` if set (recommended for cross-group restores). 2. Otherwise, the profile's own `encryption.passphraseSecret` on the target MysqlFailoverGroup — which is the natural choice when restoring into the same group that took the backup. If neither is available, the restore fails fast with a `RestoreBuildFailed` event: ```text Warning RestoreBuildFailed initFromBackup.source.mysqlBackupRef="orders-preupgrade" is encrypted but no passphrase is available; set initFromBackup.decryption.passphraseSecret or restore the profile's encryption.passphraseSecret ``` `spec.restoreInPlace.decryption` follows the exact same contract. ## Verification `MysqlBackupVerification` automatically decrypts through the same `bloodraven decrypt-download` init container. Verification does not have its own `decryption` field by design: "verify what this profile produced" is exactly the scope where the profile's passphrase is authoritative. If you remove the `encryption` block from the profile between the backup and the verification, the verification fails with a clear error event: ```text Warning VerificationBuildFailed backup "orders-nightly-20260423" is encrypted but profile "nightly-s3" has no encryption.passphraseSecret; restore the encryption field to verify this backup ``` ## PITR binlog archival When `spec.backup.pitr.profileName` references a profile with `encryption` set, the sidecar archiver transparently wraps every `Put` / `PutFile` call through the same BRV1 format. The operator mounts the passphrase Secret onto the sidecar container and sets `BLOODRAVEN_PITR_PASSPHRASE_FILE`; `sidecar.NewArchiveStore` wraps the concrete S3/PVC backend in an `encryptedStore` when that env var is present. On the restore side, the `pitr-download` init container mounts the same passphrase and passes it into its own `NewArchiveStore` call so downloaded binlogs land in the emptyDir decrypted and ready for `mysqlbinlog | mysql` replay. Per-site manifests are encrypted too — the archiver reads and writes them through the wrapped store like any other object. ## Rotation There is no in-place re-encryption today. To rotate the passphrase: 1. Create a **new** Secret with the new passphrase value. 2. Update the profile's `encryption.passphraseSecret.name` to point at the new Secret. 3. Future backups encrypt under the new key; existing backups stay encrypted under the old key. 4. When every backup that was encrypted under the old passphrase has aged out of retention, delete the old Secret. Rotate the restore-side `spec.initFromBackup.decryption` / `spec.restoreInPlace.decryption` references the same way during the transition window. ::tip **Keep the old Secret until retention clears** Deleting the old Secret too early will make the still-retained older backups unrecoverable. The `minKeep` floor on `retentionPolicy` protects against a storm of failed backups after a passphrase mistake; rotation has no equivalent safety net. :: ## Plaintext passthrough on upgrade A cluster that already had backups and then enabled encryption mid-life will have a mix of plaintext and encrypted objects in the same prefix. Both restore and verification paths handle this: the decrypt code detects the missing BRV1 magic and copies the bytes through unchanged. Over time retention will age out the plaintext objects, but you do not need to re-upload or hand-encrypt them. The same behavior covers PITR binlogs that were archived under a plaintext profile before encryption was turned on. ## Threat model boundaries **In scope:** - Confidentiality of backup dump contents and archived binlog data against anyone who can read S3 objects / PVC files but does not hold the passphrase Secret. - Tamper detection via AES-GCM: modified ciphertext fails to decrypt, so a corrupted or attacker-modified object cannot be silently loaded. - Truncation detection via the final-flag nonce byte. **Out of scope:** - Object-name confidentiality (S3 keys and PVC filenames are plaintext by design). - Backup size leakage within one chunk of granularity (64 KiB by default). - Protection of the MysqlBackup CR's metadata — `.status.location`, size, GTID coordinates are plaintext on the Kubernetes API. - Attacks that compromise the operator's ServiceAccount or the passphrase Secret directly. Those are covered by the broader Kubernetes threat model, not by backup encryption. ## Compatibility matrix | Component | Pre-encryption | Encrypted | | ---------------------------------- | ------------------------ | ---------------------------------------------------------------------- | | `spec.backup.profiles[]` | unchanged | adds `encryption:` | | Backup Job shape | single-container mysqlsh | init-container mysqlsh + main-container `bloodraven encrypt-upload` | | Restore Job shape | single-container mysqlsh | adds `decrypt-download` init container + `/restore-decrypted` emptyDir | | Verification Job shape | single-container mysqlsh | adds `decrypt-download` init container | | Sidecar archiver upload path | direct `Put`/`PutFile` | wrapped in `encryptedStore` | | `bloodraven pitr-download` | raw `GetFile` | wrapped in `encryptedStore` | | `MysqlBackup.status` | no change | adds `encrypted`, `encryptionAlgorithm` | | `.status.size`, `.sizeBytes`, GTID | unchanged | reports **plaintext** byte count, not ciphertext | The wire format carries its own magic bytes, so restore / verify paths can always tell encrypted and plaintext apart without out-of- band metadata — a cluster can safely mix the two during an upgrade. # Backup verification ![backup verification infographic](https://bloodraven.dev/img/infographics/backup-verification.webp) An untested backup is a schrödinger backup — it is both good and bad until you try to restore it, which you usually do during the incident that made you reach for it. Bloodraven's verification feature closes that loop: it periodically restores the most recent successful backup into a throwaway MySQL instance to prove the backup is actually loadable, and exports a freshness gauge so alerting can fire on stale or broken verifications. The same CRD supports dump-load verification, optional point-in-time recovery (PITR) binlog replay, and configurable scalar sanity queries. You can start a run from its schedule, a raw CR, or the `kubectl bloodraven verify-backup` command. ## Concepts - **`MysqlBackupVerification` CRD** — one CR per verification run. Created ad-hoc via `kubectl create` or by the scheduled CronJob that the operator renders from a profile's `verification` block. Tracks phase, start/completion times, resolved backup reference, and the names of the ephemeral Pod / Job / PVC used for the run. - **Verification schedule** — `spec.backup.profiles[].verification` on the failover group. When `enabled: true`, the operator materializes a CronJob whose pod runs `bloodraven trigger-verification` to create a `MysqlBackupVerification` CR. - **Ephemeral instance** — each run provisions a dedicated PVC and a dedicated Job pod. The pod boots `mysqld` on the PVC with `--bind-address=127.0.0.1`, then delegates the actual `util.loadDump()` to the shared `restore.py` script against that local instance. No external Service is created; the verification instance is never reachable from outside the pod network namespace. ## What counts as "verified" The default verification contract is **the backup dump loads into an empty mysqld without error**. That covers the failure modes that cause most real disasters — corrupt object store, truncated dump, missing credential rotation, storage class regression, mysqlsh version incompatibility. When you also set `spec.pointInTime` or `spec.sanityCheck` the contract extends to "archived binlogs replay on top of the dump" and "a caller-supplied SELECT returns a result that matches an expectation," respectively. The feature deliberately does not cover: - Logical equivalence with the live primary (a different tool). - Application-level rehearsal of writes / traffic cutover (wishlist #7, #11). Encrypted profiles are supported. The verification Job uses the same `bloodraven decrypt-download` init container as restore, with the profile's `encryption.passphraseSecret`, and loads only the decrypted artifact. See [Backup encryption: Verification](https://bloodraven.dev/docs/backup-and-restore/backup-encryption#verification). ## Scheduling a verification Attach a `verification` block to any `BackupProfile`: ```yaml apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: backup: image: container-registry.oracle.com/mysql/community-server:9.7 profiles: - name: nightly storage: type: S3 s3: bucket: orders-backups prefix: nightly region: us-east-2 credentialsSecret: orders-backups-creds retentionPolicy: count: 14 minKeep: 3 verification: enabled: true schedule: "0 5 * * *" # after the 02:00 backup finishes concurrencyPolicy: Forbid storage: storageClassName: fast-ssd # optional; defaults to cluster default retentionPolicy: keepSuccessful: 30 # keep last 30 Succeeded runs keepFailures: 10 # always keep last 10 Failed runs ``` The operator reconciles this into a `CronJob` named `mysql--verify-` whose pod invokes `bloodraven trigger-verification --group= --profile=`. That subcommand creates a `MysqlBackupVerification` CR, which the verification reconciler picks up. Use `concurrencyPolicy: Forbid` (the default) unless you are confident two verifications of the same profile can run side by side; they cannot share a PVC and `Allow` just stacks failed runs. ## Running a one-off verification Use the kubectl plugin to verify the latest successful backup for a profile: ```bash kubectl bloodraven verify-backup orders --profile nightly -n bloodraven ``` Add `--backup ` to select a historical backup, or `--wait` to block until the run reaches `Succeeded` or `Failed`. See the [kubectl plugin reference](https://bloodraven.dev/docs/operations/kubectl-plugin#verify-backup). You can also create the `MysqlBackupVerification` CR directly: ```bash kubectl create -f - <<'EOF' apiVersion: shipstream.io/v1alpha1 kind: MysqlBackupVerification metadata: generateName: orders-nightly-drill- namespace: bloodraven spec: failoverGroupRef: { name: orders } profileName: nightly # backupRef omitted → verifies the latest Succeeded MysqlBackup # for (group=orders, profile=nightly). EOF ``` To verify a specific historical backup: ```yaml spec: failoverGroupRef: { name: orders } profileName: nightly backupRef: name: orders-nightly-20260413-0200 ``` ### PITR binlog replay When the failover group has `spec.backup.pitr.enabled: true`, a verification can confirm that the archived binlog stream replays cleanly on top of the dump. Add `spec.pointInTime`: ```yaml spec: failoverGroupRef: { name: orders } profileName: nightly pointInTime: mode: latest # one of: none | latest | timestamp ``` - `mode: none` (default) — no replay; equivalent to leaving `pointInTime` unset. - `mode: latest` — replay through the newest archived event. The operator wires a `bloodraven pitr-download` init container that downloads manifests + binlog files into an emptyDir, then `verify.sh` streams them into the ephemeral mysqld via `mysqlbinlog`. - `mode: timestamp` — replay stops just before the first event after `spec.pointInTime.timestamp` (RFC3339). On success, `status.replayedThroughBinlog` is populated with the last applied file, position, and server-clock timestamp. The `bloodraven_backup_verification_replay_lag_seconds` gauge is published as `completionTime − replayedThroughBinlog.timestamp`; alert on a threshold that matches your RPO target. ### Sanity query After the dump loads (and optional PITR replay succeeds), a single scalar-returning SELECT can be evaluated: ```yaml spec: sanityCheck: query: "SELECT COUNT(*) FROM orders.orders WHERE created_at > NOW() - INTERVAL 7 DAY" expect: minRows: 1 # optional; fails if the scalar is below this maxDurationSeconds: 60 ``` The verification records the scalar value in `status.sanityCheck.resultRow` on success, or an error string on failure. `maxDurationSeconds` is enforced with a client-side timeout; exceeding it fails the run with reason `SanityCheckTimeout`, and a query error or below-floor scalar fails with `SanityCheckFailed`. ## Lifecycle A verification CR passes through these phases: | Phase | Meaning | | -------------- | --------------------------------------------------------------------------------------------------------- | | `Pending` | Accepted; finalizer stamped; no ephemeral resources yet. | | `Provisioning` | Ephemeral PVC + derived credentials Secret created. | | `Restoring` | The verification Job is running `verify.sh` + `restore.py`. | | `Checking` | Defined in the API; the current Job runs sanity queries during `Restoring` and does not enter this phase. | | `Cleaning` | Reconciler is deleting ephemeral resources. | | `Succeeded` | Terminal success. Gauge advances; ephemeral resources cleaned up. | | `Failed` | Terminal failure. See `KeepOnFailure` below. | On the happy path the CR goes Pending → Provisioning → Restoring → Succeeded. The ephemeral Pod, Job, Service-if-any, and PVC are deleted after an optional TTL. ## Storage sizing The ephemeral PVC is always dedicated per run and auto-sized from the referenced backup's `status.sizeBytes`: - Explicit `spec.storage.size` wins when set. - Otherwise: `max(10 GiB, ceil(1.5 × backupSizeBytes / 10 GiB) × 10 GiB)`. - The 1.5x multiplier gives MySQL headroom for indexes, temp files, and the occasional tablespace fragmentation that arises during load. Override `spec.storage.storageClassName` when you want a faster class than the cluster default; verification wall-clock time scales roughly linearly with datadir write throughput. ## Failure handling and `keepOnFailure` On `Succeeded`, the ephemeral Pod and PVC are always deleted after `spec.ttlSecondsAfterFinished` (default: immediately). On `Failed`, the default (`spec.keepOnFailure: true`) leaves the Pod and PVC in place so operators can `kubectl exec` into the verification instance, tail its mysqld error log, or attach an interactive mysqlsh session to inspect whatever the load got as far as. Failed verifications are still GC'd by the retention sweep once they drop out of the `keepFailures` window. Set `keepOnFailure: false` on a one-off CR to force full cleanup even on failure — useful for repeated-probe use cases where the failure signal alone is what you care about. ## Concurrency The reconciler refuses to run two verifications against the same `(group, profile)` pair simultaneously. Newer CRs land in `Failed` with the `BlockedByActiveVerification` condition reason. Scheduled runs use `concurrencyPolicy: Forbid` by default so this rejection is rare; it mostly protects against `kubectl create` being issued right as the nightly schedule fires. ## Metrics Verification metrics mirror the `bloodraven_backup_*` family and share the `(group, profile)` label set: | Metric | Type | Meaning | | --------------------------------------------------------------- | --------- | --------------------------------------------------------------------- | | `bloodraven_backup_verified_timestamp_seconds` | Gauge | Unix time of last Succeeded verification — the freshness gauge | | `bloodraven_backup_verification_last_attempt_timestamp_seconds` | Gauge | Unix time of last terminal attempt, success or failure | | `bloodraven_backup_verification_runs_total` | Counter | Terminal attempts by `result="success"` or `result="failure"` | | `bloodraven_backup_verification_duration_seconds` | Histogram | Wall-clock duration from StartTime to CompletionTime | | `bloodraven_backup_verification_replay_lag_seconds` | Gauge | `completionTime − replayedThroughBinlog.timestamp` (Succeeded + PITR) | ### Alerts The obvious alert is staleness on the freshness gauge: ```promql # Verification hasn't Succeeded in more than 48h: time() - bloodraven_backup_verified_timestamp_seconds > 48 * 3600 ``` And a failure-rate alert for fast signal: ```promql # More than 2 failed verifications in 24h: increase(bloodraven_backup_verification_runs_total{result="failure"}[24h]) > 2 ``` ## Inspecting results ```bash kubectl get mysqlbackupverifications -A # Group Profile Phase Started Completed Age # orders nightly Succeeded 12m 2m 12m kubectl describe mysqlbackupverification orders-nightly-20260420 # ... includes status.backupRef, status.durationSeconds, conditions ``` The CR's owner references point back at the `MysqlFailoverGroup`, so deleting the group cascades through verifications along with the rest of its managed state. ## Security - The verification Pod runs with the same hardened pod- and container-level SecurityContext as backup / restore Jobs (RunAsNonRoot, ReadOnlyRootFilesystem, RuntimeDefault seccomp, capability drop ALL). - Credentials flow through files mounted under `/run/bloodraven/mysql-creds`, never environment variables. - The ephemeral mysqld binds `127.0.0.1` only — no Service is created, so the verification instance is not reachable from anywhere outside the Pod's network namespace. ## Known limitations - The `Checking` phase value is defined in the CRD enum but the reconciler does not transition through it at runtime; the sanity query runs inside the same Job container as the restore and the phase goes `Restoring → Succeeded|Failed` directly. Sanity results land on `status.sanityCheck` either way. # Observability Overview ![observability overview infographic](https://bloodraven.dev/img/infographics/observability-overview.webp) Bloodraven exposes operational state through Kubernetes status, Events, structured logs, Prometheus metrics, and Grafana dashboards. ## Signals by audience | Audience | Primary signals | | -------------- | ------------------------------------------------------------------------------- | | On-call | Alerts, runbooks, `MysqlFailoverGroup` conditions, Events | | Platform | Operator health, reconcile errors, RBAC errors, DNS updates | | Database owner | Replication lag, backup freshness, verification freshness, PITR archive lag | | App team | Active-site DNS, connection errors, write success rate, pool reconnect behavior | ## Setup path 1. Enable the metrics Service in the Helm chart. 2. Configure Prometheus scraping with a `ServiceMonitor` or raw scrape config. 3. Install Grafana dashboards. 4. Install the alert package used by your environment. 5. Link each alert to [Runbooks](https://bloodraven.dev/docs/operations/runbooks). 6. Use the [Observability Change Checklist](https://bloodraven.dev/docs/observability/observability-change-checklist) for PRs or releases that change observability signals. ## Pages - [Monitoring Reference](https://bloodraven.dev/docs/observability/monitoring) - [Prometheus Setup](https://bloodraven.dev/docs/observability/monitoring-prometheus) - [Grafana Dashboards](https://bloodraven.dev/docs/observability/monitoring-grafana) - [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map) - [Observability Change Checklist](https://bloodraven.dev/docs/observability/observability-change-checklist) - [Log Schema](https://bloodraven.dev/docs/observability/log-schema) # Monitoring ![monitoring infographic](https://bloodraven.dev/img/infographics/monitoring.webp) Bloodraven exposes Prometheus metrics, a REST status API, and a WebSocket endpoint for real-time status streaming. For the structured-log field set and the stable `msg` vocabulary that downstream log pipelines key off of, see the [Log schema contract](https://bloodraven.dev/docs/observability/log-schema). ::tip **Setup guides** Use [Prometheus Setup](https://bloodraven.dev/docs/observability/monitoring-prometheus) and [Grafana Dashboards](https://bloodraven.dev/docs/observability/monitoring-grafana) for installation. This page is the complete observability reference. :: ## Minimum alert set | Signal | Why it matters | Runbook | | ---------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | Operator down | Reconciliation and DNS updates stop | [Operator unavailable](https://bloodraven.dev/docs/operations/runbooks#operator-unavailable) | | No writable site | Application writes are unavailable | [Emergency manual promotion](https://bloodraven.dev/docs/operations/runbooks#emergency-manual-promotion) | | Split-brain detected | More than one site may accept writes | [Split-brain recovery](https://bloodraven.dev/docs/operations/runbooks#split-brain-recovery) | | Replication lag high | RPO risk is increasing | [Replication lag high](https://bloodraven.dev/docs/operations/troubleshooting#replication-lag-high) | | Divergent transactions | Old primary cannot safely rejoin | [Divergent recovery](https://bloodraven.dev/docs/operations/runbooks#divergent-old-primary-recovery) | | Backup stale | Recovery point is aging | [Failed backup](https://bloodraven.dev/docs/operations/runbooks#failed-backup) | | Verification stale | Backups are not proven restorable | [Backup Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) | | PITR archive lagging | PITR RPO target may be missed | [Backup And Restore](https://bloodraven.dev/docs/backup-and-restore/backup-restore) | | Dragonfly degraded | Cache/session continuity may be unavailable during planned failover | [Playground Dragonfly co-management](https://bloodraven.dev/docs/get-started/playground#dragonfly-co-management) | ## Operator health Check these before debugging individual failover groups: ```bash kubectl rollout status deployment/bloodraven -n bloodraven kubectl logs -n bloodraven deploy/bloodraven kubectl port-forward -n bloodraven deploy/bloodraven 8080:8080 curl http://localhost:8080/metrics | grep '^bloodraven_' ``` ## Prometheus metrics Metrics are served on `:8080/metrics` in standard Prometheus exposition format. ### Available metrics | Metric | Type | Labels | Description | | --------------------------------------------------------------- | --------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bloodraven_poll_latency_seconds` | Histogram | `site` | Duration of each MySQL poll | | `bloodraven_state_transitions_total` | Counter | `site`, `from`, `to` | Count of state transitions per site | | `bloodraven_taint_operations_total` | Counter | `site`, `action` | Count of node taint/untaint operations | | `bloodraven_dns_flips_total` | Counter | `site` | Count of DNSEndpoint updates (DNS record flips) | | `bloodraven_failovers_total` | Counter | `target_site` | Total number of failovers executed. Incremented after successful MySQL promotion. | | `bloodraven_keyring_promotions_blocked_total` | Counter | `namespace`, `group`, `site`, `outcome` | Promotion decisions that skipped or refused a site because it is mid-keyring-rotation. `outcome` is `skipped` or `refused`. Incremented once per decision, not per Event reminder. | | `bloodraven_primary_reassert_total` | Counter | `site` | Times the operator restored writability on the promoted primary after finding it fenced with no writable site remaining (typically the target's own sidecar re-fenced it with a stale lease right after a promotion). A steadily increasing counter means something keeps fencing the primary — check the sidecars' connectivity to the operator's auxiliary Service. | | `bloodraven_websocket_connected_clients` | Gauge | -- | Number of currently connected WebSocket clients | | `bloodraven_replication_lag_seconds` | Gauge | `namespace`, `group`, `site`, `role` | Replication lag in seconds on a follower site. `-1` if lag is NULL (not replicating). Only present for replica sites. `role` is `spec.sites[].role`: `primary-candidate`, `dr-only`, or `read-only`. | | `bloodraven_replication_running` | Gauge | `namespace`, `group`, `site`, `role`, `thread` | Whether a replication thread is running (`1`=yes, `0`=no). Thread is `io` or `sql`. Only present for replica sites. | | `bloodraven_replication_source_state` | Gauge | `namespace`, `group`, `site`, `state` | Direct-source convergence state set for each follower, scoped by failover group. `state` is one of `converged`, `pending`, or `blocked`; one series is `1` and the others are `0`. Cleared for the active primary. | | `bloodraven_site_state` | Gauge | `namespace`, `group`, `site`, `role`, `state` | Current site state as a state-set: `1` for the current state, `0` for others. State is `writable`, `read-only`, `unreachable`, or `unknown`. `role` is `spec.sites[].role`. | | `bloodraven_divergent_transactions` | Gauge | `site` | Number of divergent transactions on a site pending recovery after emergency failover. `0` when healthy. Non-zero means the site has committed transactions that never replicated to the current primary. | | `bloodraven_archiver_upload_failures` | Gauge | `namespace`, `group`, `site` | Cumulative PITR archiver upload failures reported by the per-site sidecar. Monotonic except across sidecar restarts — use `increase()` / `rate()` in dashboards. | | `bloodraven_archiver_last_upload_timestamp_seconds` | Gauge | `namespace`, `group`, `site` | Unix timestamp of the last successful PITR binlog archive per site. `0` if nothing archived yet. | | `bloodraven_archiver_backlog_files` | Gauge | `namespace`, `group`, `site` | Sealed binlogs present in the MySQL index but missing from the archiver manifest at the end of the last scan. `>0` means archival is falling behind. | | `bloodraven_backup_verified_timestamp_seconds` | Gauge | `group`, `profile` | Unix timestamp of the last `Succeeded` `MysqlBackupVerification` per profile. Anchor staleness alerts on this gauge — a fresh `bloodraven_backup_last_success_timestamp_seconds` without a fresh verification means nobody has *proven* the backup can be restored. | | `bloodraven_backup_verification_last_attempt_timestamp_seconds` | Gauge | `group`, `profile` | Unix timestamp of the last terminal verification attempt, regardless of result. Distinguishes "verification never ran" from "verification ran but failed". | | `bloodraven_backup_verification_runs_total` | Counter | `group`, `profile`, `result` | Terminal verification attempts labelled `success` or `failure`. | | `bloodraven_backup_verification_duration_seconds` | Histogram | `group`, `profile` | Wall-clock duration of a verification run. | | `bloodraven_backup_verification_replay_lag_seconds` | Gauge | `group`, `profile` | For verifications with PITR replay enabled: `completionTime − replayedThroughBinlog.timestamp`. A rising value means archived binlogs trail the live primary — alert below your RPO target. | | `bloodraven_restore_duration_seconds` | Histogram | `namespace`, `group`, `restore_kind`, `target_site` | Data-plane duration of successful restore Jobs. `restore_kind` is `init_from_backup` or `in_place`; duration starts at Job `status.startTime` when available and ends when the operator observes terminal success. | | `bloodraven_restore_last_success_timestamp_seconds` | Gauge | `namespace`, `group`, `restore_kind`, `target_site` | Unix timestamp of the last successful restore Job observation. | | `bloodraven_restore_last_source_size_bytes` | Gauge | `namespace`, `group`, `restore_kind`, `target_site` | Source backup artifact size in bytes for the most recent successful restore when known from `MysqlBackup.status.sizeBytes`. Direct S3/PVC restores and unknown sizes clear/omit this series so an older known size is not reported as current. | | `bloodraven_dragonfly_site_up` | Gauge | `group`, `site` | Dragonfly site reachability from the operator's latest `INFO replication` poll (`1` reachable, `0` unreachable). | | `bloodraven_dragonfly_promotions_total` | Counter | `group`, `target_site`, `result` | Dragonfly promotion attempts labelled `success`, `failed`, `skipped`, or `sessions_lost`. `success` is a `REPLTAKEOVER` that preserved continuity. `sessions_lost` is a successful `REPLICAOF NO ONE` fallback — the target is writable but session continuity is not guaranteed. `failed` means the target was not promoted. | | `bloodraven_dragonfly_manager_panics_total` | Counter | `namespace`, `name` | Panics recovered inside the Dragonfly manager polling loop. Any increase should be investigated. | | `bloodraven_license_info` | Gauge | `namespace`, `group`, `organization`, `edition`, `valid` | Constant-1 info series for the license observed on a failover group. `edition` is `community`, `production`, or `organization`. `valid` is the string `true` or `false`. No license is `edition="community"`, `valid="true"`. An invalid token is `edition="community"`, `valid="false"`. An ended update period stays `valid="true"`. One series per group. | | `bloodraven_license_updates_expiry_timestamp_seconds` | Gauge | `namespace`, `group`, `organization`, `edition` | Unix timestamp when the paid update period ends. Absent for community and invalid licenses. A past value is the supported perpetual state. Alert with `(bloodraven_license_updates_expiry_timestamp_seconds - time()) / 86400 < 30`. | Dragonfly replica full-resync state is exposed on the CR as `status.dragonfly.sites[].syncInProgress`, with companion fields `linkStatus`, `lastIOSecondsAgo`, and `ready`. If you export custom-resource status with kube-state-metrics or another CRD status adapter, alert when `syncInProgress` remains true for longer than the expected warm-up window or when it toggles repeatedly outside planned failovers and Dragonfly image rollouts. Frequent full resyncs can spike master latency and reduce planned failover session-preservation confidence. ### Scrape configuration Add a scrape job for the operator: ```yaml # prometheus.yml scrape_configs: - job_name: bloodraven kubernetes_sd_configs: - role: pod namespaces: names: [bloodraven] relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] regex: bloodraven action: keep - source_labels: [__meta_kubernetes_pod_container_port_number] regex: "8080" action: keep ``` Or if using the Prometheus Operator, create a `PodMonitor`: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: bloodraven namespace: bloodraven spec: selector: matchLabels: app.kubernetes.io/name: bloodraven podMetricsEndpoints: - port: metrics interval: 15s ``` ### Alerting rules Recommended alerts: ```yaml groups: - name: bloodraven rules: # Site has no healthy primary - alert: BloodravenNoPrimary expr: | count by (site) ( bloodraven_state_transitions_total{to="writable"} ) == 0 for: 5m labels: severity: critical annotations: summary: "Site {{ $labels.site }} has no writable primary" # High poll latency may indicate network issues - alert: BloodravenHighPollLatency expr: | histogram_quantile(0.99, rate(bloodraven_poll_latency_seconds_bucket[5m])) > 1 for: 5m labels: severity: warning annotations: summary: "Poll latency to {{ $labels.site }} exceeds 1 second (p99)" # Failover occurred - alert: BloodravenFailoverOccurred expr: increase(bloodraven_failovers_total[5m]) > 0 labels: severity: warning annotations: summary: "Failover occurred: {{ $labels.target_site }} promoted as new primary" # Replication lag exceeds threshold on a data-bearing replica. # role=~"primary-candidate|dr-only" pages RPO risk and stays silent # on role: read-only readers, whose lag is designed (they are shed # from mysql--replicas). For promotable replicas only, use # role="primary-candidate". - alert: BloodravenReplicationLagging expr: bloodraven_replication_lag_seconds{role=~"primary-candidate|dr-only"} > 300 for: 2m labels: severity: warning annotations: summary: "Replication lag on {{ $labels.group }}/{{ $labels.site }} ({{ $labels.role }}) is {{ $value }}s" # Divergent transactions after emergency failover - alert: BloodravenDivergentTransactions expr: bloodraven_divergent_transactions > 0 labels: severity: critical annotations: summary: "{{ $labels.site }} has {{ $value }} divergent transactions — trigger reclone to recover" # Replication thread down on a data-bearing replica - alert: BloodravenReplicationDown expr: bloodraven_replication_running{role=~"primary-candidate|dr-only"} == 0 for: 1m labels: severity: critical annotations: summary: "Replication {{ $labels.thread }} thread stopped on {{ $labels.group }}/{{ $labels.site }} ({{ $labels.role }})" # No writable site in a failover group - alert: BloodravenNoWritableSite expr: | max by (namespace, group) (bloodraven_site_state{state="writable"}) == 0 for: 30s labels: severity: critical annotations: summary: "No site is currently writable in {{ $labels.namespace }}/{{ $labels.group }}" # Dragonfly site unavailable; MySQL remains authoritative, but planned # session/cache continuity is at risk. - alert: BloodravenDragonflySiteDown expr: bloodraven_dragonfly_site_up == 0 for: 2m labels: severity: warning annotations: summary: "Dragonfly site {{ $labels.site }} is unreachable for {{ $labels.group }}" # Failed Dragonfly promotion means the MySQL failover may have succeeded # with cache unavailable. - alert: BloodravenDragonflyPromotionFailed expr: increase(bloodraven_dragonfly_promotions_total{result="failed"}[5m]) > 0 labels: severity: warning annotations: summary: "Dragonfly promotion failed for {{ $labels.group }} target {{ $labels.target_site }}" # Empty-master fallback: target is writable but REPLTAKEOVER did not run. - alert: BloodravenDragonflySessionsLost expr: increase(bloodraven_dragonfly_promotions_total{result="sessions_lost"}[5m]) > 0 labels: severity: warning annotations: summary: "Dragonfly promoted {{ $labels.target_site }} via REPLICAOF NO ONE for {{ $labels.group }}; session continuity is not guaranteed" # A recovered panic keeps the manager alive, but it is still a bug signal. - alert: BloodravenDragonflyManagerPanic expr: increase(bloodraven_dragonfly_manager_panics_total[5m]) > 0 labels: severity: critical annotations: summary: "Dragonfly manager recovered a panic for {{ $labels.namespace }}/{{ $labels.name }}" ``` ## Grafana dashboards Bloodraven ships five ready-to-use Grafana dashboards covering every metric the operator publishes. They live in the chart at [`charts/bloodraven/dashboards/`](https://github.com/ShipStream/bloodraven/tree/main/charts/bloodraven/dashboards){rel=""nofollow""} and are also installable as ConfigMaps via the Helm chart. | Dashboard | UID | What it's for | | -------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | **Overview** | `bloodraven-overview` | Health at a glance: writable sites, state timeline, failover activity, replication lag, backup/archiver freshness. Start here. | | **Failover & Topology** | `bloodraven-failover` | Auto and planned failovers, durations, lag-wait histograms, state transitions, DNS flips, node taints, split-brain resolves. | | **Replication & Recovery** | `bloodraven-replication` | Replication lag per site, IO/SQL thread up/down, divergent transactions, reclone ops, poll latency. | | **Backups & Verification** | `bloodraven-backups` | Backup run/failure counts, duration, size, last-success age, verification runs, PITR replay lag. | | **PITR Archiver** | `bloodraven-archiver` | Per-site archiver upload age, backlog files, upload failures. | All five dashboards share a `datasource` variable and cross-link in the top-left corner, so you can roam between them without losing the time range. ### Setup — three paths Pick whichever matches your Grafana install. All three use the same JSON files. #### 1. Helm chart + kube-prometheus-stack (zero-config) If your Grafana is deployed by [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack){rel=""nofollow""} or the upstream [grafana/grafana](https://github.com/grafana/helm-charts/tree/main/charts/grafana){rel=""nofollow""} chart, its dashboard-sidecar watches for ConfigMaps labelled `grafana_dashboard: "1"` by default. Just enable the flag: ```bash helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven --create-namespace \ --set grafanaDashboards.enabled=true ``` The chart renders one ConfigMap per dashboard, the sidecar picks them up within \~30s, and the dashboards appear in a "Bloodraven" folder in Grafana. No restart, no re-login. Common tweaks in `values.yaml`: ```yaml grafanaDashboards: enabled: true # Sidecar is often scoped to the monitoring namespace namespace: monitoring # Change folder name folder: MySQL / Bloodraven # If your sidecar watches a non-default label label: grafana_dashboard labelValue: "1" ``` #### 2. Grafana file-based provisioning If you provision Grafana from disk, copy the JSON files into your provisioning directory and point a provider at them: ```bash # One-time copy (re-run on upgrade to get dashboard updates) kubectl -n monitoring cp -c grafana \ bloodraven/charts/bloodraven/dashboards \ grafana-pod:/var/lib/grafana/dashboards/bloodraven ``` ```yaml # /etc/grafana/provisioning/dashboards/bloodraven.yaml apiVersion: 1 providers: - name: bloodraven folder: Bloodraven type: file disableDeletion: false updateIntervalSeconds: 30 allowUiUpdates: true options: path: /var/lib/grafana/dashboards/bloodraven ``` #### 3. Manual UI import For one-off installs: 1. In Grafana, click **Dashboards → New → Import**. 2. Paste the contents of any file in [`charts/bloodraven/dashboards/`](https://github.com/ShipStream/bloodraven/tree/main/charts/bloodraven/dashboards){rel=""nofollow""}. 3. Pick your Prometheus datasource when prompted. 4. Repeat for each dashboard you want. Cross-dashboard links use the dashboard UID (`bloodraven-overview`, etc.) so they keep working as long as you don't edit the UID on import. ### Prerequisites - Your Prometheus must be scraping the operator — either the ServiceMonitor the chart ships (`--set metrics.serviceMonitor.enabled=true`) or an equivalent scrape job. - The dashboards use standard Prometheus histograms/gauges, so no recording rules are required. Alerts in the next section can sit alongside them. ## Kubernetes Events Bloodraven emits standard Kubernetes Events on `MysqlFailoverGroup` and `MysqlBackup` resources. These events can be forwarded to Slack, PagerDuty, or any webhook endpoint using tools like [Kubewatch](https://github.com/robusta-dev/kubewatch){rel=""nofollow""}, [Argo Events](https://argoproj.github.io/argo-events/){rel=""nofollow""}, or [Event Router](https://github.com/heptiolabs/eventrouter){rel=""nofollow""}. ### Topology and failover | Reason | Type | Description | | ------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `FailoverExecuted` | Normal | A failover completed and a new primary was promoted | | `DataLossDetected` | Warning | Divergent transactions found on old primary after emergency failover | | `RecoveryComplete` | Normal | Old primary recovered and is now replicating | | `RecloneRequested` | Normal | Admin submitted a valid `bloodraven.shipstream.io/reclone-site` annotation; CLONE INSTANCE will start on the next poll | | `RecloneRejected` | Warning | Reclone annotation failed the safety interlock (unknown site, missing/short/mismatched divergent-GTID prefix); annotation was cleared so the admin can retry | | `SplitBrainDetected` | Warning | Both sites are writable (split brain) | | `NoPrimaryDetected` | Warning | Both sites are read-only (no primary). Not emitted when the cause is a mid-rotation promotion refusal — that is `KeyringPromotionRefused`. | | `KeyringPromotionSkipped` | Warning | Emergency failover skipped a site whose `UnsealReason` is `Rotation` and promoted another candidate | | `KeyringPromotionRefused` | Warning | Emergency failover refused: every remaining candidate is mid-keyring-rotation. The group stays without a writable primary until the site is `Sealed`. | | `TotalLossDetected` | Warning | Both sites are unreachable | | `SiteRecovered` | Normal | Degraded condition cleared, topology is healthy | ### Backup lifecycle | Reason | Type | Description | | ------------------ | ------- | -------------------------------------------------- | | `BackupStarted` | Normal | Backup Job created | | `BackupSucceeded` | Normal | Backup completed successfully | | `BackupFailed` | Warning | Backup Job failed | | `InFlightFailover` | Warning | Active site changed while a backup was in progress | ### Backup scheduling | Reason | Type | Description | | ------------------------------------- | ------- | ---------------------------------------------------------------- | | `BackupScheduleInvalid` | Warning | Schedule references an unknown backup profile | | `BackupScheduleServiceAccountMissing` | Warning | Operator ServiceAccount not configured | | `BackupRetryScheduled` | Normal | Retry scheduled for a failed backup | | `BackupPITRNotImplemented` | Warning | `spec.backup.pitr` field has no effect (reserved for future use) | ### Artifact cleanup | Reason | Type | Description | | -------------------------- | ------- | ------------------------------------------------------------- | | `ArtifactCleanupStarted` | Normal | Cleanup Job created for backup artifact | | `ArtifactCleanupSucceeded` | Normal | Artifact removed successfully | | `ArtifactCleanupFailed` | Warning | Cleanup Job failed (finalizer blocks deletion until resolved) | | `ArtifactCleanupSkipped` | Warning | Referenced failover group or profile is gone | ### Restore | Reason | Type | Description | | -------------------------- | ------- | ---------------------------------------------------- | | `RestoreStarted` | Normal | Restore Job created | | `RestoreSucceeded` | Normal | Restore completed successfully | | `RestoreFailed` | Warning | Restore Job failed | | `RestoreTargetUnavailable` | Warning | Active site is not writable or not ready for restore | | `RestoreBuildFailed` | Warning | Failed to build restore Job spec | ### Credentials and secrets | Reason | Type | Description | | --------------------------- | ------- | -------------------------------- | | `CredentialReconcileFailed` | Warning | Failed to reconcile MySQL users | | `SecretNotFound` | Warning | Referenced Secret not found | | `SecretMissingKey` | Warning | Secret is missing a required key | ### Lifecycle | Reason | Type | Description | | ------------------ | ------ | -------------------------------------- | | `GracefulShutdown` | Normal | Graceful shutdown started or completed | ### License | Reason | Type | Description | | ----------------- | ------- | --------------------------------------------------------------------------------------------------------------------------- | | `LicenseVerified` | Normal | A presented token newly verified offline. Also used when a valid token's update period has already ended (perpetual state). | | `LicenseInvalid` | Warning | A presented token failed offline verification. Operator behavior is unchanged. | ### Forwarding events with Kubewatch [Kubewatch](https://github.com/robusta-dev/kubewatch){rel=""nofollow""} can watch for Kubernetes Events and forward them to Slack, PagerDuty, webhooks, and more. Example configuration to forward all Bloodraven events to Slack: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kubewatch data: .kubewatch.yaml: | handler: slack: channel: "#mysql-alerts" resource: event: true namespaces: - bloodraven --- apiVersion: apps/v1 kind: Deployment metadata: name: kubewatch spec: template: spec: containers: - name: kubewatch env: - name: KW_SLACK_TOKEN valueFrom: secretKeyRef: name: kubewatch-secrets key: slack-token ``` ::note Store credentials in a Kubernetes Secret, not in the ConfigMap. The `KW_SLACK_TOKEN` environment variable is read by Kubewatch at startup. :: [Argo Events](https://argoproj.github.io/argo-events/){rel=""nofollow""} and [Event Router](https://github.com/heptiolabs/eventrouter){rel=""nofollow""} are alternatives that offer richer filtering and routing capabilities. ## Status API The operator serves a JSON status API on `:8082`. ### `GET /status` Returns the current state of all failover groups: ```bash curl http://localhost:8082/status ``` ```json { "default/orders": { "activeSite": "iad", "sites": [ { "name": "iad", "state": "writable" }, { "name": "pdx", "state": "read-only" } ], "pollTime": "2025-01-01T00:00:00Z" } } ``` ### `GET /active-site` Returns the active (writable) site for a specific failover group. Used by the sidecar's startup safety net. ```bash curl "http://localhost:8082/active-site?namespace=default&group=orders" ``` ```json { "namespace": "default", "group": "orders", "activeSite": "iad" } ``` | Status | Meaning | | ------ | ----------------------------------------------------------------------------------------------------------- | | `200` | Failover group found. `activeSite` may be `""` if no single writable site exists (first boot, split-brain). | | `400` | Missing `namespace` or `group` query parameter. | | `404` | Failover group not found on this operator instance. | | `503` | Operator has no active topology managers (startup race or non-leader replica). | ### `GET /ws/status` WebSocket endpoint that streams the full topology of each failover group in real time. A message is sent at the end of every operator poll cycle, not only on state transitions, so dashboards can render live counters and health indicators. The default healthy cadence is every 2 seconds per group. After a confirmed persistent site failure, adaptive polling can reduce the stream to one message every 30 seconds until successful probes reset the backoff. Each message is a JSON object with camelCase keys: ```json { "namespace": "default", "group": "orders", "activeSite": "iad", "sites": [ { "name": "iad", "state": "writable", "lastSeen": "2026-04-10T00:00:00Z", "replicating": false }, { "name": "pdx", "state": "read-only", "lastSeen": "2026-04-10T00:00:00Z", "replicating": true, "secondsBehindSource": 0, "gtidExecuted": "3e11fa47-71ca-11e1-9e33-c80aa9429562:1-45839" } ], "lastFailover": "2026-04-09T22:10:00Z", "lastFailoverTarget": "iad", "promotionGtidExecuted": "3e11fa47-71ca-11e1-9e33-c80aa9429562:1-45839", "pollTime": "2026-04-10T00:00:00Z" } ``` When a site has divergent transactions after an emergency failover, the site entry includes recovery fields: ```json { "name": "pdx", "state": "read-only", "lastSeen": "2026-04-10T00:00:00Z", "replicating": false, "recoveryState": "RecoveryBlocked", "divergentGtid": "a1b2c3d4-0000-0000-0000-000000000000:11-15", "divergentTransactionCount": 5 } ``` ```javascript const ws = new WebSocket("ws://localhost:8082/ws/status"); ws.onmessage = (event) => { const msg = JSON.parse(event.data); console.log(`${msg.group}: active=${msg.activeSite}`); }; ``` Both the REST and WebSocket endpoints use camelCase JSON keys, so the same field names work for either consumer. ## Troubleshooting ### Common conditions | Symptom | Likely cause | Investigation | | ------------------------------------------------------ | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Ready=False`, `Degraded=True` | A site is unreachable or replication is broken | Check `status.sites[].state` and MySQL pod logs | | `Ready=False`, both sites `read-only` | No primary exists (may follow a failed failover) | See [Operations: Manual promotion](https://bloodraven.dev/docs/operations/operations#manual-promotion-fallback-operator-unreachable) | | `Ready=False`, both sites `writable` | Split brain | See [Operations: Split brain recovery](https://bloodraven.dev/docs/operations/operations#split-brain-recovery) | | `RecoveryPending=True`, reason `RecoveryInProgress` | Old primary is being reconfigured as a replica | Wait for the condition to clear. If it stays true, check operator logs and `status.sites[].replicating`. See [Failover: Old primary recovery](https://bloodraven.dev/docs/operations/failover#old-primary-recovery). | | `RecoveryPending=True`, reason `DivergentTransactions` | Old primary returned with divergent transactions | Check `status.sites[].divergentGtid`, then trigger a reclone: `kubectl annotate mysqlfailovergroup bloodraven.shipstream.io/reclone-site=`. See [Failover: Old primary recovery](https://bloodraven.dev/docs/operations/failover#old-primary-recovery). | | Repeated failovers | Flapping network or unstable MySQL | Check `bloodraven_state_transitions_total` and consider increasing `failureThreshold` or `failoverCooldown` | | High replication lag | Slow network, heavy write load, or undersized replica | Check `bloodraven_replication_lag_seconds` and MySQL performance metrics | ### Operator logs The operator logs structured JSON. Key fields to filter on: ```bash kubectl logs -n bloodraven deploy/bloodraven | jq 'select(.msg == "failover")' kubectl logs -n bloodraven deploy/bloodraven | jq 'select(.level == "ERROR")' ``` ### Sidecar logs Check the sidecar container for self-fencing events: ```bash IAD_POD=$(kubectl get pod -n orders \ -l app.kubernetes.io/name=mysql,shipstream.io/failover-group=orders,shipstream.io/site=iad \ -o jsonpath='{.items[0].metadata.name}') kubectl logs -n orders "$IAD_POD" -c sidecar | jq 'select(.msg | contains("self-fence"))' ``` # Prometheus Setup ![monitoring prometheus infographic](https://bloodraven.dev/img/infographics/monitoring-prometheus.webp) Bloodraven exposes Prometheus metrics on the operator metrics Service, port `8080` named `metrics`. ## Helm values ```yaml metrics: service: enabled: true serviceMonitor: enabled: true interval: 30s scrapeTimeout: 10s labels: release: kube-prometheus-stack ``` ```bash helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven \ --create-namespace \ --values bloodraven-values.yaml ``` ## ServiceMonitor The chart renders a `ServiceMonitor` when `metrics.serviceMonitor.enabled=true`. It selects the operator metrics Service in the release namespace. If you manage the monitor yourself: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: bloodraven namespace: bloodraven labels: release: kube-prometheus-stack spec: namespaceSelector: matchNames: - bloodraven selector: matchLabels: app.kubernetes.io/name: bloodraven endpoints: - port: metrics interval: 30s scrapeTimeout: 10s ``` ## Plain Prometheus scrape config ```yaml scrape_configs: - job_name: bloodraven kubernetes_sd_configs: - role: endpoints namespaces: names: - bloodraven relabel_configs: - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name] action: keep regex: bloodraven - source_labels: [__meta_kubernetes_endpoint_port_name] action: keep regex: metrics ``` ## Verify targets ```bash kubectl get service -n bloodraven kubectl get servicemonitor -n bloodraven kubectl port-forward -n bloodraven deploy/bloodraven 8080:8080 curl http://localhost:8080/metrics | grep '^bloodraven_' ``` In Prometheus, check **Status > Targets** for the `bloodraven` job or ServiceMonitor-generated target. ## Reader and source-convergence monitoring `bloodraven_replication_source_state{namespace,group,site,state}` is a state-set gauge for every follower. The `namespace` and `group` labels keep identically named sites in different failover groups in separate series. It emits the bounded `state` values `converged`, `pending`, and `blocked`; exactly one is `1` for a follower and the others are `0`. Active primaries have no active source state. Combine it with `bloodraven_replication_running{namespace,group,site,role,thread}` and `bloodraven_replication_lag_seconds{namespace,group,site,role}` when alerting on a reader. Reader failures are deliberately isolated from the failover group's shared `Ready` and `Degraded` conditions. A reader can be unreachable, lagging, or source-blocked while the core candidate/DR topology remains Ready and not Degraded. Alert on reader sites by `role` rather than by site name. For core RPO (promotable and DR replicas, not designed-to-lag readers): ```promql bloodraven_replication_lag_seconds{role=~"primary-candidate|dr-only"} > 30 ``` The issue-142 form that pages only promotable replicas is: ```promql bloodraven_replication_lag_seconds{role="primary-candidate"} > 30 ``` Alert on a specific reader by namespace, group, and site: ```promql bloodraven_replication_source_state{namespace="warehouse",group="orders",site="reader",state!="converged"} == 1 ``` ```promql bloodraven_replication_running{namespace="warehouse",group="orders",site="reader"} == 0 or bloodraven_replication_lag_seconds{namespace="warehouse",group="orders",site="reader"} > 30 ``` To page every `role: read-only` site under the operator, drop the identity labels and keep `{role="read-only"}`. Use a sustained `for` interval appropriate to the normal poll and recovery cadence. A blocked source commonly needs GTID investigation and possibly a cold reclone; a pending source may clear on a later bounded retry. ## License update-period reminder This is a compliance reminder, not an outage. The operator does not change behavior when the update period ends. ```promql (bloodraven_license_updates_expiry_timestamp_seconds - time()) / 86400 < 30 ``` ## Alerts Keep alert rules with your platform monitoring stack. Alert names should link to [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map), and metric details live in [Monitoring Reference](https://bloodraven.dev/docs/observability/monitoring). # Grafana Dashboards ![monitoring grafana infographic](https://bloodraven.dev/img/infographics/monitoring-grafana.webp) The Helm chart ships five dashboards in `charts/bloodraven/dashboards/`. | Dashboard | Audience | Purpose | | ----------- | ----------------- | -------------------------------------------------------------------------- | | Overview | On-call, platform | Writable site, unreachable sites, lag, backup freshness, failover activity | | Failover | On-call | Failover durations, DNS flips, taints, split-brain signals | | Replication | Database owner | Lag, IO/SQL thread state, divergent transactions, reclones | | Backups | Backup owner | Backup success, duration, size, verification freshness | | Archiver | Backup owner | PITR archive freshness, backlog, failures | ## Helm sidecar ConfigMaps Use this path when Grafana has a dashboard sidecar, as in kube-prometheus-stack. ```yaml grafanaDashboards: enabled: true namespace: monitoring label: grafana_dashboard labelValue: "1" folder: Bloodraven ``` ```bash helm upgrade --install bloodraven bloodraven/bloodraven \ --namespace bloodraven \ --values bloodraven-values.yaml ``` ## File provisioning Copy `charts/bloodraven/dashboards/*.json` into Grafana's provisioning path and configure a provider: ```yaml apiVersion: 1 providers: - name: bloodraven folder: Bloodraven type: file allowUiUpdates: true options: path: /var/lib/grafana/dashboards/bloodraven ``` ## Manual import In Grafana, open **Dashboards > New > Import**, paste each JSON file, and select the Prometheus datasource. ## Datasource variable All dashboards use a `datasource` variable. If panels show no data, open dashboard settings, confirm the variable points at the Prometheus datasource scraping Bloodraven, and save the dashboard. ## Troubleshooting | Symptom | Check | | ---------------------- | ----------------------------------------------------------------------- | | No data | Prometheus target up, datasource variable, time range | | Wrong datasource | Dashboard variable default and folder provisioning | | Sidecar did not import | ConfigMap namespace, `grafana_dashboard` label, sidecar namespace scope | | Stale dashboards | ConfigMap updated, Grafana sidecar logs, dashboard `uid` stable | ## Related pages - [Prometheus Setup](https://bloodraven.dev/docs/observability/monitoring-prometheus) - [Monitoring Reference](https://bloodraven.dev/docs/observability/monitoring) - [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map) # Alert To Runbook Map ![alert runbook map infographic](https://bloodraven.dev/img/infographics/alert-runbook-map.webp) Keep this table synchronized with your PrometheusRule package. Alert annotations should link to the matching page or heading. Use the [Observability Change Checklist](https://bloodraven.dev/docs/observability/observability-change-checklist) when adding, removing, or changing alerts, alert annotations, runbook links, dashboard links, or actionable Events. ## Alerts | Alert | Primary runbook | First checks | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `BloodravenOperatorDown` | [Operator unavailable](https://bloodraven.dev/docs/operations/runbooks#operator-unavailable) | Deployment, leader election, logs | | `BloodravenNoWritableSite` | [Emergency manual promotion](https://bloodraven.dev/docs/operations/runbooks#emergency-manual-promotion) or [Total site loss](https://bloodraven.dev/docs/operations/runbooks#total-site-loss) | Active site, pod reachability, fencing | | `BloodravenFailoverOccurred` | [Failover](https://bloodraven.dev/docs/operations/failover) | DNS, app writes, old primary state | | `BloodravenDivergentTransactions` | [Divergent old primary recovery](https://bloodraven.dev/docs/operations/runbooks#divergent-old-primary-recovery) | Events, GTIDs, old primary fenced | | `BloodravenReplicationLagging` | [Replication lag high](https://bloodraven.dev/docs/operations/troubleshooting#replication-lag-high) | lag metric, MySQL replica status | | `BloodravenBackupStale` | [Failed backup](https://bloodraven.dev/docs/operations/runbooks#failed-backup) | latest `MysqlBackup`, Job logs | | `BloodravenBackupVerificationStale` | [Backup Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) | verification CRs, restore logs | | `BloodravenPITRArchiveLagging` | [Backup And Restore](https://bloodraven.dev/docs/backup-and-restore/backup-restore) | archiver metrics, object storage | | `BloodravenDNSUpdateFailed` | [DNS failover stuck](https://bloodraven.dev/docs/operations/runbooks#dns-failover-stuck) | `DNSEndpoint`, external-dns logs | | `BloodravenSplitBrainDetected` | [Split-brain recovery](https://bloodraven.dev/docs/operations/runbooks#split-brain-recovery) | writable sites, app traffic, GTIDs | | `BloodravenDragonflySiteDown` | [Dragonfly degraded](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded) | `status.dragonfly.sites[]`, Dragonfly pods, active Service endpoints | | `BloodravenDragonflyPromotionFailed` | [Dragonfly degraded](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded) | planned-failover Dragonfly status, Events, Redis client impact | | `BloodravenDragonflySessionsLost` | [Dragonfly degraded](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded) | `status.dragonfly.replTakeoverSupported`, `DragonflySessionsLost` Events, image pin | | `BloodravenDragonflyManagerPanic` | [Dragonfly degraded](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded) | operator logs, panic counter, current `status.dragonfly` | | `BloodravenKeyringNotSealed` | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#status-and-monitoring) | `status.encryptionAtRest.sites[].phase` and `.message`, escrow Secret exists, sidecar `/keyring/status` | | `BloodravenKeyringEscrowFailing` | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#disaster-recovery) | `bloodraven_keyring_escrow_pushes_total{outcome="failure"}`, sidecar logs, operator `/keyring/escrow` rejections | | `BloodravenUnencryptedTablespaces` | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#coverage) | `bloodraven_encryption_unencrypted_tablespaces`, whether the group was adopted with partial coverage | | `BloodravenLicenseUpdatesExpiring` | [Licensing](https://bloodraven.dev/docs/licensing) | `bloodraven_license_updates_expiry_timestamp_seconds`. Renewal reminder only — the operator does not gate or change behavior when the timestamp is in the past. | ## Kubernetes Events | Event category | Expected operator action | Runbook | | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Planned failover requested/started/completed | Drain writes, promote target, update DNS | [Planned Failover](https://bloodraven.dev/docs/operations/planned-failover) | | Emergency failover started/completed | Promote best candidate, taint old site, update DNS | [Failover](https://bloodraven.dev/docs/operations/failover) | | Reclone started/completed/failed | Rebuild replica from active primary | [Divergent old primary recovery](https://bloodraven.dev/docs/operations/runbooks#divergent-old-primary-recovery) | | Backup started/completed/failed | Create/update `MysqlBackup` Job and status | [Failed backup](https://bloodraven.dev/docs/operations/runbooks#failed-backup) | | Restore started/completed/failed | Gate bootstrap or restore workflow | [Failed restore](https://bloodraven.dev/docs/operations/runbooks#failed-restore) | | Verification started/completed/failed | Restore backup into ephemeral MySQL | [Backup Verification](https://bloodraven.dev/docs/backup-and-restore/backup-verification) | | DNS update created/failed | Write `DNSEndpoint` for external-dns | [DNS failover stuck](https://bloodraven.dev/docs/operations/runbooks#dns-failover-stuck) | | Split-brain or recovery pending | Fence losers or wait for manual recovery | [Split-brain recovery](https://bloodraven.dev/docs/operations/runbooks#split-brain-recovery) | | Dragonfly promotion/sync/upgrade events | Preserve or restore cache/session continuity while MySQL remains authoritative | [Dragonfly degraded](https://bloodraven.dev/docs/operations/runbooks#dragonfly-degraded) | | `KeyringPhase` / `KeyringUnsealed` | Informational — a site is moving through the keyring lifecycle | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#how-the-keyring-lifecycle-works) | | `KeyringEscrowMissing` / `KeyringEscrowCorrupt` | **Urgent.** A sealed site's only copy of its keys is gone or altered; do not restart that pod | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#disaster-recovery) | | `KeyringNotReadOnly` / `KeyringDigestMismatch` | Investigate the rendering — the site is not actually protected | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#status-and-monitoring) | | `KeyringRotationRefused` / `EncryptionAdoptionRefused` | Follow the supported rotation or adoption procedure | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#rotating-keys) | | `KeyringPromotionSkipped` / `KeyringPromotionRefused` | A rotating site is not a promotion target. If the group has no primary, finish the rotation (there is no cancel); then failover proceeds on the next poll | [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest#rotating-keys) | | `LicenseInvalid` | None required for availability. Check the token string and `kid`. Operator behavior is unchanged. | [Licensing](https://bloodraven.dev/docs/licensing) | | `LicenseVerified` | Informational | [Licensing](https://bloodraven.dev/docs/licensing) | ## Minimum alert annotations Each alert should include: - `summary` with the failing group and namespace. - `description` with immediate user impact. - `runbook_url` pointing to this docs site. - `dashboard_url` pointing to the relevant Grafana dashboard. If an alert intentionally has no runbook or dashboard link, the change must include a specific operational rationale in the pull request. Do not use a bare `N/A`. # Observability Change Checklist Use this checklist for pull requests and releases that add, remove, or change observability signals. It applies to metrics, recording rules, alerts, dashboard panels, Kubernetes Events, structured-log Events, and runbook links. If a pull request does not affect any of those artifact classes, say so in the PR description and skip the rest of the checklist. ## Documentation destinations | Artifact | Required destination | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Prometheus metrics | [Monitoring Reference](https://bloodraven.dev/docs/observability/monitoring) | | Recording rules | [Monitoring Reference](https://bloodraven.dev/docs/observability/monitoring) or the rule package docs | | Alerts | [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map) and the alert package docs | | Dashboard panels | [Grafana Dashboards](https://bloodraven.dev/docs/observability/monitoring-grafana) and shipped JSON under `charts/bloodraven/dashboards/` | | Kubernetes Events | [Monitoring Reference](https://bloodraven.dev/docs/observability/monitoring) for the Event reason registry; [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map) when the Event needs operator action | | Structured-log Events | [Log Schema](https://bloodraven.dev/docs/observability/log-schema) | | Runbook links | [Runbooks](https://bloodraven.dev/docs/operations/runbooks), page-specific runbooks, and [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map) | ## Metrics For each new, renamed, or behavior-changing metric, document: - Metric name. - Prometheus type: counter, gauge, histogram, or summary. - Units, or `1` for dimensionless values. - Full label list. - Stability expectations and any deprecation path for renamed or removed metrics. - Cardinality evidence for every label. Cardinality evidence must include: - Label name. - Expected value domain. - Whether values are bounded or unbounded. - Expected upper bound per `MysqlFailoverGroup`, namespace, cluster, or other relevant scope. - Justification for high-cardinality labels, or a change that removes them. ## Recording rules Complete this section only when the change adds, removes, or changes recording rules. A PR with no recording-rule changes does not need placeholder entries. For each changed rule, document: - Rule name. - Source expression. - Output labels. - Dependency on raw metrics or other recording rules. - Consumer: alert, dashboard panel, runbook, or external SLO tooling. - Migration notes if the rule name or label set changes. ## Alerts For each new, renamed, or behavior-changing alert, document: - Alert name and severity. - Expression and duration. - User impact in plain language. - Required metric or recording-rule dependencies. - Expected false-positive or flap controls. - Runbook mapping in [Alert To Runbook Map](https://bloodraven.dev/docs/observability/alert-runbook-map), or a specific no-runbook rationale. Each alert must include these annotations: - `summary` with the failing group and namespace. - `description` with immediate user impact. - `runbook_url` pointing to this docs site, unless the PR gives a specific no-runbook rationale. - `dashboard_url` pointing to the relevant Grafana dashboard, unless no dashboard exists and the PR explains why. See [Runbook links](https://bloodraven.dev/#runbook-links) for no-runbook rationale requirements. Do not use a bare `N/A`. ## Dashboard panels For each new, removed, or behavior-changing dashboard panel, document: - Dashboard and panel name. - Metrics, recording rules, or log queries used by the panel. - Label filters and template variables. - Expected visual behavior during healthy, degraded, and failing states. - Screenshot, local preview, or other manual verification evidence. Manual verification is required unless the repository has an automated dashboard validator for the changed artifact. If an automated validator exists, link the command and result in the PR. ## Kubernetes Events Kubernetes Events are API-server Events emitted for cluster operators and on-call workflows. They are distinct from structured-log Events. For each changed Kubernetes Event, document: - Event reason and type. - Object the Event is attached to. - Trigger condition. - Expected operator action, if any. - Runbook link when the Event indicates an actionable failure or recovery workflow. ## Structured-log Events Structured-log Events are stable log `msg` strings and fields consumed by downstream log pipelines. They are documented in [Log Schema](https://bloodraven.dev/docs/observability/log-schema) and are a public stability contract. For each changed structured-log Event, document: - Stable `msg` string. - Field names and meanings. - Whether the change adds, removes, or renames fields. - Compatibility impact for log pipelines. - Any matching update to [Log Schema](https://bloodraven.dev/docs/observability/log-schema). ## Runbook links Every alert, actionable Kubernetes Event, and operational dashboard panel should link to a runbook or an explanatory operations page. When a runbook is not required, the PR must state the specific reason, such as: - The signal is informational only and does not require operator action. - The signal is only an implementation detail for another documented alert. - The signal is experimental and hidden from production alerting. ## PR evidence Include the following in the PR description for observability-affecting changes: - Artifact classes changed. - Documentation pages updated. - Cardinality evidence, consumer mapping, and migration notes for metrics and recording rules. - Alert annotation and runbook status. - Dashboard verification evidence. - Kubernetes Event and structured-log Event compatibility notes. # Log schema contract ![log schema infographic](https://bloodraven.dev/img/infographics/log-schema.webp) Bloodraven emits structured JSON logs from both the operator (`bloodraven`) and the per-MySQL sidecar (`bloodraven-sidecar`). This page is the contract that downstream log pipelines key off of: which fields are stable, what the `msg` values are for the events you care about, and what guarantees we make about changing them. If you only need one rule of thumb: **filter on `msg` for the event vocabulary in the [Event reference](https://bloodraven.dev/#event-reference) below — those strings are stable. Everything else is best-effort.** ## Streams Both binaries write to stdout. There are two independent JSON streams; you can tell them apart by the presence of certain keys. | Stream | Source | Identifies as | What's in it | | -------------------------- | -------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Operational (`slog`) | Operator and sidecar | Has `time`, `level`, `msg` | Failover, promotion, bootstrap, recovery, fencing, archiver, sidecar startup, divergence detection — every event a human operator or alerting pipeline would care about | | Controller-runtime (`zap`) | Operator only | Has `ts`, `level`, `msg`, `logger`, `controller`, `controllerKind`, `reconcileID` | Reconcile-loop bookkeeping from controller-runtime: CR fetches, status updates, watch events. Useful for debugging, *not* a stable interface | The contract on this page applies to the **operational stream**. The controller-runtime stream is emitted as-is by upstream `sigs.k8s.io/controller-runtime` and inherits whatever shape that library produces — we don't redefine it. To filter to operational logs in most pipelines, key on the presence of the `time` field (slog) or the absence of the `logger` field (zap). ## Common fields Every record in the operational stream carries: | Field | Type | Description | | ------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `time` | RFC3339Nano timestamp (string) | Event time, normalized to UTC by the binary's slog handler regardless of pod timezone. Always ends in `Z`. | | `level` | string | One of `DEBUG`, `INFO`, `WARN`, `ERROR`. | | `msg` | string | The event identifier. Stable for events listed in the [Event reference](https://bloodraven.dev/#event-reference); may change for ad-hoc debug logs. | Records emitted under a specific failover group also carry: | Field | Type | Description | | ----- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fg` | string | The `MysqlFailoverGroup` namespaced name (`namespace/name`). Present on every operator log scoped to a group. On the sidecar, this is the bare group name passed via `BLOODRAVEN_FAILOVER_GROUP`. | Sidecar records additionally carry: | Field | Type | Description | | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `pod` | string | The pod name (set via `BLOODRAVEN_POD_NAME`). Disambiguates per-replica logs when shipping multiple sites' sidecars to one stream. | ## Levels | Level | When | | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DEBUG` | Per-poll bookkeeping (status no-ops, transient probe errors, archiver tick events). Off by default — the operator's slog handler is set to `INFO`. | | `INFO` | State changes the operator deliberately took: failover, promotion, bootstrap, recovery, sidecar lifecycle. **Most of the event vocabulary lives here.** | | `WARN` | Degraded but not fatal: a single retry, a peer briefly unreachable, a non-critical operation that failed (connection kill, taint patch). The operator continues. | | `ERROR` | Operator-affecting failure: failover failed, self-fence triggered, status update rejected by the API server, CronJob-pod startup validation failed. **Always paired with an `error` field** (a string carrying either the underlying error or, for validation failures, a description of what was missing). | `DEBUG` records may appear or disappear without notice. `INFO`/`WARN`/`ERROR` `msg` strings listed below are stable. ## Field naming convention - Keys are `camelCase`. Common keys: `site`, `fg`, `error`, `peer`, `count`, `source`, `donor`, `recipient`. - Site identifiers (`site`, `oldPrimary`, `newPrimary`, `promotedSite`, `donor`, `recipient`, `activeSite`, `authoritativeActiveSite`) all carry the bare site name as defined in `spec.sites[].name`. - GTID fields (`promotionGtid`, `divergentGtid`, `oldPrimaryGtid`, `newPrimaryGtid`, `followerGtid`, `activeGtid`) carry MySQL GTID-set strings exactly as MySQL returns them — never parsed or canonicalised. - Counts (`count`, `divergentTransactions`, `attempt`, `maxRetries`) are JSON numbers, not strings. - Durations (`leaseTimeout`, `pollInterval`, `delay`, `timeout`) are emitted by `slog`'s default `time.Duration` rendering — currently a string like `"30s"`. Treat as opaque if you need to parse, prefer the metric of the same name. ## Event reference This is the stable vocabulary. `msg` strings here will not change without a deprecation note in the release notes. ### Failover The four events that trace one failover, in order: | Level | `msg` | Fields | Fired when | | ----- | ----------------------------------------------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INFO | `initiating failover` | `candidate`, `oldPrimary`, `fg` | Operator has chosen a promotion target and is about to run the promotion sequence. DNS flips only after promotion succeeds and the target is verified writable. | | INFO | `failover complete` | `promotedSite`, `promotionGtid`, `fg` | `Execute` finished: candidate is writable. `promotionGtid` is the candidate's `gtid_executed` snapshot taken just before clearing `super_read_only` — the upper bound on data that survived. | | INFO | `promotion confirmed: site is writable` | `site`, `fg` | Next poll observes the promoted site is `writable`. The internal post-failover guard clears here. | | ERROR | `failover failed` | `error`, `fg` | The promotion sequence returned an error. The operator does not retry automatically; the next eligible state-transition tick will re-evaluate. | | ERROR | `promotion succeeded but writable confirmation failed; DNS not flipped` | `site`, `error`, `fg` | `Execute` returned successfully but the promoted site did not report writable within the confirmation window. DNS is **not** flipped and no failover state is recorded — the promotion is treated as unconfirmed and re-evaluated on the next tick. | | ERROR | `DNS flip failed after successful promotion` | `site`, `error`, `fg` | Promotion **and** writable confirmation both succeeded, so the failover state (cooldown, split-brain target, `promotionGtidExecuted`) and `bloodraven_failovers_total` are already recorded; only the DNS update failed. `bloodraven_dns_flips_total` is left unincremented. The poll loop reconciles DNS against the current active site (see `DNS reconciled to active site` below), so a transient failure such as an RBAC denial self-heals once the write is permitted again — MySQL has already promoted regardless. | | WARN | `DNS reconcile failed` | `site`, `target`, `hostname`, `error`, `fg` | The poll-driven DNS reconcile tried to point the record at the current active site and the write was rejected. Logged once per failing episode, not once per poll: while the failure persists the retry continues silently (DEBUG `DNS reconcile still failing`) and MySQL is unaffected. | | INFO | `DNS reconciled to active site` | `site`, `target`, `hostname`, `fg` | The DNS record diverged from the desired `spec.dns` (target IP, hostname, or TTL) and was repaired — a promotion-time flip that had failed, a hostname rename, a record left stale by an operator restart, or an out-of-band edit. `bloodraven_dns_flips_total{site}` increments here, and only when the A-record target IP actually changed. No promotion is re-run and MySQL is not touched. | Supporting events emitted inside `Execute`: | Level | `msg` | Fields | | ----- | --------------------------------------------------------------------- | ------------- | | INFO | `fenced old primary with super_read_only=ON` | `fg` | | WARN | `failed to fence old primary (may be unreachable)` | `error`, `fg` | | INFO | `killed app connections on old primary` | `count`, `fg` | | WARN | `failed to kill app connections on old primary` | `error`, `fg` | | INFO | `relay log drain complete` | `fg` | | WARN | `relay log drain did not complete cleanly, proceeding with promotion` | `error`, `fg` | ### Divergence and recovery Fired after an emergency failover when the operator inspects the returning old primary. | Level | `msg` | Fields | Notes | | ----- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INFO | `initiating old primary recovery` | `oldPrimary`, `newPrimary`, `fg` | Recovery sequence starting. | | INFO | `no GTID divergence, auto-recovering old primary as replica` | `site`, `fg` | Old primary's GTID set is a subset of the new primary's — safe to attach as replica. | | WARN | `divergence detected` | `site`, `divergentTransactions`, `divergentGtid`, `oldPrimaryGtid`, `newPrimaryGtid`, `fg` | Old primary has committed transactions the new primary never saw. Operator does not auto-recover — the admin must reclone the site or replay the divergent set onto the new primary (containment then auto-rejoins). Emitted when divergence is first detected and again whenever the periodic re-verification finds the set CHANGED — an unchanged report is not re-logged each cycle. Mirrored by the `bloodraven_divergent_transactions` gauge and the `DataLossDetected` Kubernetes Event. | | INFO | `old primary recovery complete` | `site`, `source`, `fg` | Old primary is now replicating from the new primary. `source` is the new primary's host. | | ERROR | `old primary recovery failed` | `site`, `error`, `fg` | One step of the recovery sequence (fence / GTID query / `CHANGE REPLICATION SOURCE` / `START REPLICA`) returned an error. | | INFO | `recovery state cleared (site is now replicating)` | `site`, `fg` | The site is a healthy, source-converged replica again, so its recovery marker is dropped and `bloodraven_divergent_transactions{site}` returns to 0. The normal terminal state for both `RecoveryInProgress` and an externally resolved `RecoveryBlocked`. | | INFO | `recovery state cleared (site is writable)` | `site`, `fg` | The site is writable, so it is no longer "recovering as a replica". A `RecoveryBlocked` report is **not** dropped merely because the site turned writable — a rogue-writable site keeps its live divergence report (split-brain fencing re-fences it, and the periodic re-verification refreshes the report). The one case a blocked report clears here is when the site is the operator's own failover target *and* the unique writable primary, which makes the report definitionally stale. | ### Replication source convergence After topology changes, Bloodraven verifies that every follower replicates directly from the uniquely confirmed active primary. These events cover candidate, `dr-only`, and `read-only` followers; they are separate from the old-primary recovery events above. | Level | `msg` | Fields | Notes | | ----- | ----------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | INFO | `replication source convergence started` | `site`, `activeSite`, `currentSource`, `expectedSource`, `fg` | A follower needs a source or thread-state correction and passed the initial mutation gates. | | INFO | `replication source convergence complete` | `site`, `source`, `fg` | The canonical source is the active primary and both replication threads are running. | | WARN | `replication source convergence blocked` | `site`, `activeSite`, `stage`, `followerGtid`, `activeGtid`, `fg` | GTID containment failed before or after stopping replication. No source change is issued. | | ERROR | `replication source convergence failed` | `site`, `activeSite`, `stage`, `error`, `fg` | A bounded source mutation or verification attempt failed. The next poll can retry safely. | Stable `stage` values include `pre-stop-gtid`, `post-stop-gtid`, `stop`, `change-source`, `start`, and `verify`. Use the status `sourceConvergenceState` and `sourceConvergenceReason` for current state; use these logs for the detailed failure and GTID evidence. ### Bootstrap and reclone `starting bootstrap` is the single canonical event for "we are about to clone a replica". The `source` field disambiguates *why*: | `source` value | Meaning | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fresh-deploy` | Initial bootstrap of a new failover group; donor is the seed site. | | `auto-clone` | Operator detected an empty replica during steady-state and is recovering it without an admin trigger. | | `reclone` | Admin set the `bloodraven.shipstream.io/reclone-site=` annotation, the safety interlock passed, and the operator is wiping the named site. **This is the `reclone-started` event.** | | Level | `msg` | Fields | | ----- | ------------------------------------------------------------------------------- | ------------------------------------------------- | | INFO | `starting bootstrap` | `source`, `donor`, `recipient`, `donorHost`, `fg` | | INFO | `cloning from primary` | `donor`, `fg` | | INFO | `clone completed successfully` | `replica`, `fg` | | INFO | `setting up replication` | `source`, `fg` | | INFO | `replication started successfully` | `source`, `fg` | | INFO | `bootstrap completed successfully` | `source`, `fg` | | ERROR | `bootstrap failed` | `source`, `error`, `fg` | | INFO | `clone returned expected connection drop, waiting for restart` | `error`, `fg` | | INFO | `replica already has primary data (prior clone detected), skipping clone phase` | `fg` | A reclone-only narrative is therefore: filter `msg="starting bootstrap" AND source="reclone"` for the trigger event, then watch for `bootstrap completed successfully` (`source="reclone"`) or `bootstrap failed` (`source="reclone"`). ### State transitions Every per-site state change emits one record. Use this to replay the topology timeline. | Level | `msg` | Fields | | ----- | ------------------ | -------------------------- | | INFO | `state transition` | `site`, `from`, `to`, `fg` | `from` and `to` values: `unknown`, `unreachable`, `read-only`, `writable`. Mirrored by the `bloodraven_state_transitions_total` counter. ### Topology decisions | Level | `msg` | Fields | Notes | | ----- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | WARN | `ALERT` | `message`, `fg` | A cross-site `EvalCrossSite` action returned an alert string (split brain, no primary, total loss). The same conditions emit `SplitBrainDetected` / `NoPrimaryDetected` / `TotalLossDetected` Kubernetes Events. | | WARN | `split-brain auto-resolve: fencing non-preferred site per spec.splitBrainPolicy.sitePriorities` | (context) | Opt-in `splitBrainPolicy` is fencing the lower-priority site. The poll-driven retry of a persistent split-brain re-emits this at WARN at most every 30s (DEBUG in between). | | INFO | `fencing returning old primary (split brain after failover)` | `site`, `fg` | A second site came back writable after a failover; the operator is fencing every writable site except the one holding live primary authority. Emitted at INFO on the first fence of a given split-brain and at DEBUG on the poll-driven retries in between. | | ERROR | `failed to fence returning old primary` | `site`, `error`, `fg` | The `super_read_only=ON` write above failed. The next poll retries. DEBUG on retries, matching the INFO/DEBUG pairing of the fence itself. | | INFO | `post-fence application connections evicted` | `site`, `count`, `fg` | Operator-side recovery removed one or more sessions that survived the sidecar's best-effort fence eviction. It retries on later topology polls until an empty pass or timeout, without blocking the poll loop between passes. | | WARN | `post-fence application connection drain retry failed` | `site`, `error`, `fg` | A recovery drain pass could not enumerate or kill sessions. The operator retries within `spec.connectionDrainTimeout`. | | INFO | `post-fence application connection drain complete` | `site`, `fg` | An empty eviction pass proved no application sessions remain. Old-primary recovery may now complete. | | WARN | `post-fence application connection drain timed out` | `site`, `timeout`, `fg` | The dedicated drain budget expired. Recovery proceeds because `super_read_only=ON` still prevents writes; a surviving session is limited to stale reads. | | WARN | `re-asserting fenced promoted primary: no site is writable and the last failover target is GTID-complete; restoring writability` | `site`, `fg` | The last failover target was found fenced (read-only) with every site reachable and nothing writable — typically its own sidecar re-fenced it with a stale lease right after a promotion. The operator restores writability on the target. Mirrored by `bloodraven_primary_reassert_total`. Rate-limited to once per `failoverCooldown`. | | WARN | `primary re-assert refused: peer has transactions the target lacks — divergence needs human review` | `site`, `peerGtid`, `targetGtid`, `fg` | The no-writable-site wedge was detected but restoring the last failover target would abandon peer transactions. The group stays read-only until an admin resolves the divergence. | | WARN | `primary re-assert refused: target no longer contains the recorded promotion GTID set (wiped or restored since promotion?)` | `site`, `promotionGtid`, `targetGtid`, `fg` | The failover history no longer describes the target's data lineage; the operator will not restore writability automatically. | | WARN | `primary re-assert refused: recorded promotion GTID set failed to parse — status corrupted or manually edited?` | `site`, `promotionGtid`, `error`, `fg` | `status.promotionGtidExecuted` is non-empty but malformed. The operator wrote this value from MySQL itself, so a parse failure means corruption or manual tampering — the re-assert safety argument depends on it, so the operator refuses. | | INFO | `failover blocked by anti-flap cooldown` | (context) | A failover decision was deferred because `failoverCooldown` has not elapsed since the last one. | | WARN | `skipping promotion candidate: site is mid-keyring-rotation` | `site`, `unsealReason`, `fg` | Emergency failover chose another sealed candidate. Mirrored by Warning Event `KeyringPromotionSkipped` and `bloodraven_keyring_promotions_blocked_total{outcome="skipped"}`. | | WARN | `promotion refused: every remaining candidate is mid-keyring-rotation` | `sites`, `fg` | No sealed candidate remained. The group stays without a writable primary until rotation finishes. Mirrored by Warning Event `KeyringPromotionRefused`, `Degraded` reason `NoPrimary` with an `UnsealReason=Rotation` message, and `bloodraven_keyring_promotions_blocked_total{outcome="refused"}`. | | WARN | `promotion refused: split-brain winner is mid-keyring-rotation` | `site`, `unsealReason`, `fg` | Split-brain auto-resolve would have picked a rotating site. Losers are not fenced. | | WARN | `promotion refused: candidate is mid-keyring-rotation` | `site`, `unsealReason`, `fg` | Last-line check refused `failover.Execute` on a rotating candidate. | | WARN | `primary re-assert refused: target is mid-keyring-rotation` | `site`, `unsealReason`, `fg` | The last failover target is mid-rotation; writability is not restored. | | WARN | `ignored out-of-order local failover record` | `target`, `lastFailover`, `currentTarget`, `currentLastFailover`, `fg` | Two local promotion paths reported out of timestamp order; the newer in-process anti-flap pair remains authoritative. | | ERROR | `out-of-band anti-flap state write failed; retrying every poll` | `fg`, `target`, `lastFailover`, `error` | The annotation write was rejected. The newest record stays pending and is retried on each poll. | | INFO | `cross-site action deferred: in-place restore in progress` | `fg` | Decisions are paused while `restoreInPlace` runs. | | INFO | `cross-site action deferred: planned failover in progress` | `fg` | Decisions are paused while a planned-failover annotation is being processed. | ### Sidecar fencing The per-MySQL sidecar emits these in its operational stream. `SELF-FENCING:` is a stable prefix — `msg` strings that begin with it indicate the sidecar wrote `super_read_only=ON` to its local MySQL without operator instruction. | Level | `msg` | Fields | Notes | | ----- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ERROR | `SELF-FENCING: topology mismatch — operator-authoritative active site disagrees with our site, setting super_read_only=ON` | `site`, `authoritativeActiveSite`, `observedAt`, `pod` | The operator (or a peer relaying the operator's view) reports a different active site than this sidecar is on. Fired even when the operator is reachable. | | ERROR | `SELF-FENCING: Bloodraven and every peer unreachable beyond lease timeout, setting super_read_only=ON` | `bloodravenLastOk`, `latestPeerOk`, `peers`, `leaseTimeout`, `pod` | Backstop rule: nothing is reachable, so we can't be sure we're still primary. | | INFO | `SELF-FENCING: killed app connections` | `count`, `pod` | Application sessions evicted after fencing succeeded. Server-internal threads are spared, so a fenced replica keeps applying from the authoritative primary. | | WARN | `SELF-FENCING: failed to kill connections after fencing` | `error`, `count`, `pod` | The fence write landed but eviction was incomplete. `count` is how many sessions were killed — `0` when the process-list query itself failed, so no session was ever enumerated. `error` joins every cause that applied: the listing query failed, iteration ended early, some rows would not scan, some `KILL`s were refused, or the fence hit its 20s deadline. `super_read_only=ON` still holds, so surviving sessions cannot write. The sidecar does not retry because it cannot safely distinguish a pooled operator promotion session. After promotion, the operator drains the fenced former primary until a pass finds no sessions or `spec.connectionDrainTimeout` expires. The sidecar's `/status` reports `self_fenced`, scoped to the current sidecar process. | | WARN | `SELF-FENCING: super_read_only write failed but the fence is in place; skipping connection eviction` | `error`, `pod` | The `SET GLOBAL` returned an error, but a follow-up `@@super_read_only` read proved the write landed. The fence counts as established: `/status` reports `self_fenced=true` and `SELF-FENCED` follows. Eviction is skipped because the fence budget is spent; operator-side recovery performs the bounded follow-up drain. | | ERROR | `SELF-FENCING: super_read_only write outcome is unconfirmed; will probe again` | `error`, `probeError`, `pod` | Both the `SET GLOBAL` result and its independent confirmation probe were unavailable. The monitor retains a process-local pending marker and retries `@@super_read_only` on later ticks instead of getting stuck behind `read_only=ON` with `self_fenced=false`. | | WARN | `fencing: could not resolve pending super_read_only write` | `error`, `pod` | A later tick still could not read `@@super_read_only`; the pending result remains armed for another tick. | | WARN | `SELF-FENCING: previously unconfirmed super_read_only write is now confirmed` | `pod` | A later tick proved the ambiguous write established `super_read_only=ON`; `/status` now reports `self_fenced=true`. | | ERROR | `SELF-FENCING FAILED: could not set super_read_only` | `error`, `pod` | The fence write failed and a follow-up `@@super_read_only` read confirmed the instance was not super-fenced. The sidecar retries on the next tick. | | WARN | `fencing: could not confirm whether the super_read_only write landed` | `error`, `pod` | The immediate `@@super_read_only` confirmation read failed. The pending outcome is retained and retried on later ticks. | | ERROR | `SELF-FENCED: super_read_only=ON has been set, only Bloodraven can restore` | `pod` | Final status; the sidecar will not unfence on its own. The next operator promotion clears it. | | INFO | `fencing: MySQL is writable after prior self-fence; rearming monitor` | `pod` | An actor with SUPER privileges (the operator, per the restore contract) made MySQL writable again after a self-fence. The monitor re-arms with a fresh lease window — it will not re-fence until a full `leaseTimeout` passes with the operator and every peer unreachable again. | | INFO | `fencing: adopted active-site view from peer` | `peer`, `activeSite`, `observedAt`, `pod` | Peer sidecar relayed a fresher view than what this sidecar had cached. Drives the topology-mismatch rule. | Safety-net events (sidecar startup): | Level | `msg` | Fields | | ----- | ----------------------------------------------------------------- | --------------------------- | | INFO | `safety net: set super_read_only=ON as precaution on startup` | `pod` | | INFO | `safety net: this is the active site, clearing super_read_only` | `site`, `pod` | | INFO | `safety net: confirmed standby site, staying fenced` | `site`, `activeSite`, `pod` | | INFO | `safety net: no active site reported by operator, staying fenced` | `pod` | | WARN | `safety net: could not query active site, staying fenced` | `error`, `pod` | | ERROR | `safety net: failed to clear super_read_only on active site` | `error`, `pod` | ### PITR archiver Emitted by the sidecar's `BinlogArchiver`. | Level | `msg` | Fields | | ----- | -------------------------- | ---------------------------------------------------------------- | | INFO | `binlog archiver starting` | `storageType`, `binlogDir`, `binlogIndex`, `pollInterval`, `pod` | | INFO | `archived sealed binlogs` | `count`, `pod` | | INFO | `retention sweep complete` | (sweep stats), `pod` | | WARN | `archive binlog` | `file`, `error`, `pod` | | WARN | `retention: delete object` | `key`, `error`, `pod` | Per-upload success/failure is also reflected in the `bloodraven_archiver_upload_failures` and `bloodraven_archiver_last_upload_timestamp_seconds` metrics — prefer those for alerting. ### Data-at-rest encryption Emitted when `spec.encryptionAtRest.enabled=true`. See [Data-at-rest encryption](https://bloodraven.dev/docs/configuration/encryption-at-rest) for the lifecycle these events trace. Operator events: | Level | `msg` | Fields | Notes | | ----- | ---------------------------------------------------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INFO | `keyring phase transition` | `site`, `from`, `to`, `reason`, `version`, `fg` | The per-site keyring lifecycle advanced. The normal sequence is `Pending → Unsealed → Escrowed → Sealed`. Any transition **to** `Unsealed` from `Sealed` means the site is deliberately running with a writable keyring (clone or rotation). | | INFO | `keyring escrowed` | `namespace`, `group`, `site`, `version`, `secret`, `digest` | The operator accepted a sidecar escrow push and stored a new immutable Secret version. Also emitted by the sidecar with `site`, `version`, `secret`, `digest`, `pod`. | | INFO | `refusing to enable encryption at rest on a live failover group` | `activeSite`, `fg` | `spec.encryptionAtRest.enabled` was turned on for a group that is already serving. Rendering is unchanged; existing tablespaces would have stayed plaintext. | | WARN | `keyring escrow rejected` | `namespace`, `group`, `site`, `reason` | A push to `/keyring/escrow` failed authentication. The response to the caller is deliberately vague; the reason is only here. | | WARN | `keyring escrow digest mismatch` | `namespace`, `group`, `site`, `claimed`, `computed` | An escrow push carried a digest that did not match its payload. Nothing was stored. | | ERROR | `keyring escrow: store failed` | `namespace`, `group`, `site`, `error` | The operator could not write the escrow Secret. The site stays unsealed and the sidecar keeps retrying. | Sidecar events: | Level | `msg` | Fields | Notes | | ----- | ----------------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | INFO | `keyring agent enabled` | `path`, `escrowArmed`, `rotate`, `pod` | Sidecar startup. `escrowArmed=false` means this pod renders a sealed keyring and only reports its digest. | | INFO | `rotated innodb master key` | `site`, `pod` | `ALTER INSTANCE ROTATE INNODB MASTER KEY` succeeded. | | INFO | `encrypted the mysql system tablespace` | `site`, `pod` | `ALTER TABLESPACE mysql ENCRYPTION='Y'` succeeded on the writable site. | | WARN | `keyring escrow push failed, will retry` | `error`, `site`, `digest`, `pod` | The push was rejected or the operator echoed a digest that did not match. Escrow is retried until it succeeds; the site is not sealed in the meantime. | | WARN | `could not encrypt the mysql system tablespace` | `error`, `site`, `pod` | Retried on the next tick. | | ERROR | `keyring rotation failed` | `error`, `site`, `pod` | The rotation statement failed. The agent still escrows whatever is on disk, because a failed rotation may already have written a key. | Kubernetes Event reasons emitted on the `MysqlFailoverGroup`: | Reason | Type | When | | --------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `KeyringPhase` | Normal / Warning | Any keyring phase transition. Warning when the new phase is `Failed`. | | `KeyringUnsealed` | Normal | A site was unsealed so a `CLONE INSTANCE` can rewrap tablespace keys. | | `KeyringRotationRefused` | Warning | A rotation was requested for the active primary, or while an ordered update or planned failover was in flight. | | `KeyringPromotionSkipped` | Warning | Emergency failover skipped a mid-rotation site and promoted another sealed candidate. | | `KeyringPromotionRefused` | Warning | Emergency failover refused to promote because every remaining candidate is mid-keyring-rotation. Finish the rotation, then the next poll can promote. | | `KeyringEscrowMissing` | Warning | A sealed site's escrow Secret no longer exists. **That site cannot restart.** See the DR runbook. | | `KeyringEscrowCorrupt` | Warning | A sealed site's escrow Secret no longer hashes to the recorded digest. | | `KeyringDigestMismatch` | Warning | A pod rendered sealed is running a keyring that does not match its escrow; the operator refuses to call it sealed. | | `KeyringNotReadOnly` | Warning | MySQL reports a writable keyring on a site the operator considers sealed. | | `EncryptionAdoptionRefused` | Warning | Encryption was enabled on a group that is already serving. | ### Dragonfly Bloodraven optionally co-manages per-site Dragonfly instances and emits the following events when `spec.dragonfly.enabled=true`. Mirrored by the `bloodraven_dragonfly_site_up` gauge and the `bloodraven_dragonfly_promotions_total{result}` counter, plus the matching `Dragonfly*` Kubernetes Events on the `MysqlFailoverGroup`. | Level | `msg` | Fields | Notes | | ----- | ------------------------------------------------------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INFO | `dragonfly: configured replica` | `site`, `host`, `port`, `fg` | Operator issued `REPLICAOF` against a non-active site to align it with the active master. | | WARN | `dragonfly: stale master on non-active site` | `site`, `active`, `fg` | A site reports `role=master` but is not the active site. Auto-rejoin is attempted only when the stale instance has `connected_slaves=0` AND `master_repl_offset=0` (provably never accepted writes); otherwise the stale master is shed from the active Service via the traffic-label gate and left for human intervention. | | INFO | `stale-master reconfigure: REPLICAOF applied` | `site`, `host`, `port`, `fg` | Auto-rejoin succeeded: the stale master is now linked as a replica of the active master. | | WARN | `stale-master reconfigure: REPLICAOF failed` | `site`, `host`, `port`, `error`, `fg` | Auto-rejoin attempt failed; the next tick retries. | | INFO | `client-kill: evicted clients from old master` | `site`, `fg` | After a planned-failover Dragonfly promotion, the operator issued `CLIENT KILL TYPE NORMAL` against the demoted source so application clients reconnect through the active Service. | | INFO | `dragonfly/mysql active-site drift: promoting Dragonfly replica to match MySQL` | `oldSource`, `target`, `mysqlActiveSite`, `fg` | MySQL active site and Dragonfly master diverged; the manager is promoting the synced Dragonfly replica on the MySQL active site. | | INFO | `dragonfly-only emergency: active master unreachable; promoting replica` | `oldSource`, `target`, `fg` | Dragonfly master failed without a MySQL failover; the manager is promoting the single healthy replica and leaving MySQL `status.activeSite` unchanged. | | INFO | `dragonfly emergency: REPLTAKEOVER succeeded` | `site`, `fg` | After an emergency MySQL failover, Dragonfly was promoted with sessions preserved. | | WARN | `dragonfly emergency: REPLTAKEOVER failed; falling back` | `site`, `error`, `fg` | Emergency promote could not preserve sessions; falling back to `REPLICAOF NO ONE`. | | INFO | `dragonfly emergency: target promoted via REPLICAOF NO ONE (sessions lost)` | `site`, `fg` | Emergency promote completed via the empty-master fallback after REPLTAKEOVER failed. Mirrored by Warning Events `DragonflyPromotionCompleted` and `DragonflySessionsLost`, and by `bloodraven_dragonfly_promotions_total{result="sessions_lost"}`. | | WARN | `dragonfly: REPLTAKEOVER not advertised` | `fg` | Capability probe found REPLTAKEOVER missing from the running command table. | | INFO | `dragonfly: REPLTAKEOVER capability restored` | `fg` | A previously missing REPLTAKEOVER command is now advertised (typically after an image upgrade). | | WARN | `dragonfly emergency: REPLICAOF NO ONE failed` | `site`, `error`, `fg` | Both promotion paths failed; cache is unavailable. MySQL emergency failover was not affected. | | WARN | `dragonfly emergency: target unreachable; skipping promotion` | `site`, `error`, `fg` | Bounded budget expired before the operator could reach the target. | Kubernetes Event reasons emitted on the `MysqlFailoverGroup` (visible via `kubectl describe`): | Reason | When | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DragonflyPromotionStarted` | Planned-failover state machine entered `PromotingDragonfly`. | | `DragonflyPromotionCompleted` | Dragonfly target was promoted (planned or emergency), including the emergency `REPLICAOF NO ONE` fallback. | | `DragonflyPromotionFailed` | Promotion command failed; behavior depends on `spec.dragonfly.plannedFailover.onSyncTimeout` (planned) or is best-effort (emergency). | | `DragonflySessionsLost` | Emergency promotion fell back to `REPLICAOF NO ONE` after `REPLTAKEOVER` failed. Session continuity is not guaranteed. Emitted in addition to `DragonflyPromotionCompleted`. | | `DragonflyReplTakeoverUnsupported` | Runtime probe found `REPLTAKEOVER` missing from the running Dragonfly command table. | | `DragonflyReplTakeoverSupported` | Runtime probe transitioned from unsupported to supported (typically after an image upgrade). | | `DragonflyStaleMasterDetected` | A non-active site reports master role. Logged + dedup'd in 5-minute windows. Auto-rejoin is attempted in `reconcileReplication` when `connected_slaves=0 AND master_repl_offset=0`. | | `DragonflyOldSiteReconfigured` | A stale master passed the auto-rejoin gate and was attached as a replica of the active master via `REPLICAOF`. | | `DragonflySyncTimeout` | `WaitingForDragonflySync` exhausted `spec.dragonfly.plannedFailover.maxSyncWait`. | | `DragonflyUpgradeStarted` | Snapshot-restore Dragonfly upgrade annotation was accepted and `status.dragonfly.upgrade` was initialized. | | `DragonflyUpgradeRejected` | Snapshot-restore upgrade request was invalid or another coordinated operation was running. | | `DragonflyUpgradeSnapshotStarted` | Active Dragonfly traffic was shed and the operator is about to issue `SAVE`. | | `DragonflyUpgradeSnapshotCompleted` | `SAVE` completed against the active Dragonfly master using `spec.dragonfly.snapshot.dir`. | | `DragonflyUpgradeCompleted` | Active and replica Dragonfly pods are on the target image, active traffic is restored, and replicas are linked. | | `DragonflyUpgradeFailed` | Snapshot-restore upgrade reached a terminal failure; the operator best-effort restored active traffic. | ### License Offline license observation. These events never change failover or reconcile behavior. `updatesUntil` is an RFC3339 UTC timestamp string. Per-group records (always include `fg` as `namespace/name`): | Level | `msg` | Fields | Fired when | | ----- | ------------------------------ | ------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | INFO | `license verified` | `fg`, `organization`, `edition`, `updatesUntil`, `issuedFor`, `kid` | A group's resolved token newly verified. | | INFO | `license updates period ended` | `fg`, `organization`, `edition`, `updatesUntil` | A valid token's `updatesUntil` is in the past. Perpetual state, not an error. | | WARN | `license invalid` | `fg`, `reason`, `kid` | A presented token failed verification. The group continues as Community. | Process-level records for the operator default (no `fg`): | Level | `msg` | Fields | Fired when | | ----- | --------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------- | | INFO | `operator license verified` | `organization`, `edition`, `updatesUntil`, `issuedFor`, `kid` | `--license` / `BLOODRAVEN_LICENSE` verified at process start. | | INFO | `operator license updates period ended` | `organization`, `edition`, `updatesUntil` | Operator default token is valid but the update period has ended. | | WARN | `operator license invalid` | `reason`, `kid` | Operator default token failed verification. Process still starts. | Kubernetes Event reasons emitted on the `MysqlFailoverGroup`: | Reason | Type | When | | ----------------- | ------- | ------------------------------------------------------------------------------ | | `LicenseVerified` | Normal | A presented token newly verified, including the perpetual ended-updates state. | | `LicenseInvalid` | Warning | A presented token failed offline verification. Advisory only. | ### Lifecycle | Level | `msg` | Fields | | ----- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | INFO | `starting bloodraven manager` | (none) | | INFO | `starting auxiliary HTTP server` | `addr` | | INFO | `topology manager runner starting` | (none) | | INFO | `starting topology manager` | `fg` | | INFO | `topology manager stopped` | `fg` | | INFO | `stopping topology manager` | `fg` | | INFO | `config changed, restarting topology manager` | `fg` | | INFO | `restored lastFailoverTarget from CR status` | `fg`, `target` | | INFO | `restored lastFailover from CR status` | `fg`, `lastFailover` | | WARN | `restored lastFailoverTarget from out-of-band annotations` | `fg`, `target`, `statusTarget` | | WARN | `restored lastFailover from out-of-band annotations` | `fg`, `lastFailover`, `statusLastFailover` | | ERROR | `out-of-band anti-flap annotation unreadable; falling back to CR status` | `fg`, `error` | | ERROR | `CR status anti-flap state unreadable; ignoring unsafe copy` | `fg`, `error` | | INFO | `starting graceful shutdown` | `fg` | | INFO | `CR deleted — DNSEndpoint will be garbage-collected` | (none) | | INFO | `sidecar image version differs from the operator` | `operatorImage`, `sidecarImage`, `operatorTag`, `sidecarTag` | | INFO | `sidecar starting` | `listenAddr`, `peerAddresses`, `bloodravenAddress`, `leaseTimeout`, `peerCheckInterval`, `site`, `namespace`, `fg`, `pod` | | INFO | `sidecar stopped` | `pod` | | INFO | `received signal, shutting down` | `signal`, `pod` | Kubernetes Event reasons emitted on the `MysqlFailoverGroup`: | Reason | Type | When | | -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SidecarVersionSkew` | Warning | `spec.sidecarImage` is tagged with a different release than the running operator. The two halves share a rendering contract, so a mismatch can break the sidecar silently — on a `spec.tls` group an older sidecar cannot reach MySQL at all. Advisory only: the operator reports skew but never blocks on it, because a re-tagged or locally built image is indistinguishable from a real mismatch. | ## Stability commitments | What | Stability | | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `msg` strings listed in the [Event reference](https://bloodraven.dev/#event-reference) | **Stable.** Changes go through a deprecation note in the release notes. | | Field names listed alongside a stable `msg` | **Stable.** New fields may be added to existing events; existing fields will not be renamed or removed without a deprecation note. | | Field value shapes (strings, numbers, durations) | **Stable** for the values listed. GTIDs are passed through verbatim from MySQL — their shape is whatever MySQL emits. | | `time`, `level`, `msg` field names themselves | **Stable.** Tied to `log/slog` defaults. | | `DEBUG`-level records | **Unstable.** May appear, disappear, or change shape without notice. Disabled by default. | | Ad-hoc `INFO`/`WARN`/`ERROR` records *not* listed above (e.g. retry warnings, transient probe errors) | **Best-effort.** Field set is intended to be useful but not contractual. Don't build alerts that key on the exact `msg` string. | | Controller-runtime (`zap`) stream | **Inherited from upstream.** Bloodraven does not redefine this stream's shape. | ## Pipeline integration tips ### Filtering operational vs. controller-runtime Most aggregators (Loki, Elasticsearch, Vector) let you split streams by JSON shape. A reliable predicate: ```jsonpath $.time && $.msg // operational (slog) $.ts && $.logger // controller-runtime (zap) ``` ### Per-event alerts Because every key event has a stable `msg`, pipeline alerts can be expressed as exact-match filters rather than fragile regexes. Examples for Loki: ```logql # Failover started {app="bloodraven"} | json | msg = "initiating failover" # Failover failed (escalate) {app="bloodraven"} | json | level = "ERROR" and msg = "failover failed" # Divergence requires manual reclone {app="bloodraven"} | json | msg = "divergence detected" # Reclone triggered (track who/what asked for it via fg + recipient) {app="bloodraven"} | json | msg = "starting bootstrap" and source = "reclone" # Sidecar self-fenced — page on this {app="bloodraven-sidecar"} | json | msg =~ "^SELF-FENCING:" ``` ### Correlating with metrics and Kubernetes Events Several stable log events are mirrored by other observable signals — when one fires, the others fire too: | Log event | Metric | Kubernetes Event | | ------------------------------- | ------------------------------------------------------ | ----------------------------- | | `failover complete` | `bloodraven_failovers_total{target_site}` | `FailoverExecuted` | | `divergence detected` | `bloodraven_divergent_transactions{site}` > 0 | `DataLossDetected` | | `old primary recovery complete` | `bloodraven_divergent_transactions{site}` returns to 0 | `RecoveryComplete` | | `state transition` | `bloodraven_state_transitions_total{site, from, to}` | (none — too noisy for events) | Prefer metrics for alert thresholds and Kubernetes Events for human notification routing; logs are richest for forensics and timeline reconstruction. ### Useful structured fields to index If your pipeline supports indexing specific fields, the high-value ones are: - `fg` — partitions everything by failover group - `site` (and `oldPrimary` / `newPrimary` / `promotedSite` / `donor` / `recipient`) — for per-site timelines - `level` — for severity routing - `source` — for bootstrap/reclone disambiguation - `error` — full error string from the operator's `error` chain # Example Manifests ![examples infographic](https://bloodraven.dev/img/infographics/examples.webp) Complete example manifests live in the repository `examples/` directory so they can be reviewed and validated separately from prose. | Example | File | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Minimal failover group with two candidates and a placement-field-free reader | [`examples/minimal-failovergroup.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/minimal-failovergroup.yaml){rel=""nofollow""} | | Per-role credentials | [`examples/per-role-credentials.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/per-role-credentials.yaml){rel=""nofollow""} | | TLS-enabled failover group | [`examples/tls-enabled-failovergroup.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/tls-enabled-failovergroup.yaml){rel=""nofollow""} | | S3 backups with scheduled verification | [`examples/s3-backups-with-verification.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/s3-backups-with-verification.yaml){rel=""nofollow""} | | PVC backups | [`examples/pvc-backups.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/pvc-backups.yaml){rel=""nofollow""} | | Production Helm values | [`examples/production-values.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/production-values.yaml){rel=""nofollow""} | | NetworkPolicy | [`examples/networkpolicy.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/networkpolicy.yaml){rel=""nofollow""} | | PrometheusRule | [`examples/prometheusrule.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/prometheusrule.yaml){rel=""nofollow""} | | Argo CD Application | [`examples/argocd-application.yaml`](https://github.com/ShipStream/bloodraven/blob/main/examples/argocd-application.yaml){rel=""nofollow""} | Use these files as starting points, not as production-ready manifests. Replace placeholder passwords, image tags, bucket names, DNS names, node labels, and StorageClasses before applying them. # Docs Maintenance Use this page before releases and when changing CRDs, Helm values, metrics, dashboards, Events, or logs. ## Ownership | Area | Owner | | ------------------------------- | --------------------------------- | | Platform install and chart docs | Platform team | | CRD reference and behavior docs | Operator maintainers | | Backup and restore docs | Database/infra team | | App integration docs | Application platform team | | Security model | Security and operator maintainers | | Runbooks | On-call owners | ## Standard task page structure 1. Overview. 2. Prerequisites. 3. Example manifest or command. 4. Apply. 5. Verify. 6. Troubleshoot. 7. Next steps. ## Standard reference page structure 1. What this page covers. 2. Concepts. 3. Field or behavior reference. 4. Examples. 5. Failure modes. 6. Related pages. ## Terminology Use these terms consistently: | Preferred | Avoid | | ---------------------------- | -------------------------------------------------------- | | primary | master | | replica | slave | | site | datacenter unless specifically referring to a datacenter | | active site | current writable site | | standby site or replica site | passive promotion candidate | | dr-only site | passive site that is never auto-promoted | | failover group | cluster, unless discussing Kubernetes clusters | ## Example names | Item | Default example | | ------------------ | ----------------------- | | Failover group | `orders` | | Namespace | `orders` | | Sites | `iad`, `pdx` | | DNS name | `orders.az.example.com` | | Operator namespace | `bloodraven` | | StorageClass | `fast-ssd` | ## Release checklist - Regenerate API docs if fields changed. - Check Helm values references. - Check dashboard list and UIDs. - Check metrics list and alert names. - Check Event and log schema docs. - Check upgrade policy. - Check known limitations. - Build this site locally with `npm run build` from `site/`, then confirm `/llms.txt` and `/llms-full.txt` are served by the built output. - Review examples against the release tag. ## Publishing automation This page lives in the Docus site under `site/`, which deploys to Railway and serves {rel=""nofollow""}. | Gate | Workflow | What it proves | | --------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pull requests and `main` pushes | `.github/workflows/ci.yml` | The site under `site/` builds and `llms-full.txt` includes every page under `site/content/docs/`. | | Release tags | `.github/workflows/release.yml` | The release candidate passes the same site build and `llms-full.txt` coverage check before images and charts publish. | | Nightly and manual runs | `.github/workflows/docs-link-check.yml` | The public site under `https://bloodraven.dev/` has no same-site broken links. | | Pushes to `main` touching `site/` | `.github/workflows/deploy-site.yml` | The Docus site under `site/` is uploaded to the Railway service `bloodraven-site`, which builds it and serves {rel=""nofollow""}. | ## Validation checklist - YAML examples use `shipstream.io/v1alpha1`. - Required CRD fields are present. - Deprecated fields are labelled as deprecated. - Command blocks use explicit namespaces. - Example passwords are clearly placeholders. - Helm values match `charts/bloodraven/values.yaml`. - Internal links build cleanly. # Bloodraven - MySQL failover for Kubernetes ::home-hero --- links: - label: Get started to: /docs/get-started/getting-started variant: primary icon: i-lucide-arrow-right - label: Try the playground to: /docs/get-started/playground variant: outline icon: i-lucide-flask-conical - label: GitHub to: https://github.com/ShipStream/bloodraven target: _blank variant: ghost icon: i-simple-icons-github accent: a survival instinct. description: Bloodraven runs MySQL async replication failover groups across sites. It owns detection, fencing, promotion, DNS steering, clone bootstrap and backups — so losing an entire site is a status change, not an incident. kicker: Kubernetes operator for MySQL title: Give your MySQL topology --- :: ::home-stats --- items: - value: ~6s label: to declare a site unreachable (3 failed 2s polls) - value: RPO 0 label: on a planned switchover, by construction - value: 47+ label: scripted chaos scenarios run in CI - value: "1" label: CRD, one controller, one reconcile loop --- :: ::home-paths --- paths: - n: "01" meta: Docs label: Get started title: Install the operator description: Prerequisites, the Helm chart, and your first two-site MysqlFailoverGroup — with the checks that tell you it actually converged. to: /docs/get-started/getting-started - n: "02" meta: ~2 min label: Playground title: Break it on a laptop description: One script stands up two MySQL sites, a live dashboard, a counter app and a chaos menu on k3d, kind or minikube. to: /docs/get-started/playground - n: "03" meta: Free label: Course title: Train the rotation description: Bloodraven in Production — seven hands-on units on a real cluster your team breaks on purpose, with graded projects. to: /courses/ target: _blank kicker: Three ways in source-label: github.com/ShipStream/bloodraven source-to: https://github.com/ShipStream/bloodraven --- :: ::home-ask-ai --- questions: - How does Bloodraven decide which site to promote? - Show me a minimal two-site MysqlFailoverGroup. - What is my RPO when the primary site dies? - How do I schedule encrypted backups to S3? - A site is stuck in RecoveryBlocked — what now? - Which alerts should page someone at 3am? accent: Just ask. description: The assistant is grounded in this entire documentation set — the CRD reference, every runbook, the failure-mode matrix and the log schema. Ask it in your own words and it answers with links to the exact page. kicker: Sixty pages of docs, one prompt llms: bloodraven.dev/llms-full.txt title: Don't read 60 pages. --- :: ::home-doctor --- accent: AI diagnostics in the CLI. command: npx skills add shipstream/bloodraven description: Give your AI coding assistant deep diagnostic vision over live MySQLFailoverGroup clusters. bloodraven-doctor performs 60-second non-destructive triage, detects GTID divergence, audits keyring status, and crafts safe step-by-step remediation plans. kicker: Day-2+ Diagnostic Copilot title: Meet Bloodraven Doctor. --- :: ::home-safety --- cards: - token: DNS title: Automated DNS failover description: Be genuinely geo-redundant without a global load balancer in front of your database. Bloodraven writes an external-dns DNSEndpoint with your hostname and TTL, and moves it to the promoted site as part of the promotion sequence — no cross-region LB bill, no anycast VIP, no proxy hop on every query. detail: external-dns DNSEndpoint · your hostname · your TTL to: /docs/architecture/multi-site - token: SAFE title: Split-brain safe, and tested that way description: Two sites never both accept writes. The operator fences the old primary, and each sidecar self-fences with super_read_only when it can reach neither the operator nor its peer. GTID divergence is detected, reported in divergentGtid, and blocks an unsafe rejoin until a human decides. detail: operator fencing · sidecar super_read_only · divergentGtid to: /docs/operations/network-partitions - token: ROLL title: Zero-downtime updates description: OrderedUpdate upgrades the standby first, fails over to it, then upgrades the old active — the direction MySQL's rolling-upgrade contract requires. Node taints and the placement contract keep application workloads on the same site as the writable MySQL as the primary moves. detail: OrderedUpdate · standby first · placement contract to: /docs/operations/upgrade-policy id: features kicker: Detection · fencing · promotion lede: The operator watches every site on a two-second poll, decides with a small documented state machine, and executes the same promotion sequence every time. title: Losing a site title-two: should be boring. --- :: ::home-data --- backup-points: - S3 or PVC — object storage for durability, PVC for labs - Scheduled and on-demand — a MysqlBackup whenever you need one - Structured retention — plus exponential-backoff retries on failure - Point-in-time recovery — binlog archiving between full dumps - Encrypted artifacts — application-level, independent of the store - Prometheus metrics — every run, every failure, every sweep - Automatic cleanup — deleting the object removes its artifacts - Verification — load the dump into a throwaway MySQL and prove it - initFromBackup — bootstrap a brand-new failover group from a dump cards: - token: CLONE title: Clone-based bootstrap description: New replicas seed themselves with MySQL's clone plugin and pick up replication with GTID auto-positioning. No mysqldump window, no snapshot juggling, no manual data transfer. The same path repairs a site whose PVC was lost, or one you deliberately reclone after divergence. to: /docs/operations/failover#failover-sequence code: |- CLONE INSTANCE FROM 'replicator'@'orders-iad:3306' … CHANGE REPLICATION SOURCE TO SOURCE_AUTO_POSITION = 1; START REPLICA; - token: AES title: Data-at-rest encryption, for free description: InnoDB tablespace encryption on ordinary PVCs, using the GPL keyring component that ships with MySQL Community Edition. No Oracle Enterprise licence, no encrypted CSI storage class, and the master key never lands on the data PVC or a worker-node disk. Rotation, sealing and escrow are handled by the operator. to: /docs/configuration/encryption-at-rest code: |- spec: encryptionAtRest: enabled: true - token: RO title: Read-only replicas description: Append a read-only site and it follows whichever site is active — never promoted, never a planned-failover target, never a DNS target, never a clone donor. It gets its own client Service and its own mysqlConf, so you can size it for reporting, analytics or a CDC tap without touching the write path. Fall behind readOnlyMaxLagSeconds and its endpoint sheds until it catches up; lose its data and it reclones itself. to: /docs/architecture/multi-site code: |- - name: reader role: read-only zone: us-east-1b flow: - n: "01" title: Dump detail: consistent, scheduled or on demand - n: "02" title: Retain detail: S3 or PVC · structured retention - n: "03" title: Verify detail: restore into a throwaway MySQL - n: "04" title: Bootstrap detail: initFromBackup into a new group backup-description: Not a cron job that shells out to mysqldump. Backup is a first-class part of the operator, from the schedule all the way through to proving the artifact can actually be restored. backup-title: Backup and restore. backup-title-two: The whole nine yards. backup-to: /docs/backup-and-restore/backup-overview kicker: Bootstrap, scale out, encrypt, back up lede: Bootstrap, read scale-out, encryption and backup are part of the operator, not four more systems you have to wire together. title: Data that looks title-two: after itself. --- :: ::home-control-plane --- points: - title: No coordination problem description: Nothing to elect, nothing to split. State lives in the CR's status and in MySQL itself. - title: The data plane doesn't need the operator description: A healthy primary and replica keep serving reads and writes with zero operator involvement — it is on the detection path, not the request path. - title: Correct even while the operator is down description: Sidecars self-fence when they can reach neither the operator nor their peer, so no split brain is possible while the control plane is missing. - title: Every failure mode is written down description: A documented matrix of faults, what the operator does, how long it takes, and what it costs you. filename: orders.yaml kicker: No quorum. No coordinator. lede: There is no distributed consensus in Bloodraven, no coordinator to keep quorum, and no second system to reconcile against. A single reconcile loop reads the observed topology and writes the decision — which is why the failure modes fit on one page. link-label: Read the failure-mode matrix link-to: /docs/operations/failure-mode-matrix title: One controller. One CRD. title-two: One loop. yaml: |- apiVersion: shipstream.io/v1alpha1 kind: MysqlFailoverGroup metadata: name: orders spec: dns: hostname: orders.az.example.com ttl: 60 replication: readOnlyMaxLagSeconds: 30 sites: - name: iad zone: us-east-1a lbIP: 10.0.1.1 - name: pdx zone: us-west-2a lbIP: 10.0.2.1 - name: reader role: read-only zone: us-east-1b --- :: ::home-app-layer --- df-steps: - cmd: WAIT detail: replica offset catches up - cmd: REPLTAKEOVER detail: promote without a restart - cmd: CLIENT KILL detail: old-master clients reconnect to the active endpoint df-description: Turn on spec.dragonfly and Bloodraven runs a Redis-compatible cache and session store per site that follows MySQL. Cache and session continuity, not another failover to operate. df-label: Dragonfly follows the writer df-title: Cache and sessions df-title-two: move with the primary. df-to: /docs/configuration/app-integration kicker: The app finds out in milliseconds lede: A failover the app never notices is the point. Bloodraven pushes topology changes to connected clients and moves your cache along with the database. title: Your application finds title-two: out immediately. ws-code: |- const ws = new WebSocket('ws://bloodraven:8082/ws/status') ws.onmessage = e => pool.reconnectIfPrimaryMoved(JSON.parse(e.data)) ws-description: A WebSocket stream publishes the full topology of every failover group the moment it changes, so an app can force a pool reconnect on promotion instead of discovering it through a wall of write errors. A REST status API and Prometheus metrics expose the same state. ws-label: Real-time status push ws-title: Push the topology ws-title-two: the instant it changes. ws-to: /docs/observability/monitoring --- :: ::home-proof --- chaos-commands: - make chaos-run SCENARIO=06-self-fence-isolated-primary - make chaos-run-all-profile PROFILE=smoke chaos-layers: - DST - COMPONENT - ENVTEST - K3D play-targets: - k3d - kind - minikube - docker or podman title-two-accent: true chaos-description: 47+ scripted chaos scenarios, including operator crashes mid-failover, rolling updates, Dragonfly failover, and backup and PITR verification. Each one states a hypothesis, injects a real fault, asserts on operator behaviour and captures full forensics when it fails. A smoke subset gates every release before artifacts are published. chaos-label: Chaos-tested in CI chaos-title: Primary kills. Partitions. Self-fencing. GTID divergence. Data wipes. chaos-to: /docs/get-started/playground#automated-chaos-suite id: proof kicker: Run it on your laptop first lede: Every claim on this page is exercised against a real Kubernetes cluster — nightly in CI, and on your laptop in about two minutes. play-command: ./playground/setup.sh play-description: Two MySQL sites, a live dashboard, a counter app that writes through the failover hostname, DNS visualisation and a chaos menu. Break it on purpose and watch the whole promotion happen in front of you. play-image: /img/playground.png play-image-alt: The Bloodraven playground dashboard, showing site state, DNS records and a live event log play-label: Interactive playground play-title: A whole cluster. play-title-two: One script. play-to: /docs/get-started/playground title: Proof, title-two: not promises. --- :: ::home-course --- stats: - value: "7" label: units - value: "27" label: topics - value: "34" label: quizzes and tests units: - n: "1" title: Meet the group — stand up a real three-site failover group and read its status. - n: "2" title: How the operator decides — predict its next move from a status dump alone. - n: "3" title: Emergency failover end to end — time the promotion, audit what it cost. - n: "4" title: Where failover meets your application — pools, reconnects and stale reads. - n: "5" title: When the world misbehaves — self-fencing, split brain, five kinds of partition. - n: "6" title: Backups, disaster recovery and a go-live checklist you would actually sign. - n: "7" title: Day 0 and day 2 — build a group from nothing, then upgrade under live traffic. description: Train your on-call team on a real cluster, not on slides. Hold a site down and time the promotion. Audit the exact transactions an emergency failover cost you. Wire an application that survives one. Roll a MySQL upgrade underneath live traffic. kicker: Train the on-call rotation note: "Every number in the course traces to a source: operator code, the shipped CRDs, recorded chaos-run forensics, and the MySQL manual." primary-label: Start the course primary-to: /courses/ secondary-label: Browse the docs secondary-to: /docs title: Bloodraven title-two: in Production. --- :: ::home-pricing --- links: - label: See pricing to: /pricing variant: primary - label: Why source-available to: /docs/licensing variant: outline points: - Free forever for individuals, non-commercial use, non-profits and companies under $1M annual revenue - Free forever for dev, test, staging, CI and evaluation, at any scale - Production at a company over $1M annual revenue is a one-time licence, not a subscription - No activation, no licence server, no feature gating — the software is fully functional without a key - Every version converts to Apache 2.0 two years after it is published kicker: Licensing lede: Bloodraven is source-available under the Business Source License. Prices, the eligibility table and the checkout links are on one page, so there is exactly one place to read what you owe. title: Free for most people. title-two: One-time for everyone else. --- :: ::home-final-cta --- links: - label: Get started to: /docs/get-started/getting-started variant: primary icon: i-lucide-arrow-right - label: Run the playground to: /docs/get-started/playground variant: outline icon: i-lucide-flask-conical - label: Star on GitHub to: https://github.com/ShipStream/bloodraven target: _blank variant: ghost icon: i-simple-icons-github description: Install the operator, create a failover group, then kill a site on purpose and watch it recover without you. Everything on this page runs on a laptop first. mark: Bloodraven · ShipStream title: Try it in two minutes. title-two: Then break it on purpose. --- ::