Data-at-rest encryption

Bloodraven can run MySQL with InnoDB data-at-rest encryption turned on, keeping the encryption key off the MySQL data PVC and off worker-node disks. It uses the GPL component_keyring_file keyring component that ships in MySQL Community Edition, so it needs no Oracle Enterprise licence and no CSI-level encrypted storage.

spec:
  tls:                             # required
    secretName: mysql-tls
  encryptionAtRest:
    enabled: true

The operator installation must also enable the chart's dedicated escrow TLS listener. Everything below explains that prerequisite, what the feature protects, and how the keyring lifecycle works.

What this is not

Threat model

In scope — what this protects against:

Out of scope — what this does not protect against:

• A live root-level compromise of a node, or of the MySQL process. A running MySQL necessarily holds decrypted keys in memory. Anything that can read that memory, or that can simply ask MySQL for the data, wins.
• A compromised Kubernetes control plane. See etcd is now part of your key custody.
• Backup, restore, and verification staging. See Boundaries.

etcd is now part of your key custody

The live keyring is projected from a Kubernetes Secret. Kubernetes stores Secrets unencrypted in etcd by default. Without API-server encryption at rest, enabling this feature does not protect your keys — it just moves them from the MySQL data disk to the control-plane disk.

Before enabling encryptionAtRest, you must:

1. Turn on API-server encryption at rest for Secrets, ideally KMS-backed (kms provider v2), not aescbc with a local key file. See the Kubernetes encryption-at-rest guide.
2. Restrict RBAC on Secrets in the failover group's namespace. Anyone who can get secrets there can read the keyring and decrypt a stolen PVC.
3. Disable or encrypt swap on worker nodes. Kubernetes asks for tmpfs with noswap, but kernel support for that mount option is only official from Linux 6.3. On older kernels kubelet logs a warning and memory-backed volume contents can reach swap. Verify with mount | grep tmpfs on a node and check kubelet's startup logs.

None of these are optional. Bloodraven cannot verify them for you.

Requirements

• spec.tls must be set. MySQL requires a secure connection to clone encrypted data, and Bloodraven bootstraps every replica with CLONE INSTANCE. The CRD rejects encryptionAtRest.enabled: true without TLS.
• The spec.tls certificate must cover the per-site Services.spec.tls makes the operator set require_secure_transport=ON, so the sidecar's own MySQL connection is TLS too. It dials loopback — a name no certificate carries — and therefore verifies against mysql-<group>-<site>.<namespace>.svc.cluster.local. That SAN is already required for the operator's failover probes (see Credentials and TLS); if it is missing, the sidecar cannot connect, /health fails, and the pod never becomes ready.
• The operator escrow TLS listener must be enabled. Set auxiliary.escrowTLS.enabled=true and auxiliary.escrowTLS.existingSecret=<secret> on the Bloodraven chart. The Secret must contain tls.crt and tls.key; the certificate SAN must cover the auxiliary Service FQDN, and every encrypted group's spec.tls ca.crt must trust its issuer. Keyring bytes and their bearer token are never sent over the plaintext auxiliary listener.
• MySQL 9.x Community (or any build shipping component_keyring_file). Verified against mysql:9.7.
• Image-specific paths. Bloodraven copies the image's mysqld binary to a private runtime path and places the global component manifest next to that copy. It places the global component config in the image's plugin_dir. The private executable path also prevents a host-level, path-attached /usr/sbin/mysqld AppArmor profile from breaking component loading inside kind. Defaults match the official mysql:9.x images. If you run a different image, check the source binary and plugin paths:

  docker run --rm --entrypoint sh <image> -c \
    'command -v mysqld; mysqld --verbose --help 2>/dev/null | grep "^plugin_dir"'
  

  
  and set spec.encryptionAtRest.keyring.mysqldDir / .pluginDir accordingly, before you enable encryption: the keyring paths cannot change while it is enabled.
• Runtime confinement is left alone. Encryption keeps the container runtime's default AppArmor and seccomp profiles, so Pod Security Baseline still admits the pod. If a node's default profile stops mysqld from loading the keyring component (MY-013712, or a missing errmsg.sys for a path that exists), set appArmorProfile and seccompProfile in spec.containerSecurityContext. Releases before this change set both to Unconfined whenever encryption was enabled. Upgrading rolls encrypted groups once to drop that setting.

How the keyring lifecycle works

Every site has its own independent keyring. That is not a simplification — MySQL's encrypted clone decrypts the donor's tablespace keys in transit and the recipient re-encrypts them under a new master key of its own, so sites cannot share one.

A site is in one of two renderings at any moment.

Sealed — the steady state

    Kubernetes Secret  mysql-<group>-<site>-keyring-v<N>   (immutable)
              │
              │  kubelet projects it onto node-local tmpfs
              ▼
    /run/mysql-keyring/keyring   (read-only, mode 0444)
              │
              │  component_keyring_file with "read_only": true
              ▼
    mysqld ──────────────► encrypted /var/lib/mysql

"read_only": true is doing real work here, not documentation. With it set, MySQL physically cannot add a key:

mysql> ALTER INSTANCE ROTATE INNODB MASTER KEY;
ERROR 3185 (HY000): Can't find master key from keyring, ...

That is the guarantee that makes the design safe: in the steady state there is no window in which MySQL can create a key that is not already escrowed, because it cannot create keys at all.

Unsealed — bootstrap, clone, and rotation

Three operations genuinely need to write the keyring:

For those, the operator re-renders the site with the keyring on a memory-backed emptyDir, seeded from the current escrow version by an init container, and arms the sidecar's escrow agent:

    keyring-init ──seed──► /run/mysql-keyring/keyring  (tmpfs, rw)
                                      │
                    mysqld writes ────┤
                                      │
                          sidecar reads, hashes
                                      │
                  HTTPS POST /keyring/escrow  (bearer token)
                                      ▼
                            operator writes a new
                            immutable Secret version
                                      │
                    operator re-reads it and compares
                    against the live digest
                                      ▼
                            re-render sealed, roll pod

The operator will not seal a site until all of the following hold:

1. an escrow Secret exists for the site,
2. its contents, re-hashed by the operator, match the live keyring digest the sidecar reports,
3. the Deployment is actually rendered with the Secret projection and the pod is ready,
4. MySQL reports Read_only: Yes in performance_schema.keyring_component_status.

Key loss is bounded by construction

The only window in which a keyring can be lost is while a site is deliberately unsealed. That window is made survivable rather than merely short:

• Bootstrap: the site has no data worth keeping. Recovery is to wipe and re-bootstrap.
• Clone: the site is being overwritten from a donor anyway. Recovery is to re-clone.
• Rotation: the operator refuses to rotate the active primary, and refuses to promote a site whose UnsealReason is Rotation until it is Sealed. A replica that loses its keyring mid-rotation is re-cloneable from the primary; a primary that lost one would not be. See Rotating keys.

So a lost keyring always costs you a re-clone of one site, never your data. The one thing that would be unrecoverable — losing the escrow Secret of a sealed site — is why the operator raises a Failed phase and a KeyringEscrowMissing event the moment it notices one is gone.

Status and monitoring

kubectl get mysqlfailovergroup orders -o jsonpath='{.status.encryptionAtRest}' | jq

{
  "sites": [
    {
      "name": "iad",
      "phase": "Sealed",
      "keyringSecret": "mysql-orders-iad-keyring-v1",
      "keyringVersion": 1,
      "keyringDigest": "sha256:9f2c…",
      "coverage": {
        "keyringComponent": "component_keyring_file",
        "keyringReadOnly": true,
        "systemTablespaceEncrypted": true,
        "unencryptedTablespaces": 0,
        "redoLogEncrypted": true,
        "undoLogEncrypted": true,
        "binlogEncrypted": true
      }
    }
  ]
}

status.conditions[type=EncryptionAtRestReady] summarizes the group: True with reason AllSitesSealed means every site is sealed. There is no separate sealed field; wait or alert on the condition and the keyring metrics below.

kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.conditions[?(@.type=="EncryptionAtRestReady")].reason}'

The coverage booleans are optional: one the probe did not report (no row, NULL, or an unrecognized value) is absent, never false.

When a site will not seal

A site that stays at phase: Unsealed carries the reason in sites[].message, including the sidecar's own last error where it has one. The three common causes:

kubectl get mysqlfailovergroup orders \
  -o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'

Metrics

Suggested alert:

- alert: BloodravenKeyringNotSealed
  expr: bloodraven_keyring_phase{phase="sealed"} == 0
  for: 15m
  annotations:
    summary: "{{ $labels.site }} is running with a writable keyring"

Coverage

With the defaults, Bloodraven enforces these as operator-owned settings — a spec.mysqlConf entry naming the same key (in any spelling, including loose-, skip- or disable- prefixed forms) is ignored, so a stray override cannot silently downgrade a site:

default_table_encryption=ON
table_encryption_privilege_check=ON
innodb_redo_log_encrypt=ON
innodb_undo_log_encrypt=ON
binlog_encryption=ON          # covers relay logs on replicas too

Plus ALTER TABLESPACE mysql ENCRYPTION='Y', run once by the sidecar on the writable site. This one matters more than it looks: default_table_encryption does not cover the mysql system tablespace, which holds the data dictionary. Leave it unencrypted and every schema, table, and column name is readable on a stolen PVC even though the row data is not.

Each can be turned off individually — and each one you turn off narrows the claim:

spec:
  encryptionAtRest:
    enabled: true
    coverage:
      binaryLog: false     # binlogs and relay logs land in plaintext

Enabling encryption

On a new failover group

Set spec.encryptionAtRest.enabled: true and spec.tls before the group first bootstraps. The first PrimaryCandidate site comes up unsealed, creates its keys, escrows, and seals. Every other site is seeded with CLONE INSTANCE: the operator unseals the recipient, holds it unsealed until the clone and the post-clone mysqld restart finish, then re-escrows and reseals. A single-site group has no clone step.

On an existing group

Decide before you set the flag. Once spec.encryptionAtRest.enabled: true is stored, admission refuses setting it back to false or removing the block (see Encryption cannot be turned off). That holds even when the operator refused the adoption below and nothing was encrypted: a refused request cannot be taken back either.

The operator refuses by default (spec.encryptionAtRest.existingData: Refuse), and EncryptionAtRestReady is False with reason AdoptionRefused:

Warning  EncryptionAdoptionRefused  spec.encryptionAtRest.enabled was turned on for a
  group that is already serving from site "iad". Existing tablespaces stay plaintext —
  MySQL only encrypts data written after the fact. spec.encryptionAtRest.enabled cannot
  be turned back off. Supported path: restore a backup into a new encrypted group and
  move the applications to it, or accept partial coverage with
  spec.encryptionAtRest.existingData: LeavePlaintext.

This is not a mistake in the refusal — it is the honest answer. Flipping the flag on a live group gives you a cluster that reports encrypted while most of the bytes on the PVC are not. A refused group keeps running unencrypted. From there, use the supported path below, or accept partial coverage with the escape hatch, which starts the adoption.

Supported path: a new encrypted group, no plaintext left behind. Leave the serving group as it is and replace it with a new group that has spec.encryptionAtRest.enabled: true and spec.tls from its creation, restored from a final backup of the old one. Follow A group that holds data: writes stop, a final backup, the encrypted group restored from it with spec.initFromBackup, verification, and the cut-over of the clients to the new group's endpoints. Before the cut-over, also check that EncryptionAtRestReady is True with reason AllSitesSealed and unencryptedTablespaces is 0 on every site. After the old group is retired, check that its PersistentVolumes are gone too: a Retain reclaim policy keeps them, and they still hold plaintext. Its backups are plaintext as well unless their profile encrypts them.

Escape hatch — accept partial coverage:

If you understand that pre-existing tables stay in the clear until they are rebuilt, say so in the spec, next to enabled:

spec:
  encryptionAtRest:
    enabled: true
    existingData: LeavePlaintext

The policy is read only when encryption is first turned on; changing it later does not undo an adoption.

Adoption rolls one site at a time. Encrypted configuration is written to a new immutable, content-addressed ConfigMap; an unrolled Deployment continues referencing its original unencrypted ConfigMap. The ordered updater switches the ConfigMap reference and keyring volume/init wiring together in one pod template revision. A restart of a site that has not taken its update therefore still uses a coherent unencrypted configuration rather than requesting a keyring the pod does not provide.

New writes are then encrypted, redo/undo/binlog encryption engages, and unencryptedTablespaces tells you how much is still exposed. Rebuild the remainder at your own pace:

ALTER TABLE mydb.orders ENCRYPTION='Y';   -- rebuilds the table

Note that this rewrites each table in place, so plan it like any other large ALTER.

Rotating keys

Rotation is the one operation where a lost keyring would cost you data rather than a re-clone, because ALTER INSTANCE ROTATE INNODB MASTER KEY re-wraps the tablespace keys under a new master key. The operator therefore refuses to rotate the active primary.

The supported procedure rotates every site while it is a replica:

# 1. Rotate each replica, one at a time. Wait for Sealed between them.
kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/rotate-keyring=pdx --overwrite

kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="pdx")].phase}'
# Pending → Unsealed → Escrowed → Sealed

# 2. Planned failover onto a rotated site.
kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/planned-failover=pdx

# 3. Rotate the old primary, now a replica.
kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/rotate-keyring=iad --overwrite

The annotation is cleared automatically once the requested rotation has completed (the site is Sealed again on its new key). A site that returns to Sealed by other means, such as an escrow repair from Failed, keeps the request. Rotation is also refused while an ordered update or a planned failover is in flight. The operator also decides against its live topology state, because status.activeSite and status.updatePhase lag: it refuses while an ordered update is pending or running (an ordered update is in progress), while a planned failover is in flight, while a promotion is in flight or not yet observed, and until it has itself observed a unique writable primary with the target read-only, so a restart while status.activeSite still names a former primary cannot admit a rotation of the real one. It also refuses while no topology manager runs for the group yet, and while one being replaced still has an ordered update or promotion in flight. All of these, like the status-based update, planned-failover and unsettled-topology refusals, clear on their own: the annotation is kept, the operator retries every 10s, and it emits one KeyringRotationRejected event per reason. While a rotation waits, the site's sealed-state checks keep running, so a missing or altered escrow still fails it loudly; its status message reads sealed against <secret>; rotation deferred: <reason>. Only a request to rotate the active primary, whether status names it or the operator observes it writable, is refused outright and its annotation removed.

The inverse coupling also holds: a site mid-rotation is not a promotion target. Emergency failover skips it (or refuses entirely if it is the only remaining candidate). A planned failover whose target is still UnsealReason=Rotation is rejected with status.plannedFailover.result.reason: KeyringRotation — wait for Sealed, then re-annotate. An ordered update never hands the active site to it either: the update re-checks the standby when it picks the handoff target and, with no other eligible standby, ends without fencing anything. There is no cancel-rotation command; if rotation is stalled, repair escrow or the sidecar and let the site reach Sealed. If that leaves the group with no promotion target, the operator stays read-only on purpose: promoting would make the unescrowed keyring the sole authoritative copy.

Each rotation mints a new immutable escrow Secret version. keyring.retainVersions (default 5) controls how many superseded versions are kept; the version a site is currently sealed against is never pruned, even if retention would otherwise reach it.

Disaster recovery

A site's pod is gone. Nothing to do. The keyring lives in the escrow Secret; kubelet re-projects it when the pod is rescheduled.

A site's escrow Secret was deleted. The site is Failed and a KeyringEscrowMissing event fires. If the pod is still running, its keyring is alive on tmpfs — do not delete or restart that pod. Recover by recreating the exact missing Secret directly from the running pod; this avoids a rollout while the only surviving key copy is in memory. Follow Keyring escrow lost in the runbooks, or run the doctor helper keyring-escrow-recover.sh (keyring-escrow-recover.sh -n orders orders iad). Either one:

• runs only while the site is Failed or Sealed with no unsealReason. Any other state is a rotation, bootstrap or clone that owns the Secret;
• checks the running keyring against the recorded keyringDigest before writing anything, and never deletes a Secret;
• does not touch the site's status. The operator returns the site to Sealed once the Secret hashes to the recorded digest and MySQL confirms it is running that keyring (Event KeyringRecovered). Do not set the phase by hand: a rotation started meanwhile would be overwritten with Sealed, and the pod would roll back onto the old keyring, losing the new master key.

If the Secret still exists but no longer matches (KeyringEscrowCorrupt), both refuse. Replacing it is a separate, deliberate step: the helper's --replace-corrupt <secret-name> backs the Secret up first.

If the pod is already gone, that site's data is unrecoverable. Delete its unreadable PVC and use the confirmed reclone annotation; the operator starts the replacement with a fresh keyring and clones from the healthy primary.

The whole namespace was lost. Restore from a backup (backup encryption is a separate, independent key), then bootstrap a fresh encrypted group and load it. Backup artifacts are logical dumps, so they do not depend on any site's keyring.

Rolling a site back onto an older keyring version. Patch the site's keyringSecret/keyringVersion/keyringDigest in status to an older retained version and let the pod roll. Only do this if you know the older version still decrypts the data on that PVC — a master-key rotation happened in between if the version changed, and the newer key is the one the tablespace headers reference.

Boundaries

Live MySQL encryption does not cover Bloodraven's logical backup/restore/verification workflows:

Backup artifacts at rest in S3 or on a PVC are separately protected by spec.backup.profiles[].encryption — turn that on too. The gap above is the transient staging inside the pod's own emptyDir, which lands on the node's ephemeral storage.

If your requirement is "no plaintext database content ever touches node storage", this feature alone does not satisfy it and you should also constrain where backup Jobs schedule, or use encrypted ephemeral storage on those nodes.

Encryption cannot be turned off

Once spec.encryptionAtRest.enabled is true, admission refuses setting it back to false or removing the block:

spec.encryptionAtRest.enabled cannot be turned off or removed once it is true, even if encryption never started. A group without encryption is a new group restored from a backup

The rule does not depend on what the operator did with the flag. Once encryption has started, the tablespaces, redo and undo logs and binary logs stay encrypted with keys only the keyring holds, so a group without the keyring could not read its own data. A request the operator refused (AdoptionRefused) encrypted nothing, and it cannot be taken back either. keyring.dataFileDir, mysqldDir and pluginDir cannot change while encryption is enabled either (other keyring and coverage fields can). To run a group without encryption, restore a backup into a new group without spec.encryptionAtRest and move the applications to it (see A group that holds data).

Interaction with other features

Reference

Resources the operator creates

Annotations

Further reading

• InnoDB data-at-rest encryption
• Keyring component installation
• component_keyring_file
• Encrypting binary and relay logs
• Cloning encrypted data
• Kubernetes: encrypting data at rest
• Kubernetes: swap and memory-backed volumes