Data-at-rest encryption

Bloodraven can run MySQL with InnoDB data-at-rest encryption turned on, keeping the encryption key off the MySQL data PVC and off worker-node disks. It uses the GPL component_keyring_file keyring component that ships in MySQL Community Edition, so it needs no Oracle Enterprise licence and no CSI-level encrypted storage.

spec:
  tls:                             # required
    issuerRef: {name: ca, kind: Issuer}
    secretName: mysql-tls
  encryptionAtRest:
    enabled: true

The operator installation must also enable the chart's dedicated escrow TLS listener. Everything below explains that prerequisite, what the feature protects, and how the keyring lifecycle works.

What this is not

Threat model

In scope — what this protects against:

Out of scope — what this does not protect against:

• A live root-level compromise of a node, or of the MySQL process. A running MySQL necessarily holds decrypted keys in memory. Anything that can read that memory, or that can simply ask MySQL for the data, wins.
• A compromised Kubernetes control plane. See etcd is now part of your key custody.
• Backup, restore, and verification staging. See Boundaries.

etcd is now part of your key custody

The live keyring is projected from a Kubernetes Secret. Kubernetes stores Secrets unencrypted in etcd by default. Without API-server encryption at rest, enabling this feature does not protect your keys — it just moves them from the MySQL data disk to the control-plane disk.

Before enabling encryptionAtRest, you must:

1. Turn on API-server encryption at rest for Secrets, ideally KMS-backed (kms provider v2), not aescbc with a local key file. See the Kubernetes encryption-at-rest guide.
2. Restrict RBAC on Secrets in the failover group's namespace. Anyone who can get secrets there can read the keyring and decrypt a stolen PVC.
3. Disable or encrypt swap on worker nodes. Kubernetes asks for tmpfs with noswap, but kernel support for that mount option is only official from Linux 6.3. On older kernels kubelet logs a warning and memory-backed volume contents can reach swap. Verify with mount | grep tmpfs on a node and check kubelet's startup logs.

None of these are optional. Bloodraven cannot verify them for you.

Requirements

• spec.tls must be set. MySQL requires a secure connection to clone encrypted data, and Bloodraven bootstraps every replica with CLONE INSTANCE. The CRD rejects encryptionAtRest.enabled: true without TLS.
• The spec.tls certificate must cover the per-site Services.spec.tls makes the operator set require_secure_transport=ON, so the sidecar's own MySQL connection is TLS too. It dials loopback — a name no certificate carries — and therefore verifies against mysql-<group>-<site>.<namespace>.svc.cluster.local. That SAN is already required for the operator's failover probes (see Credentials and TLS); if it is missing, the sidecar cannot connect, /health fails, and the liveness probe restarts the container.
• The operator escrow TLS listener must be enabled. Set auxiliary.escrowTLS.enabled=true and auxiliary.escrowTLS.existingSecret=<secret> on the Bloodraven chart. The Secret must contain tls.crt and tls.key; the certificate SAN must cover the auxiliary Service FQDN, and every encrypted group's spec.tls ca.crt must trust its issuer. Keyring bytes and their bearer token are never sent over the plaintext auxiliary listener.
• MySQL 9.x Community (or any build shipping component_keyring_file). Verified against mysql:9.7.
• Image-specific paths. Bloodraven copies the image's mysqld binary to a private runtime path and places the global component manifest next to that copy. It places the global component config in the image's plugin_dir. The private executable path also prevents a host-level, path-attached /usr/sbin/mysqld AppArmor profile from breaking component loading inside kind. Defaults match the official mysql:9.x images. If you run a different image, check the source binary and plugin paths:

  docker run --rm --entrypoint sh <image> -c \
    'command -v mysqld; mysqld --verbose --help 2>/dev/null | grep "^plugin_dir"'
  

  
  and set spec.encryptionAtRest.keyring.mysqldDir / .pluginDir accordingly.

How the keyring lifecycle works

Every site has its own independent keyring. That is not a simplification — MySQL's encrypted clone decrypts the donor's tablespace keys in transit and the recipient re-encrypts them under a new master key of its own, so sites cannot share one.

A site is in one of two renderings at any moment.

Sealed — the steady state

    Kubernetes Secret  mysql-<group>-<site>-keyring-v<N>   (immutable)
              │
              │  kubelet projects it onto node-local tmpfs
              ▼
    /run/mysql-keyring/keyring   (read-only, mode 0444)
              │
              │  component_keyring_file with "read_only": true
              ▼
    mysqld ──────────────► encrypted /var/lib/mysql

"read_only": true is doing real work here, not documentation. With it set, MySQL physically cannot add a key:

mysql> ALTER INSTANCE ROTATE INNODB MASTER KEY;
ERROR 3185 (HY000): Can't find master key from keyring, ...

That is the guarantee that makes the design safe: in the steady state there is no window in which MySQL can create a key that is not already escrowed, because it cannot create keys at all.

Unsealed — bootstrap, clone, and rotation

Three operations genuinely need to write the keyring:

For those, the operator re-renders the site with the keyring on a memory-backed emptyDir, seeded from the current escrow version by an init container, and arms the sidecar's escrow agent:

    keyring-init ──seed──► /run/mysql-keyring/keyring  (tmpfs, rw)
                                      │
                    mysqld writes ────┤
                                      │
                          sidecar reads, hashes
                                      │
                  HTTPS POST /keyring/escrow  (bearer token)
                                      ▼
                            operator writes a new
                            immutable Secret version
                                      │
                    operator re-reads it and compares
                    against the live digest
                                      ▼
                            re-render sealed, roll pod

The operator will not seal a site until all of the following hold:

1. an escrow Secret exists for the site,
2. its contents, re-hashed by the operator, match the live keyring digest the sidecar reports,
3. the Deployment is actually rendered with the Secret projection and the pod is ready,
4. MySQL reports Read_only: Yes in performance_schema.keyring_component_status.

Key loss is bounded by construction

The only window in which a keyring can be lost is while a site is deliberately unsealed. That window is made survivable rather than merely short:

• Bootstrap: the site has no data worth keeping. Recovery is to wipe and re-bootstrap.
• Clone: the site is being overwritten from a donor anyway. Recovery is to re-clone.
• Rotation: the operator refuses to rotate the active primary. A replica that loses its keyring mid-rotation is re-cloneable from the primary; a primary that lost one would not be. See Rotating keys.

So a lost keyring always costs you a re-clone of one site, never your data. The one thing that would be unrecoverable — losing the escrow Secret of a sealed site — is why the operator raises a Failed phase and a KeyringEscrowMissing event the moment it notices one is gone.

Status and monitoring

kubectl get mysqlfailovergroup orders -o jsonpath='{.status.encryptionAtRest}' | jq

{
  "sealed": true,
  "sites": [
    {
      "name": "iad",
      "phase": "Sealed",
      "keyringSecret": "mysql-orders-iad-keyring-v1",
      "keyringVersion": 1,
      "keyringDigest": "sha256:9f2c…",
      "coverage": {
        "keyringComponent": "component_keyring_file",
        "keyringReadOnly": true,
        "systemTablespaceEncrypted": true,
        "unencryptedTablespaces": 0,
        "redoLogEncrypted": true,
        "undoLogEncrypted": true,
        "binlogEncrypted": true
      }
    }
  ]
}

status.conditions[type=EncryptionAtRestReady] summarizes the group.

When a site will not seal

A site that stays at phase: Unsealed carries the reason in sites[].message, including the sidecar's own last error where it has one. The three common causes:

kubectl get mysqlfailovergroup orders \
  -o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'

Metrics

Suggested alert:

- alert: BloodravenKeyringNotSealed
  expr: bloodraven_keyring_phase{phase="sealed"} == 0
  for: 15m
  annotations:
    summary: "{{ $labels.site }} is running with a writable keyring"

Coverage

With the defaults, Bloodraven enforces these as operator-owned settings — a spec.mysqlConf entry naming the same key is ignored, so a stray override cannot silently downgrade a site:

default_table_encryption=ON
table_encryption_privilege_check=ON
innodb_redo_log_encrypt=ON
innodb_undo_log_encrypt=ON
binlog_encryption=ON          # covers relay logs on replicas too

Plus ALTER TABLESPACE mysql ENCRYPTION='Y', run once by the sidecar on the writable site. This one matters more than it looks: default_table_encryption does not cover the mysql system tablespace, which holds the data dictionary. Leave it unencrypted and every schema, table, and column name is readable on a stolen PVC even though the row data is not.

Each can be turned off individually — and each one you turn off narrows the claim:

spec:
  encryptionAtRest:
    enabled: true
    coverage:
      binaryLog: false     # binlogs and relay logs land in plaintext

Enabling encryption

On a new failover group

Set spec.encryptionAtRest.enabled: true and spec.tls before the group first bootstraps. Every site comes up unsealed, creates its keys during initialization, escrows, and seals. Nothing else to do.

On an existing group

The operator refuses by default:

Warning  EncryptionAdoptionRefused  spec.encryptionAtRest.enabled was turned on for a
  group that is already serving from site "iad". Existing tablespaces stay plaintext —
  MySQL only encrypts data written after the fact.

This is not a mistake in the refusal — it is the honest answer. Flipping the flag on a live group gives you a cluster that reports encrypted while most of the bytes on the PVC are not.

Supported path — replica-first, no plaintext left behind:

1. Stand up the encrypted group (or add an encrypted site) and let it bootstrap and seal.
2. Load it from the existing cluster (spec.initFromBackup, or replication from the old primary).
3. Verify status.encryptionAtRest.sealed == true and unencryptedTablespaces == 0 on every site.
4. Cut traffic over with a planned failover.
5. Decommission the old cluster and wipe its PVCs — they still hold plaintext.

Escape hatch — accept partial coverage:

If you understand that pre-existing tables stay in the clear until they are rebuilt:

kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/encryption-adopt=confirm

Adoption rolls one site at a time. Encrypted configuration is written to a new immutable, content-addressed ConfigMap; an unrolled Deployment continues referencing its original unencrypted ConfigMap. The ordered updater switches the ConfigMap reference and keyring volume/init wiring together in one pod template revision. A restart of a site that has not taken its update therefore still uses a coherent unencrypted configuration rather than requesting a keyring the pod does not provide.

New writes are then encrypted, redo/undo/binlog encryption engages, and unencryptedTablespaces tells you how much is still exposed. Rebuild the remainder at your own pace:

ALTER TABLE mydb.orders ENCRYPTION='Y';   -- rebuilds the table

Note that this rewrites each table in place, so plan it like any other large ALTER.

Rotating keys

Rotation is the one operation where a lost keyring would cost you data rather than a re-clone, because ALTER INSTANCE ROTATE INNODB MASTER KEY re-wraps the tablespace keys under a new master key. The operator therefore refuses to rotate the active primary.

The supported procedure rotates every site while it is a replica:

# 1. Rotate each replica, one at a time. Wait for Sealed between them.
kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/rotate-keyring=pdx --overwrite

kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="pdx")].phase}'
# Pending → Unsealed → Escrowed → Sealed

# 2. Planned failover onto a rotated site.
kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/planned-failover=pdx

# 3. Rotate the old primary, now a replica.
kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/rotate-keyring=iad --overwrite

The annotation is cleared automatically once the target site is Sealed again. Rotation is also refused while an ordered update or a planned failover is in flight.

Each rotation mints a new immutable escrow Secret version. keyring.retainVersions (default 5) controls how many superseded versions are kept; the version a site is currently sealed against is never pruned, even if retention would otherwise reach it.

Disaster recovery

A site's pod is gone. Nothing to do. The keyring lives in the escrow Secret; kubelet re-projects it when the pod is rescheduled.

A site's escrow Secret was deleted. The site is Failed and a KeyringEscrowMissing event fires. If the pod is still running, its keyring is alive on tmpfs — do not delete or restart that pod. Recover by recreating the exact missing Secret directly from the running pod. This avoids a rollout while the only surviving key copy is in memory.

set -o pipefail
POD=$(kubectl get pod \
  -l shipstream.io/failover-group=orders,shipstream.io/site=iad \
  -o jsonpath='{.items[0].metadata.name}')
SECRET=$(kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringSecret}')
VERSION=$(kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringVersion}')
DIGEST=$(kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.encryptionAtRest.sites[?(@.name=="iad")].keyringDigest}')
KEYRING_DIR=$(kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.spec.encryptionAtRest.keyring.dataFileDir}')
KEYRING_DIR=${KEYRING_DIR:-/run/mysql-keyring}
FG_UID=$(kubectl get mysqlfailovergroup orders -o jsonpath='{.metadata.uid}')

kubectl exec "$POD" -c mysql -- cat "$KEYRING_DIR/keyring" | \
  kubectl create secret generic "$SECRET" --from-file=keyring=/dev/stdin
ACTUAL_DIGEST=$(kubectl get secret "$SECRET" -o jsonpath='{.data.keyring}' | \
  base64 -d | sha256sum | cut -d' ' -f1)
if [[ "sha256:${ACTUAL_DIGEST}" != "$DIGEST" ]]; then
  kubectl delete secret "$SECRET"
  echo "restored keyring digest does not match status; refusing to continue" >&2
  exit 1
fi
kubectl label secret "$SECRET" \
  app.kubernetes.io/name=mysql-keyring \
  app.kubernetes.io/managed-by=bloodraven \
  shipstream.io/failover-group=orders shipstream.io/site=iad \
  shipstream.io/keyring-version="$VERSION"
kubectl annotate secret "$SECRET" shipstream.io/keyring-digest="$DIGEST"
kubectl patch secret "$SECRET" --type=merge -p="{
  \"metadata\":{\"ownerReferences\":[{
    \"apiVersion\":\"shipstream.io/v1alpha1\",\"kind\":\"MysqlFailoverGroup\",
    \"name\":\"orders\",\"uid\":\"${FG_UID}\",\"controller\":true,\"blockOwnerDeletion\":true
  }]},\"immutable\":true
}"

SITE_INDEX=$(kubectl get mysqlfailovergroup orders -o json | \
  jq -er '.status.encryptionAtRest.sites | map(.name) | index("iad")')
kubectl patch mysqlfailovergroup orders --subresource=status --type=json -p="[
  {\"op\":\"test\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/name\",\"value\":\"iad\"},
  {\"op\":\"replace\",\"path\":\"/status/encryptionAtRest/sites/${SITE_INDEX}/phase\",\"value\":\"Sealed\"}
]"

If the pod is already gone, that site's data is unrecoverable. Delete its unreadable PVC and use the confirmed reclone annotation; the operator starts the replacement with a fresh keyring and clones from the healthy primary.

The whole namespace was lost. Restore from a backup (backup encryption is a separate, independent key), then bootstrap a fresh encrypted group and load it. Backup artifacts are logical dumps, so they do not depend on any site's keyring.

Rolling a site back onto an older keyring version. Patch the site's keyringSecret/keyringVersion/keyringDigest in status to an older retained version and let the pod roll. Only do this if you know the older version still decrypts the data on that PVC — a master-key rotation happened in between if the version changed, and the newer key is the one the tablespace headers reference.

Boundaries

Live MySQL encryption does not cover Bloodraven's logical backup/restore/verification workflows:

Backup artifacts at rest in S3 or on a PVC are separately protected by spec.backup.profiles[].encryption — turn that on too. The gap above is the transient staging inside the pod's own emptyDir, which lands on the node's ephemeral storage.

If your requirement is "no plaintext database content ever touches node storage", this feature alone does not satisfy it and you should also constrain where backup Jobs schedule, or use encrypted ephemeral storage on those nodes.

Interaction with other features

Reference

Resources the operator creates

Annotations

Further reading

• InnoDB data-at-rest encryption
• Keyring component installation
• component_keyring_file
• Encrypting binary and relay logs
• Cloning encrypted data
• Kubernetes: encrypting data at rest
• Kubernetes: swap and memory-backed volumes