Encryption at rest and the keyring lifecycle
Five phases, one steady state, and the moment etcd becomes part of your key custody. Plus the one site you must never rotate.
By the end of this topic you can
- Walk a site through the keyring phases and say which phase is the steady state
- Explain why enabling encryption at rest makes etcd part of your key custody
- Rotate a keyring without rotating the one site that must not be rotated
Restoring playground proved you can get the data back. Your security review asks the other question: what happens when someone else gets it? Today the playground PVCs hold InnoDB tablespaces, redo, undo and binlogs in the clear on three k3d worker nodes. So you add spec.encryptionAtRest.enabled: true and apply — and the API server rejects the object before the operator ever sees it. That rejection is the first honest thing this feature tells you.
A lifecycle, not a flag
spec.encryptionAtRest.enabled is a boolean. What it starts is not. Each site in playground — iad, pdx and the reader — walks its own keyring through a phase machine, and the CRD enum names five values (objective 7):
// +kubebuilder:validation:Enum="";Pending;Unsealed;Escrowed;Sealed;Failed
type SiteKeyringPhase string
Sealed is the steady state and the only phase that means done. It means the keyring data file is projected read-only from that site’s escrow Secret, so mysqld physically cannot add a key. Unsealed means the opposite — the keyring lives on a memory-backed volume MySQL can write, because something has to create a key. A site sitting in Unsealed is mid-flight. Reading the group as “encryption is on” because the flag says true is exactly the mistake.
Rotation is what makes the phase unreadable on its own. Rotation re-enters Unsealed from Sealed:
if site.Phase == v1alpha1.KeyringPhaseSealed && rotateTarget == site.Name {One string, two meanings: a site that has never been protected, and a protected site opened deliberately to mint a new master key. Read phase beside unsealReason and the rotation target — never alone.
The escrow, and who else now holds your keys
The escrow lives in per-site versioned Kubernetes Secrets, owner-ref’d to the group, immutable once written, with the digest recomputed from the Secret’s contents rather than trusted from its annotation. That is careful engineering. It is also where the security argument turns, and the docs do not soften it (objective 8):
The live keyring is projected from a Kubernetes Secret. Kubernetes stores Secrets unencrypted in etcd by default. Without API-server encryption at rest, enabling this feature does not protect your keys — it just moves them from the MySQL data disk to the control-plane disk.
So before you enable this you turn on API-server encryption at rest for Secrets (KMS-backed, not aescbc with a local key file), restrict RBAC on Secrets in the namespace, and deal with swap on the worker nodes. Then the sentence that should end the argument in any review:
None of these are optional. Bloodraven cannot verify them for you.
These are prerequisites you satisfy outside Bloodraven. Nothing in the operator checks them, and no status field will ever tell you that you skipped them.
Why the API server refused you
One prerequisite is enforced, as a CEL rule on the CRD — a hard admission rejection, not advice in a doc:
spec.encryptionAtRest.enabled requires spec.tls: MySQL requires a secure
connection to clone encrypted dataThat is not defensive coding. Oracle’s clone documentation states plainly that “a secure connection is required when cloning encrypted data regardless of whether this clause is specified” — the REQUIRE SSL clause does not enter into it. And CLONE INSTANCE is precisely how Bloodraven reseeds a diverged site after a reclone. On an encrypted group without TLS, reclone would simply be impossible, so the CRD refuses the combination up front rather than letting you discover it at 3am with a RecoveryBlocked site. Add spec.tls and the object is admitted.
Rotating every site, including the one you cannot rotate
Now the constraint objective 9 turns on: the operator refuses to rotate the active primary. The comment says why.
The operator refuses to rotate the active primary. Rotation necessarily runs with a writable keyring, and the only window in which a keyring can be lost is that one.
Lose a replica’s keyring in that window and you re-clone the site from a healthy peer. Lose the primary’s and you have lost data. So the site you cannot rotate is not a fixed name — it is whichever site is currently active. On playground right now that is iad.
Work through the consequence for the three sites. pdx is a replica and rotatable. The reader is role: read-only and never a primary, so it is rotatable too. iad becomes rotatable by no longer being primary — via the planned failover you already run from Unit 4, at RPO 0 by construction.
Put these in the order they happen.
- Rotate pdx — kubectl annotate mysqlfailovergroup playground bloodraven.shipstream.io/rotate-keyring=pdx --overwrite — wait for Sealed.
- Rotate reader — Same annotation, target reader. One site at a time; each rotation mints a new immutable escrow Secret version.
- Planned failover to pdx — kubectl annotate mysqlfailovergroup playground bloodraven.shipstream.io/planned-failover=pdx — iad is now a replica.
- Rotate iad — Now permitted. Rotation is also refused while an ordered update or a planned failover is in flight, and while no active primary is known.
Rotation is a per-instance physical operation — each site holds its own keyring — so 1.0.0 issues
ALTER INSTANCE ROTATE INNODB MASTER KEY with sql_log_bin = 0. If that statement replicated, the
replica would be one transaction ahead of its source and the next promotion would brick the
ex-primary. Recloning an encrypted site unseals the recipient, holds it unsealed until
CLONE INSTANCE and the post-clone restart finish, then reseals.
What you can now read
Where you previously read .status.sites[] for replication state, you now read one more block:
kubectl get mysqlfailovergroup playground \
-o jsonpath='{range .status.encryptionAtRest.sites[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'Three sites at Sealed, each sealed against mysql-playground-<site>-keyring-v1, with status.encryptionAtRest.sealed: true — that is the finished state. During step 1 above, pdx reads Unsealed with unsealReason: Rotation, and comes back as v2. The phase string is identical to the one it showed at bootstrap; only the reason and the version distinguish them.
One last currency check, because keyring advice ages badly. The current component is component_keyring_file, verified against the default mysql:9.7 image. The keyring_file plugin was removed in MySQL 8.4.0 along with the keyring_file_data system variable — any runbook or blog post naming the plugin is describing something that no longer exists. And Oracle’s own caveat belongs in the conversation if someone is enabling this for an auditor: component_keyring_file and component_keyring_encrypted_file “are not intended as a regulatory compliance solution”.
Handoff
You can enable encryption at rest on playground with its real prerequisites — TLS, because the CRD makes you; API-server encryption and Secret RBAC, because nothing will make you — read any site’s keyring phase and know whether it means protected or mid-flight, and rotate all three sites in an order that never opens the primary’s keyring. What none of this tells you is when a site quietly stops being sealed at 3am. That is a signal, and signals need somewhere to go.
Flashcards
The keyring phases a site of playground can be in
Five: Pending, Unsealed, Escrowed, Sealed, Failed — the CRD enum is "";Pending;Unsealed;Escrowed;Sealed;Failed.
What Sealed means physically, not just as a status string
The keyring data file is projected read-only from the site's escrow Secret, so mysqld cannot add a key.
Oracle's own caveat on file-based keyring components
component_keyring_file and component_keyring_encrypted_file are not intended as a regulatory compliance solution.
The one edge in the keyring lifecycle that runs backwards
Rotation: a site at Sealed whose name matches the rotation target re-enters Unsealed.
A site of playground reads phase: Unsealed. What do you read next before concluding anything?
unsealReason — Bootstrap, Clone or Rotation — plus the rotation target, because the phase string alone cannot tell you which.
Where the escrowed keyring is stored
In per-site versioned, immutable Kubernetes Secrets, owner-ref'd to the failover group.
The digest annotation on an escrow Secret is authoritative — true or false, and why?
False: it is informational. The operator always recomputes the digest from the Secret's contents rather than trusting the annotation.
What you must turn on outside Bloodraven before enabling spec.encryptionAtRest
API-server encryption at rest for Secrets (ideally KMS-backed, not aescbc with a local key file), plus restricted RBAC on Secrets in the group's namespace.
The admission error you get from encryptionAtRest.enabled: true with no spec.tls
The CEL rejection spec.encryptionAtRest.enabled requires spec.tls: MySQL requires a secure connection to clone encrypted data.
Why a clone of encrypted data genuinely needs a secure connection
MySQL upstream requires it when cloning encrypted data regardless of whether the REQUIRE SSL clause is specified — and CLONE INSTANCE is how Bloodraven reseeds a diverged site.
Why rotation is refused on the active primary specifically
Rotation runs with a writable keyring, the only window in which a keyring can be lost; on a replica that loss is recoverable by re-cloning from a healthy peer, on the primary it is not.
keyring_file versus component_keyring_file
The keyring_file plugin was removed in MySQL 8.4.0 along with keyring_file_data; component_keyring_file is the current component and what Bloodraven uses.
Quiz
Show answer
Answer: Sealed — the keyring is projected read-only and mysqld cannot add keys
Sealed is the steady state: the keyring data file is projected read-only from the escrow Secret, so mysqld physically cannot add a key. Escrowed is genuinely reassuring — the bytes are captured and the digest verified — but the Deployment is still rolling onto the sealed rendering, so the pod is still running a writable keyring. Unsealed is the trap: MySQL is running and encrypting, which looks finished, but the keyring is on a memory-backed volume MySQL can write, and the site is not considered protected. Pending is the very start — no escrowed keyring exists at all. (objective 7)
Sealed is the steady state: the keyring data file is projected read-only from the escrow Secret, so mysqld physically cannot add a key. Escrowed is genuinely reassuring — the bytes are captured and the digest verified — but the Deployment is still rolling onto the sealed rendering, so the pod is still running a writable keyring. Unsealed is the trap: MySQL is running and encrypting, which looks finished, but the keyring is on a memory-backed volume MySQL can write, and the site is not considered protected. Pending is the very start — no escrowed keyring exists at all. (objective 7)
Show answer
Answer: The site is running a writable memory-backed keyring so it can mint a new master key, and is not protected until it re-seals
Rotation is the one transition that moves backwards out of Sealed, re-entering Unsealed deliberately — a rotation needs a writable keyring, so the site is genuinely unprotected until it escrows the new keyring and re-seals. It is not a failure state: a failed lifecycle lands in Failed, and the escrow Secret versions are immutable and retained, so the old keyring has not gone anywhere. No clone is involved — unsealReason would read Clone rather than Rotation if it were. And the phase is not stale: this is exactly why you read phase beside unsealReason and the rotation target rather than alone, since bootstrap and rotation produce the identical phase string. (objective 7)
Rotation is the one transition that moves backwards out of Sealed, re-entering Unsealed deliberately — a rotation needs a writable keyring, so the site is genuinely unprotected until it escrows the new keyring and re-seals. It is not a failure state: a failed lifecycle lands in Failed, and the escrow Secret versions are immutable and retained, so the old keyring has not gone anywhere. No clone is involved — unsealReason would read Clone rather than Rotation if it were. And the phase is not stale: this is exactly why you read phase beside unsealReason and the rotation target rather than alone, since bootstrap and rotation produce the identical phase string. (objective 7)
Show answer
Answer: You have moved the keys from the MySQL data disk to the control-plane disk, and etcd is now part of your key custody
The live keyring is projected from a Kubernetes Secret, and Kubernetes stores Secrets unencrypted in etcd by default — so without API-server encryption the keys have relocated from the MySQL data disk to the control-plane disk rather than been protected. The first option confuses the pod's view with the cluster's: it is true that the keyring only ever lands on tmpfs inside the pod, but the Secret backing it is at rest in etcd. The second inverts the load-bearing sentence: 'None of these are optional. Bloodraven cannot verify them for you' — nothing in the operator checks, so sites seal happily. The fourth invents a dependency; rotation mints a new master key but changes nothing about where the escrow is stored. (objective 8)
The live keyring is projected from a Kubernetes Secret, and Kubernetes stores Secrets unencrypted in etcd by default — so without API-server encryption the keys have relocated from the MySQL data disk to the control-plane disk rather than been protected. The first option confuses the pod's view with the cluster's: it is true that the keyring only ever lands on tmpfs inside the pod, but the Secret backing it is at rest in etcd. The second inverts the load-bearing sentence: 'None of these are optional. Bloodraven cannot verify them for you' — nothing in the operator checks, so sites seal happily. The fourth invents a dependency; rotation mints a new master key but changes nothing about where the escrow is stored. (objective 8)
Show answer
Answer: False
The reversal: it is a hard CEL rejection at admission, not a recommendation — spec.encryptionAtRest.enabled requires spec.tls: MySQL requires a secure connection to clone encrypted data. The reason is upstream, not defensive: MySQL requires a secure connection when cloning encrypted data regardless of any REQUIRE SSL clause, and CLONE INSTANCE is exactly how Bloodraven reseeds a diverged site. Enabling the pair without TLS would produce a group that cannot be reclone-recovered, so the CRD refuses the combination up front instead of failing you during a recovery. (objective 8)
The reversal: it is a hard CEL rejection at admission, not a recommendation — spec.encryptionAtRest.enabled requires spec.tls: MySQL requires a secure connection to clone encrypted data. The reason is upstream, not defensive: MySQL requires a secure connection when cloning encrypted data regardless of any REQUIRE SSL clause, and CLONE INSTANCE is exactly how Bloodraven reseeds a diverged site. Enabling the pair without TLS would produce a group that cannot be reclone-recovered, so the CRD refuses the combination up front instead of failing you during a recovery. (objective 8)
Show answer
Answer:
iad, because it is the active primary and the operator refuses to rotate it — rotation runs with a writable keyring, and that is the only window in which a keyring can be lost, so on the primary a loss would cost data rather than a re-clone. Rotate pdx and reader first, one at a time, waiting for each to return to Sealed. Then run a planned failover onto pdx. iad is now a replica, so the refusal no longer applies and you rotate it last.
A full-credit answer shows: A strong answer names iad and ties the refusal to its role as active primary rather than to its name; explains that rotation needs a writable keyring and that a keyring lost on a replica is recoverable by re-cloning while one lost on the primary is not; and gives the workaround as rotate the other two sites, planned-failover away from iad, then rotate it. Answers that say the reader cannot be rotated (it can — it is never a primary) or that treat iad as permanently unrotatable have missed the point.
The unrotatable site is defined by current role, not identity: whichever site is active is the one the operator refuses, so the answer changes the moment you fail over. reader is a tempting wrong answer because it is excluded from other things — it can never be promoted and is never a backup source — but nothing stops it rotating. The fix is the planned failover from Unit 4: make iad a replica and the refusal evaporates. (objective 9)
Sample answer
iad, because it is the active primary and the operator refuses to rotate it — rotation runs with a writable keyring, and that is the only window in which a keyring can be lost, so on the primary a loss would cost data rather than a re-clone. Rotate pdx and reader first, one at a time, waiting for each to return to Sealed. Then run a planned failover onto pdx. iad is now a replica, so the refusal no longer applies and you rotate it last.
A full-credit answer shows
A strong answer names iad and ties the refusal to its role as active primary rather than to its name; explains that rotation needs a writable keyring and that a keyring lost on a replica is recoverable by re-cloning while one lost on the primary is not; and gives the workaround as rotate the other two sites, planned-failover away from iad, then rotate it. Answers that say the reader cannot be rotated (it can — it is never a primary) or that treat iad as permanently unrotatable have missed the point.
The unrotatable site is defined by current role, not identity: whichever site is active is the one the operator refuses, so the answer changes the moment you fail over. reader is a tempting wrong answer because it is excluded from other things — it can never be promoted and is never a backup source — but nothing stops it rotating. The fix is the planned failover from Unit 4: make iad a replica and the refusal evaporates. (objective 9)