Backup encryption at rest

Bloodraven can encrypt every backup artifact — full dump objects written by backup Jobs, archived binlog files uploaded by the sidecar, and their accompanying manifests — using a per-profile passphrase Secret. Encryption is opt-in, enabled by a single new field on a backup profile:

spec:
  backup:
    profiles:
      - name: nightly-s3
        storage:
          type: S3
          s3:
            bucket: shipstream-backups
            prefix: orders
            region: us-east-1
            credentialsSecretName: s3-backup-creds
        encryption:
          algorithm: AES-256-GCM        # default; reserved for future algos
          passphraseSecretKeyRef:
            name: orders-backup-passphrase
            key: passphrase             # defaults to "passphrase"

Operators who prefer storage-layer protection (AWS SSE-KMS, LUKS on PVCs, bucket default encryption) can leave the encryption: block unset and rely on that. For compliance-sensitive workloads — or any setup where the operator team wants to own the keys end-to-end — turn on client-side encryption.

Each backup Job pins its encryption configuration when it is created. Enabling or disabling profile encryption affects Jobs created afterward; an in-flight plaintext backup stays plaintext. status.encryption (mode Passphrase or None, and algorithm) is written from the producing Job when the Job is created and derived again from it at completion, even if the profile changes or is removed before completion. If a completed backup has incorrect attribution and its owned Job still exists, reconciliation repairs the status and emits EncryptionAttributionCorrected. If the Job is gone, the operator preserves the existing attribution.

Why client-side envelope encryption

Bucket default SSE-KMS already encrypts objects at rest in AWS, but the data-encryption keys live in the AWS account that hosts the backups. With client-side envelope encryption:

• The passphrase is projected from a Kubernetes Secret. Reads happen inside the operator-controlled pods (backup Job, restore Job, sidecar archiver, verification Job, pitr-download init container), which mount the Secret as a file. Except for the restore-side decryption Secrets, the mounted Secret is an immutable key snapshot the operator copied from your Secret (see Key snapshots), so a later edit of your Secret cannot change what a running or restarted pod loads. Job projections use mode 0400 with the Job's fsGroup providing group access. The live PITR sidecar uses 0444, matching its S3 credentials projection, because MySQL pods have no fsGroup by default. The passphrase volume is mounted read-only only into the consuming container; the live MySQL container does not mount it. The passphrase is never passed as an environment variable.
• The bucket / PVC credentials are divorced from the key material. A leaked S3 access key grants access to encrypted objects only; an attacker still needs the passphrase to recover plaintext.
• The mysqlsh image never handles encryption. Dumps are staged in a local emptyDir by the mysqlsh init container and then encrypted and uploaded by a separate bloodraven encrypt-upload main container; on restore the mirror bloodraven decrypt-download init container decrypts into the emptyDir before mysqlsh runs util.loadDump(). Every encryption / decryption step runs in a container that carries the operator-authored bloodraven binary, so MySQL-tooling CVEs can't exfiltrate plaintext if the mysqlsh container is subsequently compromised.

Wire format

Every encrypted object is a self-describing byte stream in the BRV1 format:

Header (32 bytes):
  [0..3]   magic          "BRV1"
  [4]      version        0x01
  [5]      algorithm      0x01  (AES-256-GCM)
  [6..7]   chunk_log2     uint16 BE. Plaintext chunk = 1 << chunk_log2.
  [8..23]  salt           16 bytes random. Fed to HKDF-SHA256.
  [24..31] nonce_prefix   first 8 bytes of each 12-byte GCM nonce.

Body: repeated chunks of AES-256-GCM(ciphertext || tag). Nonce layout:
  nonce[0..7]   = nonce_prefix
  nonce[8..10]  = counter (24-bit big-endian, starts at 0)
  nonce[11]     = final_flag (0x00 except on the last chunk: 0x01)

• Key derivation. dk = HKDF-SHA256(secret=passphrase, salt, info="bloodraven-backup-encryption-v1", len=32). A fresh random salt per file means two encryptions of the same plaintext do not share a data-encryption key. HKDF adds no work factor, so the passphrase must itself be high-entropy random data; a short or human-chosen passphrase can be brute-forced offline from any backup object.
• Tamper detection. AES-GCM authenticates every chunk; the final chunk carries a distinct AAD byte so a truncated stream is rejected rather than silently decoded as a shorter plaintext. A post-final trailer of extra bytes is also rejected.
• File size ceiling. 2^24 chunks × 64 KiB default chunk size = 1 TiB per file. MySQL dump objects almost never hit this (a consistent dump is split across thousands of smaller chunk files by mysqlsh), and single sealed binlog files are capped by MySQL's max_binlog_size which users tune to hundreds of MiB. If you ever need bigger single files, raise chunk_log2 in a future release.

Object names (S3 keys, PVC filenames) remain plaintext — the format doesn't try to hide them. Manifest files (manifest-<site>.json) are wrapped in the same BRV1 envelope so per-site GTID ranges and file lists do not leak either.

The format is implemented by internal/backupcrypto. The package test suite covers round-trip, wrong-passphrase rejection, truncation detection, empty plaintext, and a 4-MiB random payload torture test.

Wiring it up

1. Create the passphrase Secret

Bloodraven does not generate the passphrase itself — this keeps the Secret's lifecycle and backup policy firmly in the operator's hands. Use a long (≥32 bytes) random string:

head -c 48 /dev/urandom | base64 > passphrase.txt
kubectl create secret generic orders-backup-passphrase \
  --namespace orders \
  --from-file=passphrase=passphrase.txt
shred -u passphrase.txt

2. Reference it from the profile

Add encryption to the backup profile in the MysqlFailoverGroup spec, as shown at the top of this page. That single field enables:

• Client-side encryption of every full dump produced by Jobs from this profile.
• Client-side encryption of every sealed binlog the sidecar archiver uploads under this profile's storage (when spec.backup.pitr points at the same profile).
• Decryption on the matching restore, in-place restore, and verification paths — the operator stamps MysqlBackup.status.encryption.mode: Passphrase and records the backup's own dump-key snapshot in status.artifactStorage.passphraseSecretKeyRef, which restore and verification read; they never fall back to the profile's live Secret.

3. Apply; the operator validates

On apply the operator checks that the referenced Secret exists and carries the configured key (defaulting to passphrase), and that the value, trimmed of leading and trailing \n, \r, space and tab, is at least 32 bytes. A misconfiguration emits a warning event:

Warning  BackupEncryptionInvalid
  profile "nightly-s3": encryption.passphraseSecretKeyRef Secret "orders-backup-passphrase"
  not found in namespace "orders"

A MysqlBackup of an encrypted profile whose Secret or key is missing waits Pending with reason PassphraseSecretMissing (the operator cannot snapshot a key it cannot read); the event makes the root cause visible in kubectl describe mysqlfailovergroup. A passphrase shorter than 32 bytes only produces the warning: backups and PITR archiving keep encrypting with it, so replace it with a long random value: for dumps, by pointing the profile at a new Secret; for a PITR archive, by the rotation procedure (a new prefix). Nothing re-encrypts existing artifacts; keep the old passphrase to restore them.

Observing encryption

On each MysqlBackup CR:

kubectl get mysqlbackup orders-nightly-20260423 -o jsonpath='{.status}' | jq .

{
  "phase": "Succeeded",
  "artifactStorage": {
    "type": "S3",
    "location": "s3://shipstream-backups/orders/orders-nightly-20260423/",
    "bucket": "shipstream-backups",
    "passphraseSecretKeyRef": {"name": "mysqlbackup-orders-nightly-20260423-dump-key-1a2b3c4d", "key": "passphrase"},
    ...
  },
  "sizeBytes": 18874368,
  "encryption": {"mode": "Passphrase", "algorithm": "AES-256-GCM"},
  ...
}

In Job logs the bloodraven encrypt-upload main container emits a BLOODRAVEN_DUMP_COMPLETE sentinel identical in shape to the unencrypted flow, plus encrypted=true and algorithm=AES-256-GCM:

BLOODRAVEN_DUMP_COMPLETE v=2 location=s3://... sizeBytes=18874368 size=18.0%20MiB \
  gtidExecuted=abc:1-100 binlogFile=mysql-bin.000042 binlogPos=118 \
  ciphertextBytes=19006 files=42 encrypted=true algorithm=AES-256-GCM

Version 2 percent-encodes token values while preserving literal underscores in paths, binlog names, and GTID tags. The operator decodes the artifact metadata into status; the producing Job remains authoritative for encryption attribution. Unversioned lines from older operator images are still accepted.

Object-level verification — fetch the first 32 bytes of any artifact:

aws s3 cp --range bytes=0-31 s3://shipstream-backups/orders/orders-nightly-20260423/@.json head
xxd head | head -1
# 00000000: 4252 5631 0101 0010 ... -- "BRV1" magic + version + algorithm

Restore paths

Encryption is transparent to the restore caller: when the source backup's .status.encryption.mode is Passphrase the operator automatically adds a decrypt-download init container to the restore Job. The mysqlsh main container reads plaintext from a shared emptyDir and runs util.loadDump() as before.

The restore Job picks the dump passphrase in this order:

1. spec.initFromBackup.decryption.passphraseSecretKeyRef if set (recommended for cross-group restores and for direct source.s3 / source.pvc sources).
2. Otherwise, for a backupRef source, the backup's own dump-key snapshot recorded in status.artifactStorage.passphraseSecretKeyRef. The Secret must exist.

The profile's live encryption.passphraseSecretKeyRef is not a fallback: it may hold a different key than the one the dump was written with. If neither source is available, the restore fails with PITRBackupDumpKeyMissing before any Job is created. Restore the recorded snapshot Secret from a backup of it. spec.initFromBackup is fixed at creation, so adding a decryption block to it means deleting the never-bootstrapped group and creating it again with the block (see A failed bootstrap restore); for restoreInPlace, add the block and bump confirm.

spec.restoreInPlace.decryption follows the exact same contract.

A decryption block supplies only the passphrase. A backupRef source locates the artifact through the backup's recorded status.artifactStorage, not through its profile. A PVC backup needs no profile: the restore mounts the recorded claim. An S3 backup still reads its S3 credentials (credentialsSecretName) from the backup's profile, so that profile must still exist and store to S3. If it was deleted, or moved from S3 to PVC, the restore fails with a RestoreBuildFailed event (RestoreInPlaceBuildFailed for an in-place restore) before any Job is created:

Warning  RestoreBuildFailed
  initFromBackup.source.backupRef="orders-preupgrade" resolves to an
  S3 location ("s3://shipstream-backups/orders/orders-preupgrade/") but
  profile "nightly-s3" now stores to PVC (it supplies the S3 credentials);
  either restore the profile's S3 storage or delete the group and create it
  again with initFromBackup.source.s3 set (initFromBackup is fixed at
  creation; see
  https://bloodraven.dev/docs/backup-and-restore/backup-restore#a-failed-bootstrap-restore)

Restoring the profile's storage block is an edit of spec.backup.profiles and retries the restore in place. To restore an S3 backup without its original profile, point source.s3 at the artifact directly and keep the decryption block: in spec.restoreInPlace by editing it and bumping confirm; in spec.initFromBackup, which is fixed at creation, by deleting the never-bootstrapped group and creating it again with the corrected block.

Verification

MysqlBackupVerification automatically decrypts through the same bloodraven decrypt-download init container, with the backup's own dump-key snapshot from status.artifactStorage.passphraseSecretKeyRef. Verification has no decryption field: editing, rotating or removing the profile's encryption after the backup does not affect it. If the recorded snapshot Secret is missing, the verification fails at launch with PITRBackupDumpKeyMissing ("backup <b> is encrypted and the Secret <s> it recorded as its dump key is gone; restore that Secret").

An encrypted backup taken before v1.7.0 recorded no dump-key snapshot, so it cannot be verified: the verification fails PITRBackupDumpKeyMissing, and there is no override. Restore it with an explicit decryption instead (into a scratch group to check it), and verify a backup taken after the upgrade; see Upgrade policy: v1.7.0.

A point-in-time verification also reads the PITR archive. It mounts the archive's binding key snapshot, never the profile's Secret; see Backup verification.

PITR binlog archival

When spec.backup.pitr.profileName references a profile with encryption set, the sidecar archiver transparently wraps every Put / PutFile call through the same BRV1 format. The operator mounts the archive's key snapshot onto the sidecar container and sets BLOODRAVEN_PITR_PASSPHRASE_FILE; sidecar.NewArchiveStore wraps the concrete S3/PVC backend in an encryptedStore when that env var is present.

The sidecar reads the passphrase once when initializing its archive store. What it reads is decided by the group's archive encryption binding, not by the profile: once the binding is Bound, editing the profile's encryption, its Secret reference or the Secret's bytes changes nothing in the pods. The edit is reported and refused instead (see Rotation). Unset algorithm and key fields normalize to AES-256-GCM and passphrase.

On the restore side, the pitr-download init container mounts the archive binding's key snapshot, checks it against the descriptor's keyCheck, and passes it into its own NewArchiveStore call so the staged binlogs land in the emptyDir decrypted and ready for the segment replay (mysqlbinlog --include-gtids=<segment> into one mysql session). The dump-meta init container decrypts the dump's @.json, @.done.json and backup stamp with the dump passphrase (the backup's own dump-key snapshot, or decryption); there is no plaintext fallback. Per-site manifests are encrypted too — the archiver reads and writes them through the wrapped store like any other object.

Encryption preserves the archive identity and key layout. Encrypted content IDs are HMAC-SHA256 digests of sealed plaintext hashes, keyed by the archive passphrase with the domain bloodraven-binlog-content-id-v1. They remain stable for deduplication without revealing a plaintext SHA-256 in object names. Binlog payloads and manifests, including legacy contentId fields, are encrypted. Filenames and object counts remain visible. Encryption and decryption now stream through bounded pipes; uploads, archive inspection and downloads do not create ciphertext/plaintext scratch copies. Only the final staging .part is written on download.

Archive descriptor and keyCheck

The archive descriptor, <prefix>/binlogs/archive.json, is the one archive object that is not encrypted. It holds the archive ID, the source group (namespace, name, uid), the encryption mode (None or Passphrase), the algorithm (AES-256-GCM, or empty for None) and, for Passphrase, a keyCheck. Anyone who can read the prefix can therefore see which group wrote the archive and whether it is encrypted; binlog payloads and manifests stay encrypted.

The keyCheck is a 43-character base64url HMAC-SHA256 over the descriptor's identity fields, keyed with an HKDF-SHA256 key derived from the trimmed passphrase and a salt bound to the archive ID. It proves that a sidecar or Job holds the archive's passphrase without decrypting a manifest, and it is bound to one archive, so it cannot be replayed onto another. Like the BRV1 key derivation, it is not a password hash: a short passphrase can be brute-forced offline from the descriptor alone, so use high-entropy random data. The passphrase is trimmed of leading and trailing \n, \r, space and tab only, the same rule every reader and writer uses; other whitespace is part of the key.

Every writer and reader checks the descriptor against the identity it was given before it touches a manifest:

• an encryption mode or algorithm that differs from the descriptor's refuses PITRArchiveEncryptionMismatch;
• a passphrase that does not verify the keyCheck refuses PITRArchiveKeyMismatch.

Nothing migrates the archive: automatic archive migration, multiple key generations and in-place re-encryption are unsupported. Bindings make sure a spec or Secret edit never reaches a sidecar as a key the archive refuses.

The backup stamp (<location>.bloodraven-backup.json) is BRV1-encrypted with the dump passphrase when the profile encrypts its dumps; see Backup stamp.

Archive encryption bindings

For every archive destination the group writes (the active PITR profile) or reads (initFromBackup.pointInTime.archiveProfileName until the restore succeeds, and point-in-time MysqlBackupVerifications through archiveProfileName or the active profile), the group records the encryption identity it accepted in status.pitrArchiveBindings[]:

kubectl get mfg orders -o jsonpath='{range .status.pitrArchiveBindings[*]}{.id}{"\t"}{.state}{"/"}{.reason}{"\t"}{.encryption.keySecretName}{"\t"}{.destination}{"\n"}{end}'

A group holds at most 16 bindings, and only entries that stay count against the limit: Bound ones (kept until released) and those a target uses. Unused bindings that never became Bound are dropped automatically. A 17th destination gets none while 16 entries stay: the condition turns False with BindingLimitReached (Warning PITRArchiveBindingLimitReached), the group keeps its held render (or renders PITR disabled when it holds none) instead of rendering the new destination, and nothing renders for it until you release an unused binding. A reader (a restore or a verification) gets BindingLimitReached only when all 16 entries are Bound; while the list is full of entries that are not Bound it waits ("the binding list is full of entries that are not Bound; waiting for the group's binding pass to make room for <destination>").

How a binding becomes Bound

A binding is Bound only by proof, never by assumption:

• Consumer proof (write targets): a sidecar reports the destination's descriptor Verified with the binding's mode and exact key revision. The operator sets BLOODRAVEN_PITR_KEY_REVISION on the sidecar to the key snapshot's name, or plaintext for an unencrypted archive; the sidecar echoes it in status.pitr.archive.keyRevision. One verified site is enough. A sidecar with an empty revision (an older render) never proves anything.
• Check Job: a one-shot Job running bloodraven pitr-archive-check (labels app.kubernetes.io/name=mysql-backup, bloodraven.shipstream.io/resource=pitr-archive-check, bloodraven.shipstream.io/pitr-key-secret=<snapshot>, and annotation bloodraven.shipstream.io/pitr-key-secret-uid=<snapshot UID>; 120 s deadline, no retries, no ServiceAccount token) opens the descriptor with the candidate key and writes one termination line, BLOODRAVEN_PITR_ARCHIVE_CHECK v=2 state=<Absent|Verified|Refused> reason=… archiveID=… sourceUID=… sourceNamespace=… sourceName=… destinationDigest=…. destinationDigest is the hex SHA-256 of the normalized destination (domain-separated), so the line stays well under the 4096-byte termination message limit whatever the prefix and endpoint length; the operator compares it with the destination in the Job's own BLOODRAVEN_PITR_DESTINATION, which must equal the binding's. Check Jobs run only for a read target that is not yet Bound (so an initFromBackup point-in-time restore runs one check before its Job), and for a write target that is not yet Bound while a writer is already running elsewhere: a prefix switch, or site Deployments from before bindings existed. Verified binds; Absent on a write target means an empty prefix (Pending/Establishing: the sidecar creates the descriptor and proves it); Absent on a read target is ArchiveNotFound. A key that does not verify tries the next candidate (an earlier snapshot for that destination, then the profile's current bytes) and ends Refused/KeyMismatch. A check pod Pending for more than 60 s, or one that fails without a result line, is deleted and retried after a minute. So is a check that exits 0 with a line that does not parse ("check Job <job> reported a verdict line that does not parse: <parser error>") or that cannot write its verdict to the termination log (bloodraven pitr-archive-check exits 4 and logs ERROR "write the verdict to the termination log; the check fails"; the controller reports "check Job <job> failed: ..."). A check that cannot list the prefix (ArchiveListFailed) is not recorded as a refusal: the entry stays Pending/Authenticating with the message "ArchiveListFailed: the check of <candidate> could not list the prefix at <destination>; grant list permission on the prefix (s3:ListBucket) or make the volume readable; retrying in 1m0s", the check is retried every minute, and restores and verifications wait. An ArchiveListFailed refusal stored by an older operator is retried the same way. A check Job started for the other role than the destination is now checked as (writer or reader) is dropped and the target checked again ("check Job <job> ran as <role>, but the destination is now checked as <role>; checking again"). An Establishing entry whose destination is no longer written becomes Pending/Authenticating ("the destination is no longer written; authenticating it for its readers") and gets a reader check, and a Refused entry recorded by the other role's check is checked once more. Check refusals name the role: "<token> reported by the writer check of ..." or "... reported by the reader check of ...". A Verified verdict is dropped, and the entry checked again, if the key snapshot was replaced under the same name since the Job was created ("the key snapshot the check verified was replaced: ...; checking it again"), which is what the UID annotation detects.

A fresh group on an empty prefix renders its sidecars with the profile's key straight away; the sidecar creates the descriptor and its report binds it. A refused entry is retried only when its input changes (the profile's bytes or encryption, or the sidecar's report), never on a timer; a check that could not list the prefix (ArchiveListFailed) is the exception, retried every minute.

Once a write binding is Bound and belongs to this group (the descriptor's source.uid is the group's UID and the archive ID is the one derived for this group and destination), the sidecars always render it. A Bound binding is never demoted by a sidecar report: if a sidecar later reports ArchiveKeyMismatch against it, the descriptor was replaced, and PITRArchiveReady says so. A Bound read binding of another group's archive (or of a copy) that you make the write target is refused as ArchiveForeign (or ArchiveCopy) on the condition, and the previous writer render stays.

The group holds its last accepted render

While the write binding is not Bound (being proven, refused, or over the limit), the group keeps rendering the archive identity it last accepted — destination, archive ID, storage type, bucket, prefix, claim, sub-path, encryption mode, algorithm and key snapshot — read from the bloodraven.shipstream.io/pitr-render annotation on its site Deployments. Access fields (region, endpointURL spelling, credentialsSecretName, archivePollInterval) still follow the profile, so fixing a credentials Secret rolls the pods once. A site added during a refused switch renders the same held identity as the others.

Site Deployments from before bindings existed carry no render annotation. While the write binding is not Bound or Establishing, the render stays deferred as long as any site Deployment lacks a decodable annotation, not only when all of them do: a decodable annotation on one site never authenticates what the others run, so deleting and re-creating one site Deployment does not lift the deferral. Deferred Deployments are not updated at all, and neither are their ConfigMaps (a site's my.cnf keeps its pre-upgrade PITR settings) until a check Job verifies the live key against the archive (log pitr render deferred, and on refusal a Warning PITRArchiveBindingRefused naming the Deployments: "Deployments <names> are not updated until the archive accepts a key: fix the passphrase Secret, switch to a new prefix, or disable PITR for this profile", listing only the unannotated ones). A site created meanwhile renders the held render (or the candidate when no enabled hold exists). A pre-v1.7.0 writer next to an annotated site counts as a running writer, so the write target is authenticated by a check Job like any other. The way out of a refused check is to put the original passphrase back in the Secret, switch to a new prefix, or disable PITR on the profile. Backups do not wait for it: while the render is deferred, a backup of the active profile does not wait (PITRArchiveRolloutPending) and proceeds with no archive stamp (Normal Event PITRArchiveStampSkipped), so it is no point-in-time baseline; take the baseline after the new binding is Bound.

Condition, Events and log lines

The group condition PITRArchiveBindingReady reports the first of:

The condition is removed when the group has no bindings and no targets. Refusal messages name every profile that resolves to the destination (up to 8), the destination, and the procedure.

A group with no PITR target, no binding, no binding condition, no release request and no key snapshot skips the binding pass (no uncached reads), and a group with PITR disabled lists no Deployments to render. When the pass cannot read an input (a non-NotFound API error reading a passphrase Secret), it fails and writes nothing: PITRArchiveBindingReady keeps its stored value instead of flickering to a transient True / Bound. Finished check Jobs that no entry records are deleted after each successful binding status write (with a UID precondition), never before it.

The operator logs pitr archive binding changed (with cause proof, check, recovery, integrity, c3 or input), pitr archive binding released, pitr render annotation ignored and pitr render deferred; see the log schema.

Key snapshots

For a Passphrase binding, the operator copies the profile's passphrase, trimmed, into an immutable Secret mysql-<group>-pitr-key-<h10> with key passphrase. The <h10> suffix is an HMAC under the group's digest key over the destination and the bytes, so it deduplicates (one snapshot per destination and passphrase) without exposing an unkeyed hash of the key. The name proves nothing on its own.

The MySQL pod mounts the snapshot with optional: true: if it is ever missing, mysqld keeps running and only the archiver fails closed. Restore, verification and check Jobs mount it as required.

Snapshots have no owner reference and are never pruned. They outlive the group, because they may hold the only copy of the key an archive was written with. Every pass, the operator checks that each snapshot a Bound binding names still carries the group-UID label, the right destination and an acceptance annotation whose keyCheck verifies its bytes. A snapshot that is deleted is re-created under the same name if your Secret still holds the same bytes (Warning PITRArchiveKeySnapshotRecreated, no roll; if another object of that name appears first, nothing is re-created or reported and the next pass checks that object); otherwise, and for a snapshot whose bytes no longer verify, the binding turns Bound/KeyMaterialLost (Warning PITRArchiveKeyMaterialLost). The operator never deletes a snapshot for that. The message depends on what is left:

• The snapshot is missing. "...; Secret <s> held the accepted key of archive <id>: restore it from a backup of it, or set a profile resolving to <dest> to the original passphrase so the operator re-creates it (MySQL keeps running; the archiver fails closed)". Restore the Secret from a backup of it, or put the original bytes back into your Secret.
• The snapshot is present but fails the check (labels, annotations, destination or bytes changed). "...; Secret <s> no longer passes the check of the accepted key of archive <id>: do not delete it, it may hold the only copy of that key; restore its labels and annotations, or its bytes, from a backup of it; delete it (the operator then re-creates it from a profile's passphrase) only once the original passphrase is safely kept elsewhere and a profile resolving to <dest> holds it (MySQL keeps running; the archiver fails closed)". Never delete it first: it may hold the only copy of the key.

If the group's digest key was re-created in between, the old name cannot be reproduced and only a restored copy of the Secret helps.

List a group's snapshots by metadata only. Record the group's UID somewhere durable: after the group is deleted it is the only selector that tells its snapshots apart from those of a same-name successor.

GROUP_UID=$(kubectl get mfg orders -o jsonpath='{.metadata.uid}')
kubectl get secrets -l app.kubernetes.io/name=mysql-pitr-key,bloodraven.shipstream.io/failover-group-uid="$GROUP_UID" \
  -o 'custom-columns=NAME:.metadata.name,BINDING:.metadata.labels.bloodraven\.shipstream\.io/pitr-binding-id,DESTINATION:.metadata.annotations.bloodraven\.shipstream\.io/pitr-destination'

Do not name the variable UID: bash and zsh define UID as the read-only user ID of the shell, so the assignment fails and $UID expands to a number that matches no snapshot.

If the group is already gone and you did not record its UID, list by the group name and read the UID label of each snapshot before deleting anything:

kubectl get secrets -l app.kubernetes.io/name=mysql-pitr-key,bloodraven.shipstream.io/failover-group=orders \
  -o 'custom-columns=NAME:.metadata.name,GROUP_UID:.metadata.labels.bloodraven\.shipstream\.io/failover-group-uid,DESTINATION:.metadata.annotations.bloodraven\.shipstream\.io/pitr-destination'

Status loss

The bindings are reconstructible, not a second source of truth, for the same group object. If status.pitrArchiveBindings is lost while the group keeps its UID (a manual or tool-made edit of the status subresource), the operator rebuilds every accepted encrypted binding from its snapshot's acceptance annotation after an uncached read confirms the snapshot (it exists, is not terminating, and its bytes verify against the recorded keyCheck). No Job runs and no pod changes. When several accepted snapshots exist for one destination, the newest acceptedAt wins; bindings for destinations in use are rebuilt first, and any that do not fit in 16 slots are reported by BindingLimitReached. A malformed acceptance annotation is never used (Warning PITRArchiveKeySnapshotReadFailed) and never deleted.

Recovery is scoped by the group-UID label. A group that is deleted and created again from a manifest (a backup/restore of the CR, a GitOps re-create) has a new UID, so it is a new group: none of its predecessor's snapshots are rebuilt into its status, and writing the predecessor's archive is refused ArchiveForeign (its descriptor names the old group; the profile is read-only for the new one). Give the recreated group's PITR profile a new prefix, and take a full backup once its binding is Bound. To read the predecessor's history (a restore into a new group from it), keep the predecessor's Secret or point the reading profile's encryption.passphraseSecretKeyRef at the predecessor's retained snapshot (key: passphrase); find those snapshots by the old UID, or by the group name as shown above. Record the UID before deleting a group.

Plaintext bindings have no snapshot and are not durable. Losing them is harmless: the descriptor records mode None, so any reader or writer given a key refuses EncryptionMismatch. After a loss they are proven again by the sidecar report (write) or a check Job (read), and the group keeps rendering the same identity meanwhile. The one exception is the initFromBackup archive profile after its restore succeeded: with no binding for its destination it stays read-only until a binding shows whose archive it is, so a lost plaintext read binding cannot let it write into another group's archive.

Releasing a binding

Release removes a binding and deletes its key snapshots, including snapshots of candidates that were never accepted. Annotate the group with one binding id, and the group's own name and uid as confirmation. The id is derived from the destination, so it recurs in a re-created group of the same name; the uid keeps a request left over for an earlier group from releasing the new group's binding:

kubectl annotate mfg orders \
  bloodraven.shipstream.io/release-pitr-archive-binding="pab-0123456789abcdef:confirm=orders:uid=$(kubectl get mfg orders -o jsonpath='{.metadata.uid}')"

The release is refused (Warning PITRArchiveBindingReleaseRefused; the annotation stays and is retried every minute) while the binding's destination is still a write or read target, or any of its snapshots is named by a site Deployment's render annotation, or the destination itself is what a site Deployment renders ("Deployment <d> renders <destination>"), or a site Deployment carries a bloodraven.shipstream.io/pitr-render annotation that does not decode ("... so what it renders cannot be told; repair or remove it"), or a site Deployment's pod template references a snapshot (even scaled to zero), or a pod in the namespace that has not finished references one (through a mounted or projected volume, an env secretKeyRef or envFrom, including an old ReplicaSet mid-roll and restore, verification, check, staging and maintenance pods, such as a PITR history repair maintenance pod), used by a running Job labelled bloodraven.shipstream.io/pitr-key-secret, or held by a live verification (verification-hold.bloodraven.shipstream.io/<uid>). Snapshots are deleted with a resourceVersion precondition, so a verification that takes its hold at the same moment wins and the release is refused again. A malformed value or a wrong confirm or uid is rejected (Warning PITRArchiveBindingReleaseRejected) and the annotation is removed. On success the entry is removed, the annotation is removed, and a Normal PITRArchiveBindingReleased is recorded.

Deleting the group leaves its snapshots in place. Release what you no longer need before deleting the group, or afterwards delete them by label once no archive needs them, using the UID you recorded (see Key snapshots):

kubectl delete secret -l app.kubernetes.io/name=mysql-pitr-key,bloodraven.shipstream.io/failover-group-uid="$GROUP_UID"

Read-only profiles

A profile is read-only, and is never used to write backups, when it is the initFromBackup.pointInTime.archiveProfileName and the restore has not succeeded yet, or when its destination's binding belongs to another group (source.uid differs) or is refused ArchiveForeign or ArchiveCopy. A schedule on it creates no CronJob (an existing one is deleted) and records a Warning BackupScheduleRefused; a MysqlBackup of it fails with ProfileReadOnly.

A third case also outlives the restore: the initFromBackup archive profile stays read-only after the restore succeeded when its destination has no binding at all (for instance a plaintext read binding that was lost with the status, which is not rebuilt): "profile <p> is the initFromBackup point-in-time archive profile and <dest> has no archive binding: ...". It clears once a binding shows whose archive it is, by the writer check of a write target.

The second condition outlives the restore. A group restored from another group's archive keeps that archive's binding (Bound, source.uid of the other group) after the restore succeeds, like any Bound binding, so the profile stays read-only for as long as the binding exists. To back up to that profile again, point it at a new prefix, or release the foreign binding once you no longer restore or verify from that archive through this group.

Dump-key snapshots

Each MysqlBackup of an encrypted profile gets its own immutable dump-key snapshot, created once before its Job: the Secret mysqlbackup-<name>-dump-key-<uid prefix> (labels app.kubernetes.io/name=mysql-dump-key, bloodraven.shipstream.io/mysqlbackup, bloodraven.shipstream.io/mysqlbackup-uid, bloodraven.shipstream.io/failover-group-uid), holding the trimmed bytes the profile's Secret had at that moment. The backup Job mounts it, and status.artifactStorage.passphraseSecretKeyRef records it, so restore, verification and artifact cleanup decrypt with exactly the key the dump was written with, whatever happens to the profile later. Pod retries of the Job reuse it; it is never re-created with other bytes. If the name is held by an object of another backup, the backup waits Pending with DumpKeySnapshotConflict.

Dump-key snapshots have no owner reference. The backup's finalizer deletes the snapshot only after a cleanup Job confirms the backup's artifacts are deleted. When cleanup is skipped because the group or profile is gone, or because the backup recorded neither its producing Job nor its artifact destination (a status update lost after the Job was created can hide a Job that uploaded encrypted artifacts), the snapshot is kept and the Warning ArtifactCleanupSkipped names it (… is kept because encrypted artifacts may exist; delete it once you have confirmed none remain). Artifact cleanup of an abandoned backup decrypts the stamp with the producing Job's key, else the recorded snapshot; if the recorded snapshot is missing the cleanup blocks with PassphraseSecretMissing naming it. The profile's live Secret is used only for a backup that recorded no key at all.

Shared Secrets do not roll pods

The pod fingerprint reads only what pods consume: ca.crt, tls.crt and tls.key of the TLS Secret, username and password of the credential Secrets, and the operator Secret's MYSQL_ROOT_PASSWORD. A passphrase stored in the same Secret as TLS material or credentials can therefore be edited without rolling the MySQL pods; the edit shows up only as KeyChangeRefused when that passphrase is in archive use.

Rotation

There is no in-place re-encryption and no in-place key rotation of a PITR archive.

What is refused. While spec.backup.pitr.enabled is true, the API server rejects a change to the active profile's encryption (adding or removing it, algorithm, passphraseSecretKeyRef.name or key) unless a storage field of the profile changes in the same edit:

spec.backup.profiles[pitr.profileName].encryption (presence, algorithm, passphraseSecretKeyRef) cannot change while that profile archives PITR binlogs to the same storage; create a new profile with a new storage prefix and its own passphrase, switch spec.backup.pitr.profileName to it, wait until its archive binding is Bound, then take a full backup (see Backup encryption > Rotation)

This includes renaming the Secret to a copy with the same bytes. Omitting algorithm or key is the same as their defaults, so that is not a change. The rule compares storage fields as written: a spelling-only change of a storage field (prefix: pitr to pitr/, https://S3.example:443 to https://s3.example) is the same destination, so an encryption change bundled with one is admitted but refused at reconcile, like the cases below. Everything the admission rule cannot see is refused at reconcile, on the condition, without touching pods:

• editing the bytes of the Secret in place (KeyChangeRefused);
• disabling PITR, changing encryption or the Secret, and re-enabling it (EncryptionChangeRefused or KeyChangeRefused; a rename to identical bytes is accepted);
• switching pitr.profileName to another profile on the same destination (an alias) whose encryption or bytes differ (named in the message with the original profile); an alias with identical bytes and encryption is accepted and does not roll pods;
• removing the profile and adding it back with other bytes.

Whitespace-only edits (leading or trailing \n, \r, space, tab) are accepted. A refused edit leaves the pods on the accepted key snapshot, including pods that restart afterwards, so archiving continues. Revert the edit to clear the condition.

Procedure. To move the archive to a new key:

1. Create a new Secret with the new passphrase.
2. Add a new profile whose storage has a new prefix (or sub-path) and whose encryption.passphraseSecretKeyRef names the new Secret.
3. Switch spec.backup.pitr.profileName to it. A check Job finds the new prefix empty, the sites roll once, the sidecar creates the new descriptor, and the new binding becomes Bound. The old binding is kept.
4. Once the new binding is Bound (PITRArchiveBindingReady=True), take a full backup. A backup taken earlier is stamped with the archive the sites rendered at the time, not the new one. Until one is stamped with the new archive ID, the condition message says so: point-in-time restores need a baseline.
5. Keep the old binding, its snapshot and the old prefix for as long as you may restore from the old history. This group's point-in-time verifications read it through a profile on the old prefix (archiveProfileName), even after you delete your old Secret, because they mount the retained snapshot. A restore runs in a new group (initFromBackup), which cannot discover this group's snapshots: its archive profile's encryption.passphraseSecretKeyRef must hold the old passphrase. Either keep your old Secret, or have the new group's archive profile name this group's retained snapshot (passphraseSecretKeyRef.name: mysql-<group>-pitr-key-<h10>, key: passphrase; it must be in the new group's namespace). Read the snapshot's name from the old binding's encryption.keySecretName before you delete anything.
6. When nothing needs the old archive, release the old binding by id.

Full-backup dumps are simpler: each backup keeps its own dump-key snapshot, so pointing a dump-only profile at a new Secret affects only later backups. Rotate restore-side spec.initFromBackup.decryption / spec.restoreInPlace.decryption references the same way during the transition window.

A profile used both for dumps and for the PITR archive is the exception: after a refused rotation, new dumps are encrypted with the new bytes while the archive keeps the accepted key. Use the procedure above instead.

Plaintext objects after enabling encryption

A cluster that already had backups and then enabled encryption mid-life will have a mix of plaintext and encrypted objects in the same prefix. Restore and verification use each backup's recorded encryption attribution: plaintext backups load directly, and encrypted backups use decryption. Over time retention ages out the plaintext backups; you do not need to re-upload or hand-encrypt them.

Apart from that attribution, encrypted readers reject objects without BRV1 magic by default with ErrTamperedOrDowngrade; restore, verification, and PITR do not automatically pass plaintext through. The operator does not render a plaintext-fallback flag. The sidecar's manual compatibility flag is BLOODRAVEN_PITR_ALLOW_PLAINTEXT_FALLBACK=1; the decrypt-download command uses BLOODRAVEN_ALLOW_PLAINTEXT_FALLBACK=1, which generated restore and verification Jobs do not set. These flags weaken downgrade protection and do not decrypt ciphertext written with another passphrase. A missing BRV1 magic error requires checking the source artifact and its attribution before retrying.

Threat model boundaries

In scope:

• Confidentiality of backup dump contents and archived binlog data against anyone who can read S3 objects / PVC files but does not hold the passphrase Secret.
• Tamper detection via AES-GCM: modified ciphertext fails to decrypt, so a corrupted or attacker-modified object cannot be silently loaded.
• Truncation detection via the final-flag nonce byte.

Out of scope:

• Object-name confidentiality (S3 keys and PVC filenames are plaintext by design).
• Backup size leakage within one chunk of granularity (64 KiB by default).
• Protection of the MysqlBackup CR's metadata — .status.artifactStorage.location, size, GTID coordinates are plaintext on the Kubernetes API.
• Attacks that compromise the operator's ServiceAccount or the passphrase Secret directly. Those are covered by the broader Kubernetes threat model, not by backup encryption.

Key snapshots are a second copy of the key. The archive and dump-key snapshots hold the same bytes as your passphrase Secret, in the same namespace, behind the same Secret RBAC. Anyone who can read Secrets in the namespace can read them. The acceptance annotation's keyCheck is a passphrase verifier (an HMAC over the descriptor's identity), so it lives only on the snapshot Secret and never in the group's status. Snapshot names are keyed HMACs and reveal nothing about the key without the group's digest key.

Compatibility matrix

The wire format identifies ciphertext; plaintext compatibility still requires explicit fallback as described above.

Known limitations

• No in-place key rotation for a PITR archive: rotation is a new prefix (Rotation). A group holds at most 16 archive bindings.
• Key snapshots outlive the group and are never pruned; release them by id or delete them by label (Releasing a binding).
• A MysqlStandbyCluster standby reader still reads the live bytes of its source.decryption Secret, not a snapshot. A profile used for both dumps and the PITR archive, after a refused rotation, has dumps and archive under different keys, which a single-key reader cannot open.
• Keep the original passphrase for every retained archive generation. Plaintext fallback cannot recover a lost key or decrypt another passphrase's ciphertext. Use compatible reader/writer images and preserve old keys and prefixes during migration; do not enable fallback merely to suppress an integrity error.
• Encryption does not repair missing history or ambiguous provenance. Authentication/decryption failures retain the affected bytes and can defer the site's new retirements. Repair access/key configuration or follow PITR history repair; the PITR limitations also apply to encrypted archives.