Verify, and restore
An unverified backup is a schrödinger backup. Prove it loads, then restore playground in place behind a confirmation token that has to be a timestamp.
By the end of this topic you can
- Run a backup verification and say precisely what a
Succeededresult proved - Restore
playgroundin place using the RFC 3339 confirmation token - Explain why a
pointInTimerequest is rejected when PITR is disabled
playground takes a nightly dump, ships sealed binlogs, and has a retention policy, and the counter app has been reading and writing through it throughout. You have written a great deal of backup configuration and read back none of it. So you cannot answer the only question that matters at 3am: does last night’s artifact load?
Schrödinger backups
The API type says it plainly. A verification “restores a MysqlBackup artifact into an ephemeral, throwaway MySQL instance to prove the backup can actually be loaded. Unverified backups are schrödinger backups.” Until something reads it, last night’s object in S3 is not a backup. It is a file that is the right size.
MysqlBackupVerification is the CR that reads it, and the mechanism is deliberately unglamorous. The run gets an ephemeral PVC as its datadir — dedicated per run, deleted on cleanup, lifecycle never shared with the backup PVC — and an in-pod mysqld that binds 127.0.0.1 with no Service, so the instance is unreachable from outside its own network namespace. Nothing can point at it by accident, and nothing it does can touch playground. The PVC is auto-sized to max(10Gi, ceil(1.5 × backup.status.sizeBytes / 10Gi) × 10Gi) — a 10 GiB floor, because a fresh datadir plus a small dump is already a few hundred megabytes. On failure, keepOnFailure (default true) leaves the Pod and PVC in place so you can kubectl exec in and look at the wreckage.
The sanity check has exact semantics
Loading is the default contract. spec.sanityCheck extends it from “it loaded” to “it loaded something”. The query is a single statement, run through the mysql CLI as mysql -B -N -e wrapped in timeout, and it must return one row and one column. Multi-statement input is rejected to keep the timeout budget predictable.
The detail that earns its keep: an empty result set is treated as scalar 0. That is precisely the shape a silently-empty restore produces — the query runs, nothing errors, no rows come back. Without that rule, “returned nothing” would look like success. With it, a minRows: 1 floor catches it.
Failures split into two reasons, not one. SanityCheckFailed means the scalar came back below expect.minRows, or the query errored — the data is wrong. SanityCheckTimeout means the query exceeded expect.maxDurationSeconds (default 60) — the instance is wedged. Different problems, different pagers.
Pick an assertion whose expected value is structural rather than a moving row count. SELECT COUNT(*) FROM information_schema.schemata WHERE schema_name = 'counter_db' returns 1 when the schema landed and 0 when it did not, and minRows: 1 turns that into a pass/fail.
kubectl bloodraven verify-backup playground --profile nightlyCreated MysqlBackupVerification bloodraven-playground/playground-nightly-verify-9k2rt (profile=nightly, triggeredBy=manual)
Watch with: kubectl get mysqlbackupverification playground-nightly-verify-9k2rt -n bloodraven-playground -wkubectl get mysqlbackupverification playground-nightly-verify-9k2rt -o jsonpath='{.status.phase} {.status.sanityCheck.ran} {.status.sanityCheck.resultRow}'Succeeded true 1What a Succeeded proves — and what it does not
Be exact here, because this is the sentence people quote back at you in an incident review. A Succeeded proves two things and no more: the artifact loads into a real mysqld, and your chosen scalar assertion held.
It does not prove logical equivalence with the live primary — that is a different tool’s job. It does not rehearse your application’s writes or a traffic cutover. Verification is not a DR drill. It is the cheap, automatable half of the problem, and the half that fails silently.
The strongest argument for running it is this project’s own history: the verifier shipped broken. Chaos scenario 31 failed with ERROR 1062 Duplicate entry because the ephemeral verify mysqld was started with gtid_mode=OFF, which defeats server-side GTID dedup on replay, so cross-site archived binlogs double-applied (issue #101, fixed by PR #105). The mechanism that proves your backups are good was itself wrong — an argument for running verification, not against it. A broken verifier is discovered by running it, and by nothing else.
Restore is two fields, not a CR
Operators reach for a MysqlRestore CR. There is no restore CR. Restore is two fields on the failover group, and choosing between them is the whole decision:
| Field | Shape | Use it to |
|---|---|---|
spec.initFromBackup | one-shot; gates normal bootstrap, skipped after success even if left in place | seed a brand-new group from an artifact |
spec.restoreInPlace | re-runnable; no teardown-and-rename cycle | repair the live playground you already have |
An in-place restore runs against a live cluster, so it walks discrete phases one step per reconcile, and an operator restart always lands on a well-defined observable state.
Fencing is the phase you already understand. Fencing stamps role = "fenced", which matches neither the -primary selector nor the -replicas selector, so the site sheds its Service endpoints rather than serving a half-loaded database. Cross-site mutation is suppressed wholesale while a restore is in flight — no failover is going to fire underneath your load.
The anti-fat-finger token
spec.restoreInPlace.confirm is required, must parse as an RFC 3339 timestamp, and must be strictly greater than the timestamp in status.restoreInPlace.confirmTokenUsed. Programmatic callers just send now().
Look at what that buys. Re-applying last week’s manifest does nothing: its token is no longer greater than the recorded one, so the reconciler sees a terminal status that already reflects that confirm value and returns. GitOps cannot replay a destructive restore. An invalid token on a re-arm emits a RestoreInPlaceRejected event and leaves the previous terminal status visible.
Why pointInTime gets rejected
Ask for pointInTime while spec.backup.pitr.enabled=false and you get:
pointInTime is set but spec.backup.pitr.enabled=false; PITR restore requires the failover group to have continuous binlog archival configured on the source
Both entry points share one builder, so the rejection is identical for spec.initFromBackup.pointInTime and spec.restoreInPlace.pointInTime. It is not a style preference: replay material only exists if something archived it, and with PITR off nothing did.
Where it surfaces matters more than the wording. This is a reconciler error, not an admission rejection. Your kubectl apply succeeds. The failure arrives seconds later, in the CR’s status and the operator log. Look there, not at the exit code of the apply.
Handoff
GitLab, January 2017: five backup mechanisms, none usable. pg_dump silently failing on a version mismatch, empty S3 uploads, misconfigured alert emails. They recovered from an incidental six-hour-old staging snapshot. Every one of those five would have passed a config review — the YAML was fine, the cron fired, the bucket existed. None would have passed a verification, because not one had ever been read back.
You can now verify a backup for playground, restore it in place through the confirm token, and state precisely what your verification did and did not prove. Which leaves the artifact itself: a dump of your customers’ playground, sitting in an object store, in plaintext. What happens when you encrypt the data at rest instead?
Flashcards
Schrödinger backup
A backup nothing has ever read back. Until something loads it, the artifact is not known to be a backup — it is a file of about the right size.
Where does the ephemeral verification mysqld listen, and what Service fronts it?
It binds 127.0.0.1 only, and no Service is created — the instance is unreachable from outside its own Pod network namespace.
Minimum size of the ephemeral verification datadir PVC
10 GiB. Auto-sizing is max(10Gi, 1.5 × the backup's sizeBytes rounded up to the nearest 10Gi).
Verification terminal reason: SanityCheckFailed
The sanity query returned a scalar below expect.minRows (or errored outright) — the data is wrong.
Verification terminal reason: SanityCheckTimeout
The sanity query exceeded expect.maxDurationSeconds (default 60) — the instance is wedged rather than wrong.
A verification sanity query runs cleanly but returns zero rows. What scalar does the check compare against minRows?
- An empty result set is deliberately treated as scalar 0, so a silently-empty restore fails instead of passing.
spec.initFromBackup
The one-shot restore entry point: it gates normal bootstrap of a brand-new group and is skipped on later reconciles once it has succeeded, even if the field is left in place.
spec.restoreInPlace
The re-runnable restore entry point: it loads a dump into the currently-active primary of a live group, with no teardown-and-rename cycle.
The in-place restore phases, in order
Preflight, Fencing, Restoring, Resuming, then the terminal Succeeded or Failed.
Format and acceptance rule for spec.restoreInPlace.confirm
A required RFC 3339 timestamp that must be strictly greater than the value recorded in status.restoreInPlace.confirmTokenUsed.
You set pointInTime on a restore while spec.backup.pitr.enabled=false
Rejected, identically for both spec.initFromBackup and spec.restoreInPlace: PITR restore requires continuous binlog archival configured on the source.
spec.keepOnFailure on a MysqlBackupVerification — default, and what it preserves
Defaults to true; it leaves the verification Pod and its ephemeral PVC in place after a Failed run so you can exec in and inspect why the load failed.
Quiz
Show answer
Answer: The artifact loads into a real mysqld, and your scalar assertion held. Nothing about the live primary.
A verification restores the artifact into an ephemeral throwaway mysqld and runs whatever scalar you asserted. That is the entire contract, and it says nothing about the live primary. Logical equivalence with the primary is explicitly out of scope — it needs a different tool, and the verification never contacts the primary to compare. An RTO claim is wrong twice over: the verification loads into a cold, isolated instance with its own PVC and no Service, not into your production path, so its wall clock is not your recovery time. And application-level rehearsal of writes or traffic cutover is the thing verification deliberately does not do — that is a DR drill. A Succeeded is the cheap half of the problem, and the half that otherwise fails silently. (objective 4)
A verification restores the artifact into an ephemeral throwaway mysqld and runs whatever scalar you asserted. That is the entire contract, and it says nothing about the live primary. Logical equivalence with the primary is explicitly out of scope — it needs a different tool, and the verification never contacts the primary to compare. An RTO claim is wrong twice over: the verification loads into a cold, isolated instance with its own PVC and no Service, not into your production path, so its wall clock is not your recovery time. And application-level rehearsal of writes or traffic cutover is the thing verification deliberately does not do — that is a DR drill. A Succeeded is the cheap half of the problem, and the half that otherwise fails silently. (objective 4)
Show answer
Answer: spec.restoreInPlace, with confirm set to the current RFC 3339 timestamp.
spec.restoreInPlace is the re-runnable entry point that operates against a live cluster: it fences, loads into the active primary, and resumes. spec.initFromBackup is one-shot and gates bootstrap — on an existing group that already bootstrapped, it is skipped, so setting it does nothing. There is no MysqlRestore CR; restore is two fields on the failover group, and reaching for a CR that does not exist is the most common wrong first move here. Deleting and recreating the group is exactly the teardown-and-rename cycle restoreInPlace was built to avoid. (objective 5)
spec.restoreInPlace is the re-runnable entry point that operates against a live cluster: it fences, loads into the active primary, and resumes. spec.initFromBackup is one-shot and gates bootstrap — on an existing group that already bootstrapped, it is skipped, so setting it does nothing. There is no MysqlRestore CR; restore is two fields on the failover group, and reaching for a CR that does not exist is the most common wrong first move here. Deleting and recreating the group is exactly the teardown-and-rename cycle restoreInPlace was built to avoid. (objective 5)
Show answer
Answer: False
The reversal: a re-applied manifest re-runs nothing, and that is the whole point of the token design. confirm must be an RFC 3339 timestamp strictly greater than status.restoreInPlace.confirmTokenUsed. After a run, the executed token is recorded, so the identical manifest now carries a token that is not greater — the reconciler sees a terminal status that already reflects it and returns. The same protection covers the older failure mode of applying last month's manifest by accident. To restore again you must deliberately bump confirm forward, which is a decision no sync loop makes for you. (objective 5)
The reversal: a re-applied manifest re-runs nothing, and that is the whole point of the token design. confirm must be an RFC 3339 timestamp strictly greater than status.restoreInPlace.confirmTokenUsed. After a run, the executed token is recorded, so the identical manifest now carries a token that is not greater — the reconciler sees a terminal status that already reflects it and returns. The same protection covers the older failure mode of applying last month's manifest by accident. To restore again you must deliberately bump confirm forward, which is a decision no sync loop makes for you. (objective 5)
Show answer
Answer: In the CR status and the operator log, seconds after a kubectl apply that succeeded.
The check lives in the reconciler's PITR fragment builder, not in the admission chain, so the write is accepted and the error surfaces afterwards on the object and in the operator log — watch there, not at the exit code of your apply. It is not an admission webhook rejection and not a CEL rule: neither runs this check, which is why the apply comes back clean. Nor does it wait for the restore Job — the builder returns the error while assembling the Job's init containers, so no Job is ever created. The same rejection covers spec.initFromBackup.pointInTime, because both entry points share one builder. (objective 6)
The check lives in the reconciler's PITR fragment builder, not in the admission chain, so the write is accepted and the error surfaces afterwards on the object and in the operator log — watch there, not at the exit code of your apply. It is not an admission webhook rejection and not a CEL rule: neither runs this check, which is why the apply comes back clean. Nor does it wait for the restore Job — the builder returns the error while assembling the Job's init containers, so no Job is ever created. The same rejection covers spec.initFromBackup.pointInTime, because both entry points share one builder. (objective 6)
Show answer
Answer:
It argues for running verification, not for distrusting it. The bug was in the verifier itself: the throwaway mysqld ran gtid_mode=OFF, so replayed archived binlogs re-applied transactions the dump already contained and the run died on a duplicate key. That defect was invisible in configuration review — the backup profile, the archiver and the CR schema were all correct — and it could only be found by actually restoring an artifact and watching it fail. The general lesson is that the mechanism which proves your backups are loadable is itself a piece of software that can be wrong, and the only way to discover that is to exercise it on a schedule against real artifacts, and to treat a Failed verification as information about the whole chain rather than assuming the backup is at fault.
A full-credit answer shows: A strong answer covers: (1) the conclusion is 'run verification more, not less'; (2) the bug was in the verify path, not in the backup artifact; (3) it was undetectable by config review and only surfaced by executing a real restore; (4) ideally, that a Failed verification implicates the whole chain — artifact, replay path, and verifier — so triage should not stop at the dump. Reject answers concluding that verification is unreliable and should be skipped, or that the backups themselves were corrupt.
The verifier shipping broken is the strongest available argument for verification, because it is a defect in exactly the class of thing verification exists to catch: something that looks correct in YAML and only misbehaves when data actually moves. Scenario 31 caught it (issue #101, fixed in PR #105). Nothing else would have. (objective 4)
Sample answer
It argues for running verification, not for distrusting it. The bug was in the verifier itself: the throwaway mysqld ran gtid_mode=OFF, so replayed archived binlogs re-applied transactions the dump already contained and the run died on a duplicate key. That defect was invisible in configuration review — the backup profile, the archiver and the CR schema were all correct — and it could only be found by actually restoring an artifact and watching it fail. The general lesson is that the mechanism which proves your backups are loadable is itself a piece of software that can be wrong, and the only way to discover that is to exercise it on a schedule against real artifacts, and to treat a Failed verification as information about the whole chain rather than assuming the backup is at fault.
A full-credit answer shows
A strong answer covers: (1) the conclusion is 'run verification more, not less'; (2) the bug was in the verify path, not in the backup artifact; (3) it was undetectable by config review and only surfaced by executing a real restore; (4) ideally, that a Failed verification implicates the whole chain — artifact, replay path, and verifier — so triage should not stop at the dump. Reject answers concluding that verification is unreliable and should be skipped, or that the backups themselves were corrupt.
The verifier shipping broken is the strongest available argument for verification, because it is a defect in exactly the class of thing verification exists to catch: something that looks correct in YAML and only misbehaves when data actually moves. Scenario 31 caught it (issue #101, fixed in PR #105). Nothing else would have. (objective 4)