Verify, and restore

playground takes a nightly dump, ships sealed binlogs, and has a retention policy, and the counter app has been reading and writing through it throughout. You have written a great deal of backup configuration and read back none of it. So you cannot answer the only question that matters at 3am: does last night’s artifact load?

Schrödinger backups

The API type says it plainly. A verification “restores a MysqlBackup artifact into an ephemeral, throwaway MySQL instance to prove the backup can actually be loaded. Unverified backups are schrödinger backups.” Until something reads it, last night’s object in S3 is not a backup. It is a file that is the right size.

MysqlBackupVerification is the CR that reads it, and the mechanism is deliberately unglamorous. The run gets an emptyDir datadir on the Job’s node — dedicated per run, gone with the Pod, never a PersistentVolumeClaim — and an in-pod mysqld that binds 127.0.0.1 with no Service, so the instance is unreachable from outside its own network namespace. Nothing can point at it by accident, and nothing it does can touch playground. The datadir is not size-limited: it draws on the node’s ephemeral storage, so budget roughly 1.5 × the backup’s status.sizeBytes, and bound a run with spec.resources.limits.ephemeral-storage (the kubelet evicts the Pod past it and the verification fails). On failure, keepOnFailure (default true) leaves the Job and its Pod in place so you can kubectl exec in and look at the wreckage.

The sanity check has exact semantics

Loading is the default contract. spec.sanityCheck extends it from “it loaded” to “it loaded something”. The query is a single statement, run through the mysql CLI as mysql -B -N -e wrapped in timeout, and it must return one row and one column. Multi-statement input is rejected to keep the timeout budget predictable.

The detail that earns its keep: an empty result set is treated as scalar 0. That is precisely the shape a silently-empty restore produces — the query runs, nothing errors, no rows come back. Without that rule, “returned nothing” would look like success. With it, a minResult: 1 floor catches it.

Failures split into two reasons, not one. SanityCheckFailed means the scalar came back below sanityCheck.minResult, or the query errored — the data is wrong. SanityCheckTimeout means the query ran longer than sanityCheck.timeout (default 60s; admission accepts 1s to 1h) — the instance is wedged. Different problems, different pagers. status.sanityCheck.outcome records which one you got: Passed, Failed, TimedOut, or Skipped when the load or the replay failed first and the query never ran.

Pick an assertion whose expected value is structural rather than a moving row count. SELECT COUNT(*) FROM information_schema.schemata WHERE schema_name = 'counter_db' returns 1 when the schema landed and 0 when it did not, and minResult: 1 turns that into a pass/fail.

Created MysqlBackupVerification bloodraven-playground/playground-nightly-verify-9k2rt (profile=nightly, triggeredBy=manual)
Watch with: kubectl get mysqlbackupverification playground-nightly-verify-9k2rt -n bloodraven-playground -w

Succeeded Passed 1

What a Succeeded proves — and what it does not

Be exact here, because this is the sentence people quote back at you in an incident review. A Succeeded proves two things and no more: the artifact loads into a real mysqld, and your chosen scalar assertion held.

It does not prove logical equivalence with the live primary — that is a different tool’s job. It does not rehearse your application’s writes or a traffic cutover. Verification is not a DR drill. It is the cheap, automatable half of the problem, and the half that fails silently.

The strongest argument for running it is this project’s own history: the verifier shipped broken. Chaos scenario 31 failed with ERROR 1062 Duplicate entry because the ephemeral verify mysqld was started with gtid_mode=OFF, which defeats server-side GTID dedup on replay, so cross-site archived binlogs double-applied (issue #101, fixed by PR #105). The mechanism that proves your backups are good was itself wrong — an argument for running verification, not against it. A broken verifier is discovered by running it, and by nothing else.

Restore is two fields, not a CR

Operators reach for a MysqlRestore CR. There is no restore CR. Restore is two fields on the failover group, and choosing between them is the whole decision:

An in-place restore runs against a live cluster, so it walks discrete phases one step per reconcile, and an operator restart always lands on a well-defined observable state.

Fencing is the phase you already understand. Fencing stamps role = "fenced", which matches neither the -primary selector nor the -replicas selector, so the site sheds its Service endpoints rather than serving a half-loaded database. Cross-site mutation is suppressed wholesale while a restore is in flight — no failover is going to fire underneath your load.

The anti-fat-finger token

spec.restoreInPlace.confirm is required, must parse as an RFC 3339 timestamp, and must be strictly greater than the timestamp in status.restoreInPlace.confirmTokenUsed. Programmatic callers just send now().

Look at what that buys. Re-applying last week’s manifest does nothing: its token is no longer greater than the recorded one, so the reconciler sees a terminal status that already reflects that confirm value and returns. GitOps cannot replay a destructive restore. The API server refuses a token that is not an RFC 3339 timestamp, so an invalid token never reaches the operator; only the strictly-greater comparison happens in the reconciler.

Why pointInTime gets rejected, and where it does not exist

The two restore fields treat pointInTime differently.

On spec.initFromBackup, ask for pointInTime while spec.backup.pitr.enabled=false, without a pointInTime.archiveProfileName naming the source’s archive, and you get:

pointInTime is set but spec.backup.pitr.enabled=false; PITR restore requires the failover group to have continuous binlog archival configured on the source, or pointInTime.archiveProfileName

It is not a style preference: replay material only exists if something archived it, and the restore has to be told where. A group that does not archive itself (a fresh DR group, say) can still replay the source’s binlogs by naming that archive in archiveProfileName.

spec.restoreInPlace has no pointInTime field at all (releases before v1.7.0 accepted one and refused it). The archived binlogs carry their original GTIDs, and the live primary has already executed every one of them, so MySQL would skip the whole replay and the restore would quietly stop at the backup. To get back to a point in time, restore into a new failover group with spec.initFromBackup.pointInTime.

Where the initFromBackup rejection surfaces matters more than the wording. It is a reconciliation-time check, not an admission rejection. Your kubectl apply succeeds. The failure arrives seconds later in the CR’s status — status.restore with a RestoreBuildFailed Warning Event, before any restore Job exists. Look there, not at the exit code of the apply.

Handoff

GitLab, January 2017: five backup mechanisms, none usable. pg_dump silently failing on a version mismatch, empty S3 uploads, misconfigured alert emails. They recovered from an incidental six-hour-old staging snapshot. Every one of those five would have passed a config review — the YAML was fine, the cron fired, the bucket existed. None would have passed a verification, because not one had ever been read back.

You can now verify a backup for playground, restore it in place through the confirm token, and state precisely what your verification did and did not prove. Which leaves the artifact itself: a dump of your customers’ playground, sitting in an object store, in plaintext. What happens when you encrypt the data at rest instead?

Mark as read