Repair PITR history safely

Use this procedure when the archiver or downloader reports a missing object, a divergent boundary, or GTID namespace reuse. A zero backlog means local uploads caught up; it does not prove that historical recovery is complete.

Some archive states cannot be repaired in place. A reset epoch (PITRArchiveHistoryReset), a history-integrity marker (PITRArchiveUnsafeHistory), a reissue after a reclone, a legacy prefix (PITRArchiveLegacy) and a descriptor owned by another group (PITRArchiveForeign, PITRArchiveCopy) all need a new prefix and a full backup.

None of the records below is an acknowledgement. A divergent flag, a quarantine and a canonical-backup acknowledgement change what the archiver does and what the planner may read, but a restore whose window holds lost, errant or quarantined transactions still refuses PITRArchiveDivergentHistory until those transactions are in excludeGtids, or until you use a backup taken after them. See Target history and excludeGtids.

Preserve the evidence and choose a baseline

1. Preserve the site's manifest, all archive objects, the original passphrase, and any surviving local binlogs. Record the error's site, name, ID, predecessor set and missing transaction set. /archiver/status.lastError and increases in bloodraven_archiver_upload_failures expose history problems before restore. Do not clear the manifest or overwrite objects.
2. Choose the canonical live dataset with your incident owner. Take and verify a new full backup of that dataset. Record its pre-dump status.gtidExecuted. Quarantine cannot reconstruct lost transactions. Restores whose target history does not need the row stay available; restores that need it use a backup taken after the quarantine (see the waiver below).
3. Pause PITR writers through your maintenance rollout and wait for the old sidecar processes to stop. Keep MySQL data and archive volumes intact. This prevents a concurrent manifest write from undoing the maintenance edit.

Restore bytes or quarantine one row

If an original object can be recovered, restore those exact bytes at its original key, using the original encryption settings. When a missing legacy row has a verified contentId matching surviving local bytes, the archiver can publish the ID-aware object and atomically replace the dead legacy row. Without that digest, equal names, sizes and headers do not prove replacement.

For an unrecoverable file or an abandoned divergent tail, run the matching operator image's pitr-quarantine command in a maintenance Pod with the same archive environment, writable archive PVC/S3 credentials and mounted key snapshot as the writer. Use the exact storage prefix. For example, this plaintext PVC maintenance Pod moves one legacy manifest row and preserves its object; replace all example values before applying it:

apiVersion: v1
kind: Pod
metadata:
  name: pitr-quarantine
spec:
  restartPolicy: Never
  securityContext:
    fsGroup: 65532 # reuse the archive writer's actual fsGroup
  containers:
    - name: quarantine
      image: your-registry/bloodraven:matching-version
      args:
        - pitr-quarantine
        - --site=iad
        - --name=mysql-bin.000004
        - --id=legacy
        - --reason=Object lost before identity verification; canonical backup verified
        - --resume-gtids=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-200
        - --confirm
      env:
        - {name: BLOODRAVEN_PITR_STORAGE_TYPE, value: PVC}
        - {name: BLOODRAVEN_PITR_MANIFEST_PREFIX, value: orders/binlogs}
        - {name: BLOODRAVEN_PITR_PVC_MOUNT_PATH, value: /archive}
        # Writer identity: copy these from the site's sidecar container.
        - {name: POD_NAMESPACE, value: orders}
        - {name: FAILOVER_GROUP, value: orders}
        - {name: BLOODRAVEN_FAILOVER_GROUP_UID, value: 11111111-2222-3333-4444-555555555555}
        - {name: BLOODRAVEN_PITR_DESTINATION, value: pvc://orders/your-existing-backup-pvc/orders/binlogs}
        - {name: BLOODRAVEN_PITR_ENCRYPTION_MODE, value: None}
      volumeMounts:
        - {name: archive, mountPath: /archive}
  volumes:
    - name: archive
      persistentVolumeClaim:
        claimName: your-existing-backup-pvc

Mount the archive PVC exactly at BLOODRAVEN_PITR_PVC_MOUNT_PATH, not at a parent directory: the command refuses a path that is not a mount point, and checks that it can write there.

pitr-quarantine is a writer, so it needs the same archive identity as the site's sidecar: copy BLOODRAVEN_FAILOVER_GROUP_UID, BLOODRAVEN_PITR_DESTINATION and BLOODRAVEN_PITR_ENCRYPTION_MODE from the sidecar container of the site's MySQL Deployment (kubectl get deploy mysql-<group>-<site> -o yaml), with the group's namespace and name. It checks the descriptor exactly as the sidecar does and refuses an archive this group does not own (PITRArchiveForeign, PITRArchiveCopy), a legacy prefix (PITRArchiveLegacy), or a prefix with no archive (PITRArchiveNotFound). It never creates a descriptor.

For ID-aware rows, pass the complete 64-character archive ID instead of legacy. For encrypted archives, also mount the archive's accepted key snapshot read-only (key passphrase) and set BLOODRAVEN_PITR_PASSPHRASE_FILE to its mounted file. That is the Secret the writers mount, which may differ from the profile's current Secret after a refused edit. Read its name from the group's Bound binding for the destination:

kubectl get mfg <group> -o json | jq -r --arg d '<destination>' \
  '.status.pitrArchiveBindings[] | select(.destination == $d and .state == "Bound") | .encryption.keySecretName'

(kubectl's JSONPath filters accept one comparison, not &&, hence jq.) An empty result means the destination has no Bound binding: the key cannot be read from status, so stop and resolve the binding first.

Copy <destination> exactly as status.pitrArchiveBindings[].destination shows it (the normalized form, for example s3://<bucket>/<prefix>?endpoint=<escaped endpoint>). S3 uses the existing BLOODRAVEN_PITR_S3_* and BLOODRAVEN_PITR_AWS_CREDS_DIR configuration. Never pass Secret bytes as command arguments.

A maintenance Pod that references a key snapshot (a mounted or projected volume, an env secretKeyRef or envFrom) blocks releasing that binding for as long as it has not finished, whatever its owner; delete it when the repair is done.

The command writes a quarantined record containing the original row, reason, time and resume GTIDs. It never calls object deletion. The replay planner treats a quarantined row as follows:

• Never data. A quarantined row is never replayed. A restore whose target history needs its transactions refuses PITRArchiveGap ("it is in quarantined row <site/name>; un-quarantine it, or use a backup taken after it").
• Never an acknowledgement. Its transactions committed between the backup's newest transaction and the target still count toward the divergent set D, so they must be in excludeGtids unless they are on the target history. An unreadable quarantined row counts its hinted GTIDs, to fail closed.
• Identity-checked. A readable quarantined row that overlaps the backup, the target history or the replay is downloaded and compared like any other copy (PITRArchiveNamespaceReuse on a mismatch, or when it cannot be read).
• The waiver. An unreadable quarantined row needs no identity download when all three hold: every GTID it holds is in the backup, the backup's stamp was created more than δ (the fixed 5 s clock-skew margin) after the quarantine, and none of its GTIDs is replayed. The dump was then taken from the live dataset after you declared the row lost, so the row's bytes cannot change the result. Take a full backup after quarantining and restore from it.

--resume-gtids is still required: the command checks it against prior quarantine and acknowledgement bounds, and records it, but the planner no longer uses it to scope windows. After recovering an original object, the same command with the exact site, name and ID plus --undo --confirm restores the row. Preserve the manifest copy as your incident record. Resume PITR writers, inspect status, and verify an isolated restore with application markers across the repaired boundary.

Distinguish divergence, re-logging and GTID reuse

The auditor records divergent identities when archived GTIDs are absent from a server that has established its history. Empty execution, a running clone, and a read-only server without a replication channel supply no negative evidence during bootstrap. Every eligible audit recomputes the missing sets; a recovered foreign-source set can clear contradicted flags and their status error. Known loss from the current server UUID remains unresolved when its numbers return: replacement transactions may still be in the active, unarchived binlog. Preserve the flag until recovery is verified or a canonical recovery is explicitly acknowledged; quarantine an abandoned tail promptly. Rotation alone does not prove original issuance.

A flagged row stays in the archive and is read like any other row. The planner uses the flags as loss evidence: in latest mode, a head that holds flagged transactions refuses PITRArchiveDivergentHistory, and in timestamp mode the lost transactions inside the window must be in excludeGtids (the refusal prints them). Promotion records give the same evidence without an audit: a transaction committed before a site was promoted without it was lost at that failover.

A verified reuse guard requires overlapping GTIDs of the same source server UUID with different archived content identities and different original commit or transaction-query identities. Re-logging may change file framing and immediate commit time while preserving original issuance. Smaller names, out-of-order uploads, identical-byte retries and a transient GET are not reset evidence. Unknown origins never create a reuse guard or authorize overlapping generations. Legacy rows without bound coverage are inspected in bounded batches even when retention is disabled. One worker drains a large stream; foreground rotations run first, failed inspections cool down for an hour, and restart reconstructs pending work from the manifest. Retirement waits for unresolved issuance checks. Downloads independently verify overlapping observed GTIDs before applying numeric backup cuts; unavailable or incomplete issuance proof fails loudly. Original commit/query evidence distinguishes actual reuse from a recovered execution set.

Quarantine preserves conflict and divergence evidence; undo restores the row and its divergence protection. A verified namespace-reuse guard remains even after quarantine: GTID containment alone cannot identify the right backup generation once numbers were reused. Start a new prefix/full backup for that case; acknowledgement only recovers obsolete guards without verified reuse.

For a real RESET BINARY LOGS AND GTIDS, prefer a new archive generation and full backup. A reset epoch, including one the archiver records after it verifies GTID reuse, refuses the whole archive (PITRArchiveHistoryReset), not only the affected site, whatever the backup and target. Set the selected profile's spec.backup.profiles[name].storage.s3.prefix (S3), or spec.backup.profiles[name].storage.pvc.subPath (PVC), to a new unused location; these are paths inside the existing named profile, not literal map keys. The archiver appends binlogs/ there. Alternatively create a new profile and select it with spec.backup.pitr.profileName. Wait for the rollout and take and verify a full backup in that generation. Keep older backups paired with their original generation; identical GTID numbers do not make a pre-reset backup compatible with a post-reset dataset.

Recover an obsolete guard behind a canonical full backup

Older versions could set epoch.resetReason from upload order alone. It is recoverable without deleting objects or discarding valid earlier windows. After preserving evidence, verifying a canonical full backup and stopping writers as above, use the same maintenance Pod/storage configuration with:

bloodraven pitr-quarantine --site=iad --acknowledge-history \
  --reason='Canonical full backup verified; obsolete upload-order guard' \
  --resume-gtids='aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:1-200' --confirm

This streams and identity-verifies every active archived file and requires its observed execution frontier to be covered by the canonical backup bound. Any verified namespace-reuse evidence, or missing, corrupt or uncovered history, refuses acknowledgement; first recover those exact objects or explicitly quarantine the unresolved rows. No objects are deleted or overwritten. The manifest records the reason, time, verification bound and prior guard reason, then supersedes the obsolete guard and divergence markers. Verification also compares overlapping issuance, including legacy streams whose conflicting evidence has not yet been recorded. Acknowledgement removes no rows and adds no global backup cutoff; older backups still undergo normal replay, divergence and generation checks, and the acknowledgement does not exclude anything from a restore. Quarantined rows retain their own bounds. Verified reuse cannot be cleared by acknowledgement or quarantine. Resume writers, inspect status and verify an isolated application restore.

Unsafe history marker

A site sets a permanent historyIntegrity marker in its own manifest when it finds a replication or binlog filter, binlog_format other than ROW, binlog_row_image other than FULL, or replica_preserve_commit_order off. From then on it uploads nothing, its lastError starts PITRArchiveUnsafeHistory: <site> <violation> <detail>, and the group shows PITRArchiveReady=False / ArchiveUnsafeHistory. Binlogs written while the setting was in force cannot certify history, so no repair command clears the marker, and removing the setting does not either. The marker removes every restore target that needs that site's bytes, including targets before the violation; the new prefix restores only targets after it. The operator emits a Warning PITRArchiveUnsafeHistoryDetected when a site reports the marker.

1. Undo the change. While PITR is enabled the operator refuses these options in spec.mysqlConf (condition PITRMysqlConfAccepted=False, Warning PITRMysqlConfRefused) and owns replica-preserve-commit-order, so a marker comes from an out-of-band change: SET GLOBAL, a hand-edited my.cnf. Remove any refused key from spec.mysqlConf too.
2. Point spec.backup.pitr at a new prefix (or a new profile) and wait for the rollout.
3. Take and verify a full backup in the new archive. Keep the old prefix with its backups and passphrase as evidence; do not delete the marked manifest.

Reissue risk after a reclone

A recloned server keeps its server_uuid. After a lossy failover, the old primary's lost transactions are flagged as divergent rows in its manifest (or you quarantined them). If that site is recloned and later promoted again, it issues the same GTID numbers again with different transactions. Every restore from the archive then refuses PITRArchiveHistoryReset, and quarantine cannot clear it.

When the active site changes (failover, planned failover or re-promotion) to a site whose last archiver status reports divergentRows + quarantinedRows > 0, or a site is promoted after a NoPrimary interval (active site A, then none, then B) on a group that has failed over before, the operator emits:

Warning PITRArchiveReissueRiskDetected  site <s> was promoted while its archive manifest
holds N divergent/quarantined rows. If <s> was recloned after a lossy failover,
it reissues the lost GTIDs, and every restore from this archive refuses
PITRArchiveHistoryReset. Start a new archive prefix and take a full backup.

The Warning is a heuristic: it fires on the promotion, before any reissued GTID is archived, and also when the rows came from something other than a reclone. Check whether the site was recloned since the failover that produced the rows (operator logs msg="starting bootstrap" with source="reclone" for that site, or the kubectl bloodraven reclone you ran). If it was, act at once:

1. Point spec.backup.pitr at a new prefix and wait for the rollout.
2. Take and verify a full backup in the new archive.
3. Keep the old prefix and its backups.

The old prefix stays usable for targets before the reissue only if no reissued GTID reached it. Once the promoted site archives one there, its archiver verifies the reuse and persists a reset epoch (resetReason), and from then on every restore from that prefix refuses PITRArchiveHistoryReset, targets before the reissue included. The rollout to the new prefix is therefore urgent: the promoted site keeps uploading into the old prefix until its pod rolls. If the old prefix was reset, point-in-time restores from it need a copy of the archive's manifests from before the reset (bucket versioning, or a copy of the prefix taken earlier); the retained dumps still restore without pointInTime.

After a lossy failover, a reclone of the old primary and a failback to it, start a new archive prefix and take a full backup, whether or not the Warning fired. Rotating server_uuid on reclone, which would remove the risk, is a planned follow-up.