Log schema contract

Bloodraven emits structured JSON logs from both the operator (bloodraven) and the per-MySQL sidecar (bloodraven-sidecar). This page is the contract that downstream log pipelines key off of: which fields are stable, what the msg values are for the events you care about, and what guarantees we make about changing them.
If you only need one rule of thumb: filter on msg for the event vocabulary in the Event reference below — those strings are stable. Everything else is best-effort.
Streams
Both binaries write to stdout. There are two independent JSON streams; you can tell them apart by the presence of certain keys.
| Stream | Source | Identifies as | What's in it |
|---|---|---|---|
Operational (slog) | Operator and sidecar | Has time, level, msg | Failover, promotion, bootstrap, recovery, fencing, archiver, sidecar startup, divergence detection — every event a human operator or alerting pipeline would care about |
Controller-runtime (zap) | Operator only | Has ts, level, msg, logger, controller, controllerKind, reconcileID | Reconcile-loop bookkeeping from controller-runtime: CR fetches, status updates, watch events. Useful for debugging, not a stable interface |
The contract on this page applies to the operational stream. The controller-runtime stream is emitted as-is by upstream sigs.k8s.io/controller-runtime and inherits whatever shape that library produces — we don't redefine it.
To filter to operational logs in most pipelines, key on the presence of the time field (slog) or the absence of the logger field (zap).
Common fields
Every record in the operational stream carries:
| Field | Type | Description |
|---|---|---|
time | RFC3339Nano timestamp (string) | Event time, normalized to UTC by the binary's slog handler regardless of pod timezone. Always ends in Z. |
level | string | One of DEBUG, INFO, WARN, ERROR. |
msg | string | The event identifier. Stable for events listed in the Event reference; may change for ad-hoc debug logs. |
Records emitted under a specific failover group also carry:
| Field | Type | Description |
|---|---|---|
fg | string | The MysqlFailoverGroup namespaced name (namespace/name). Present on every operator log scoped to a group. On the sidecar, this is the bare group name passed via FAILOVER_GROUP. |
Sidecar records additionally carry:
| Field | Type | Description |
|---|---|---|
pod | string | The pod name (set via MY_POD_NAME). Disambiguates per-replica logs when shipping multiple sites' sidecars to one stream. |
Levels
| Level | When |
|---|---|
DEBUG | Per-poll bookkeeping (status no-ops, transient probe errors, archiver tick events). Off by default — the operator's slog handler is set to INFO. |
INFO | State changes the operator deliberately took: failover, promotion, bootstrap, recovery, sidecar lifecycle. Most of the event vocabulary lives here. |
WARN | Degraded but not fatal: a single retry, a peer briefly unreachable, a non-critical operation that failed (connection kill, taint patch). The operator continues. |
ERROR | Operator-affecting failure: failover failed, self-fence triggered, status update rejected by the API server, CronJob-pod startup validation failed. Always paired with an error field (a string carrying either the underlying error or, for validation failures, a description of what was missing). |
DEBUG records may appear or disappear without notice. INFO/WARN/ERROR msg strings listed below are stable.
Field naming convention
- Keys are
camelCase. Common keys:site,fg,error,peer,count,source,donor,recipient. - The deployment API's
duration_msis an explicit contract exception: a JSON number in milliseconds, notdurationMs. Its group identity isgroupplusnamespace, rather than requiringfg. - Site identifiers (
site,oldPrimary,newPrimary,promotedSite,donor,recipient,activeSite,authoritativeActiveSite) all carry the bare site name as defined inspec.sites[].name. - GTID fields (
promotionGtid,divergentGtid,oldPrimaryGtid,newPrimaryGtid,followerGtid,activeGtid,extraGtid) carry MySQL GTID-set strings exactly as MySQL returns them — never parsed or canonicalised. Computed sets (divergentGtid,extraGtid) are MySQL-format sets; on the handoff, picker, ordered-update and promotion records they are truncated to 1024 characters with a trailing…. - Counts (
count,divergentTransactions,extraTransactions,attempt,maxRetries) are JSON numbers, not strings. - Durations (
leaseTimeout,pollInterval,delay,timeout) are emitted byslog's defaulttime.Durationrendering — currently a string like"30s". Treat as opaque if you need to parse, prefer the metric of the same name.
Event reference
This is the stable vocabulary. msg strings here will not change without a deprecation note in the release notes.
Deployment API
Every /deploy/v1 request emits one record, including authentication and authorization refusals. Filter on the exact msg, not on HTTP paths that can contain caller-provided identifiers.
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | deploy api | handler, group, instance, namespace, operationId, status, duration_ms | A deployment API request completes. |
handler identifies the route handler; group, instance, namespace, and operationId are strings (empty when not resolved or not applicable). instance is the MysqlDatabase resource name. status is the numeric HTTP status, and duration_ms is numeric elapsed milliseconds. No bearer token, lease token, or token hash is logged. Treat unresolved request identifiers as untrusted input, not proof of authorization.
Requests increment bloodraven_http_requests_total{server="deploy"}. Lease revocations emit Kubernetes Event DeploymentLeaseRevoked on the MFG and increment bloodraven_deploy_lease_revocations_total{group,kind,reason} with TopologyChanged or OperatorRevoked. A planned hold is visible as Deferred/DeploymentHold, PlannedFailoverDeferred, and bloodraven_planned_failovers_deferred_total{group,reason}. See Deployment API metrics for the complete lease metric family.
Failover
The four events that trace one failover, in order:
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | initiating failover | candidate, oldPrimary, fg | Operator has chosen a promotion target and is about to run the promotion sequence. DNS flips only after promotion succeeds and the target is verified writable. |
| INFO | failover complete | promotedSite, promotionGtid, fg | Execute (or the ordered-update ExecuteHandoff) finished: candidate is writable. A planned failover emits it too, only once its live confirmation found the target writable and every other site read-only; a planned promotion that ends unconfirmed or held never emits it, and a planned re-run that finds the target already writable and confirms it emits it too. promotionGtid is the candidate's gtid_executed snapshot taken just before clearing super_read_only — the upper bound on data that survived. For a planned re-run that found the target already writable it is the fenced set; for one whose target already held the run's own writes, the set read immediately before the clears (the fenced set plus those writes). |
| INFO | promotion confirmed: site is writable | site, fg | Next poll observes the promoted site is Writable. The internal post-failover guard clears here. |
| ERROR | failover failed | error, fg, retryAt | The promotion sequence returned an error. The operator re-evaluates the topology at retryAt (10s later) even if no site changes state, and earlier if one does. |
| ERROR | failover picker: refusing candidate whose data regressed | site, gtid, promotionGtid, reason, fg | A promotion candidate was excluded because promoting it would replace committed history: its gtid_executed is empty in a group known to hold data (a wiped datadir, for example after a failed CLONE), or it does not contain promotionGtid, the GTID set recorded at the last promotion. Reclone the site. If no candidate remains, no failover happens and the decision is re-evaluated every 10s. Page on it. |
| WARN | failover picker: skipping divergent candidate | site, divergentTransactions, divergentGtid, selected, fg | Emergency failover preferred selected, a candidate without a divergence report, over site, which carries one (recoveryState=RecoveryBlocked) — even if site had a fresher GTID set. Transactions only site holds, legitimate ones included, are not on the new primary; site keeps its report. |
| ERROR | failover picker: promoting a divergent candidate; no other candidate remains | site, divergentTransactions, divergentGtid, comparedTo, fg | Every remaining candidate carries a divergence report, so emergency failover promotes the freshest of them anyway: divergence never blocks an emergency failover. Its extra transactions become authoritative; once it is the confirmed primary the operator records the DivergentPromotion condition. Page on it. |
| ERROR | promotion succeeded but writable confirmation failed; DNS not flipped | site, error, fg | Execute returned successfully but the promoted site did not report writable within the confirmation window. DNS is not flipped and no failover state is recorded — the promotion is treated as unconfirmed and re-evaluated on the next tick. |
| ERROR | DNS flip failed after successful promotion | site, error, fg | Promotion and writable confirmation both succeeded, so the failover state (cooldown, split-brain target, promotionGtidExecuted) and bloodraven_failovers_total are already recorded; only the DNS update failed. bloodraven_dns_flips_total is left unincremented. The poll loop reconciles DNS against the current active site (see DNS reconciled to active site below), so a transient failure such as an RBAC denial self-heals once the write is permitted again — MySQL has already promoted regardless. |
| WARN | DNS reconcile failed | site, target, hostname, error, fg | The poll-driven DNS reconcile tried to point the record at the current active site and the write was rejected. Logged once per failing episode, not once per poll: while the failure persists the retry continues silently (DEBUG DNS reconcile still failing) and MySQL is unaffected. |
| INFO | DNS publication withheld until the initFromBackup restore succeeds | site, fg | A new group's spec.initFromBackup restore has not succeeded (pending, running, or failed), so the group hostname is not pointed at the writable seed site. Logged once per restore; the record is written on the first poll after the restore succeeds. |
| INFO | DNS reconciled to active site | site, target, hostname, fg | The DNS record diverged from the desired spec.dns (target IP, hostname, or TTL) and was repaired — a promotion-time flip that had failed, a hostname rename, a record left stale by an operator restart, or an out-of-band edit. bloodraven_dns_flips_total{site} increments here, and only when the A-record target IP actually changed. No promotion is re-run and MySQL is not touched. |
Supporting events emitted inside Execute:
| Level | msg | Fields |
|---|---|---|
| INFO | fenced old primary with super_read_only=ON | fg |
| WARN | failed to fence old primary (may be unreachable) | error, fg |
| INFO | killed app connections on old primary | count, fg |
| WARN | failed to kill app connections on old primary | error, fg |
| INFO | relay log drain complete | fg |
| WARN | relay log drain did not complete cleanly, proceeding with promotion | error, fg |
Ordered update handoff
Emitted by ExecuteHandoff when an OrderedUpdate rollout hands the primary role from the healthy active site to its standby. On success the handoff also emits failover complete (see Failover); the emergency-path Execute messages above are not emitted.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | maintenance handoff: fenced source | candidate, sourceGtid, fg | The active site is fenced with super_read_only=ON. sourceGtid is its GTID_EXECUTED read after the fence, exactly as MySQL returned it — the set the standby must contain. |
| INFO | maintenance handoff: killed app connections on source | candidate, count, fg | Application sessions on the fenced active site were evicted. |
| WARN | maintenance handoff: failed to kill app connections on source | candidate, error, fg | Best effort; the handoff continues. |
| INFO | maintenance handoff: candidate caught up to fenced source | candidate, sourceGtid, fg | The standby's GTID_EXECUTED contains sourceGtid; promotion starts. |
| WARN | maintenance handoff refused; source not fenced | candidate, error, fg | The handoff was refused before the active site was fenced, so no write was paused: the standby did not reach the active site's current set within 30s (the usual case: it is lagging), the active site stopped answering, the active site was read-only, or either site's GTID set could not be read. The update ends with ordered update failed and is retried later. |
| WARN | maintenance handoff refused; candidate has transactions the source lacks | candidate, extraTransactions, extraGtid, fg | Before fencing, the standby's GTID_EXECUTED (read first) held transactions the active site's (read second) does not. Promoting it would make that history authoritative (issue #202), so the handoff stops; no write was paused. extraGtid is the extra set, truncated to 1024 characters. The update ends with ordered update failed; once the replica GTID audit records the divergence the standby is no longer chosen. See playbook 33. |
| WARN | maintenance handoff aborted; source unfenced | candidate, error, fg | The standby did not reach the fenced set within the 30s fenced budget, its applier failed, a GTID set could not be read or parsed, the active site stopped answering, the standby's set no longer equalled the fenced set when promotion re-checked it after STOP REPLICA or RESET REPLICA ALL (missing transactions, or transactions the source lacks: the error then says transactions the source lacks), or promotion failed before the standby became writable. The active site has super_read_only and read_only cleared and keeps serving; the standby's replication is restarted (re-created after RESET REPLICA ALL); the update ends with ordered update failed and is retried later. |
| ERROR | maintenance handoff: rollback failed: source may be left read-only | candidate, cause, error, fg | The abort could not unfence the active site. Check @@super_read_only and @@read_only on it. Page on it. |
| WARN | maintenance handoff: failed to restart replication on candidate after abort | candidate, error, fg | Promotion stopped after STOP REPLICA (or after RESET REPLICA ALL, once the channel was re-created) and the standby's replication threads could not be restarted. The standby is read-only and not replicating; see playbook 23. |
| INFO | maintenance handoff: restored candidate replication channel | candidate, host, fg | Promotion stopped after RESET REPLICA ALL may have removed the standby's channel; the operator re-created it towards host with auto-positioning and restarted replication. |
| WARN | maintenance handoff: failed to restore candidate replication channel after abort | candidate, error, fg | The channel could not be re-created after an abort that may have followed RESET REPLICA ALL. The standby is read-only with no working channel and nothing re-points it automatically; restore it by hand (playbook 23). |
Emitted by the ordered update itself:
| Level | msg | Fields | Notes |
|---|---|---|---|
| WARN | ordered update: handoff promotion returned an error but the standby is writable; recording the handoff | site, error, fg | The handoff's final read_only=OFF on the standby returned an error after it may have applied, and a probe of the standby's read_only (pooled connection, up to 3 tries) found it writable. The handoff is recorded as a promotion (the status.lastFailover record, the anti-flap cooldown, DNS) before the error is reported. The source stays fenced and the old active is not rolled; the update ends with ordered update failed (error contains failover during update: maintenance handoff: promote <site>: … (source left fenced: the candidate may already accept writes)). |
| WARN | ordered update: could not read the standby's read_only after a failed handoff promotion | site, error, fg | The same failed read_only=OFF, but the probe could not read the standby's read_only. Nothing is recorded; the source stays fenced, the update ends with ordered update failed, and the next topology poll observes the standby. |
| WARN | ordered update: active-site update stalled; standby is divergent | active, site, divergentTransactions, divergentGtid, reason, fg | The active site has spec drift, but every otherwise eligible standby carries a divergence report, so none is a handoff target and nothing is fenced. reason reads <site> is divergent (N transactions the primary lacks); reclone or reconcile before the active site can be updated, the same text the update's error carries. The stall is intended; the update resumes on its own once the audit clears the report. Logged at most every 30s while it lasts. See playbook 33. |
| INFO | ordered update: deferring follower update; no drifted follower is a healthy direct replica | active, driftSites, fg | Only non-active sites have spec drift, and none of them is a read-only replica with healthy threads pointed at the active site, so no update attempt could roll one; none is started and the drift stays queued. Typical cause: the drifted site is a returning old primary that old-primary recovery has not rejoined yet (an attempt would also hold that recovery off), or it is unreachable or replicating from the wrong source. The update starts on its own once a drifted follower replicates directly. Logged at most every 30s while it lasts. |
| INFO | topology manager deferring to its predecessor's ordered update; no failover, recovery or update until it exits | fg | A topology manager restarted (configuration or credential change) while the manager it replaced still had an ordered update running. Until that update exits, the new manager takes no emergency failover, re-assert, old-primary recovery, connection drain, source repoint or update of its own, a new planned failover waits in admission (admission.waitReason: PredecessorBusy, nothing fenced), and status.updatePhase is empty. The old update's context is cancelled, so this normally lasts seconds. |
| INFO | predecessor's ordered update exited; topology manager resuming | fg, heldFor | The window above ended; heldFor is its length. |
| WARN | ordered update: stalled; site pod not Ready | site, pod, reason, error, fg | The update stopped on site, whose pod pod (empty when there is none yet) is not Ready, which here means Ready continuously for 30s, measured from the pod's last transition to Ready and its newest container start: either the update rolled it and the new pod did not get there within 5 minutes, or the site already runs the new pod template and the update refused to roll another site or hand off the active site while it is not Ready. reason says why, for example sidecar not ready: crash looping, last exit code 1 (Error) (a container that has restarted, in whatever phase of its loop: back-off, running, briefly Ready again, or terminated), pod Ready for less than 30s, pod not scheduled: … or pod readiness unknown: … when the API server could not be read. error is the full update error. Nothing else is rolled or handed off and the active site keeps serving. Logged on every attempt, at most every 30s while the stall lasts. When the stall stops an attempt that was already running, ordered update failed is logged first, its error carrying the same text. Mirrored by the Updating condition (still True, reason OrderedUpdateStalled) and the OrderedUpdateStalled Warning Event; a retry that finds the same cause rewrites neither. See Ordered updates. |
| INFO | ordered update: stalled site pod is Ready | site, fg | No update attempt was left to clear a stall (the stalled site was the last to roll, or another precondition holds the update), and the operator found the stalled site's pod Ready again, continuously for 30s, so a crash-looping pod caught between crashes does not log it. The Updating condition returns to NotUpdating. |
| ERROR | ordered update: old active has transactions the new primary lacks; not repointing it | site, newPrimary, divergentTransactions, divergentGtid, oldPrimaryGtid, newPrimaryGtid, error, fg | After a successful handoff the rollout updated the old active site, fenced it again with super_read_only=ON, and found transactions in its set (oldPrimaryGtid) that the new primary's (newPrimaryGtid) lacks. The handoff proves the new primary holds the fenced set, so this means a bug or an out-of-band write. The old active is not repointed: it stays fenced with no replication channel, and the update ends with ordered update failed. Old-primary recovery then reports the set as divergentGtid and blocks (divergence detected with detectedBy=OldPrimaryRecovery). One exception: if the old active came back on an empty datadir, its set holds only server-init transactions under a new server_uuid. Recovery treats such a site as empty and leaves it to auto-clone, so no divergentGtid appears. error carries the same text as the ordered update failed line. Page on it. When the fence or a GTID read fails instead, this record is not logged: the update ends with ordered update failed and an error containing before repointing, and the site is left for old-primary recovery the same way. |
Planned failover
Emitted by a planned failover run. Every record names the run with runId, the value of status.plannedFailover.runID (and of the hold acknowledgement). Two emitters produce them:
- Engine records come from the reconciler that drives the run and are written through the reconciler's logger, so they carry the controller-runtime shape (
ts,logger,controller,reconcileID) besides the fields below. - Topology records come from the group's topology manager on the operational stream. Their
fgcomes from the manager's logger and isnamespace/name, as on engine records. (Earlier releases also appended the bare group name as a secondfgkey; none does now.)
Some events are logged by both, as noted. Filter on msg and runId.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | planned failover: claim granted | fg, runId, source, target | The topology manager granted the run its claim after the live evidence probe (topology), and the run's Fencing record committed (engine). From here until the claim is released, the topology manager defers cross-site decisions for the group. |
| INFO | planned failover: claim released | fg, runId, why; engine also outcome | The run's claim was released. why is publishing (the promotion is confirmed, recorded and published; logged before the Publishing write, which needs no claim), terminal, admissionTimeout, notCommitted (the Fencing write was proven not to have committed: nothing was fenced), statusStalled (a terminal marker ended the run), managerAbsent, zombie (the uncached block no longer names a live run), ack (an automation hold was acknowledged) or orphanHold (topology only: an automation hold that no durable record backs any more was released; see planned failover: released an automation hold no durable record backs). outcome is the terminal outcome, empty for a non-terminal release. |
| INFO | planned failover: released an automation hold no durable record backs | fg, runId | Engine. The group's claim slot held an automation hold, but an uncached read found no status.plannedFailoverHold, no True AutomaticFailoverHeld condition, no ManualRecoveryRequired terminal marker and no acknowledgement annotation, so the hold is released (claim released with why=orphanHold) instead of holding automation with nothing to acknowledge. Typical causes: a run-less True condition removed by hand (this releases automation without an operator restart), a group deleted and recreated, or a hold hydrated by a manager start that raced an acknowledgement. runId is empty for a hold rebuilt from a run-less condition. |
| INFO | planned failover: confirming recorded side effect | fg, runId, phase, effect | The pass found a record this process did not write (after an operator restart or a leader change), so it confirms the recorded side effect live before going on. effect is fence, drain, lagWait, dragonflySync, dragonflyTakeover, promotion or rollback. Logged once per run and phase. |
| INFO | planned failover: status write ambiguous; settling | fg, runId, revision, error | A status write timed out or failed with a reset connection or a 5xx. The operator re-reads the object and, if (runId, revision) did not land, bumps the planned-failover-status-fence annotation conditioned on the same resourceVersion to prove the write can never commit. The new fence value is always past the stored one (1ns past it when the clock has not moved), and the fence counts as proof only if it moved the object's resourceVersion; otherwise the settlement tries again, and reports the write Unknown once its attempts are used up. No side effect runs until it is settled. |
| ERROR | planned failover: cannot settle status write; holding | fg, runId, error | A run that holds the claim could not settle an ambiguous write (the re-read and the fence both failed). The claim stays held and the pass requeues after 2s; the status-stall deadline bounds how long this lasts. |
| WARN | planned failover: the target is no longer a primary candidate; not promoting it | fg, runId, site, role | Topology. A promotion attempt found that the target's role in the topology manager's current configuration is not PrimaryCandidate (role carries the spec.sites[].role value, for example DisasterRecovery or ReadOnly; spec.sites changed during the run; the change restarts the topology manager, and the run's claim carries over without a re-proof). Nothing is sent; the run rolls back TargetNotPromotable. The pre-write gate repeats the check immediately before clearing super_read_only, and every promotion statement repeats it again (the line is logged there too). |
| WARN | planned failover: pre-write gate refused | fg, runId, reason, sites | Topology. Immediately before clearing super_read_only on the target, the gate found a reason not to: TargetNotPromotable (the target's role in the topology manager's current configuration is no longer PrimaryCandidate), TargetChanged or ExecuteFailed (the target is not the recorded server, not read-only, has a channel, or does not hold exactly the fenced set), SourceLost, SourceUnfenced, AuthorityChanged (the source holds transactions past the fenced set: its fence was lifted, written through and set again) or ForeignWritable (the source), ForeignWritable or PeerUnreadable (another site). sites is the list of sites at fault. Nothing was written to the target by this attempt. A source that does not answer is refused as SourceLost only once the 10s source-loss window has expired (counted from the first failed source observation in the drain, lag wait or gate); until then the attempt is retried. The run rolls back or leaves the source fenced; when an earlier attempt may already have written, it ends ManualRecoveryRequired (ForeignWritable, also for a source with history past the fenced set), or PromotedUnconfirmed for PeerUnreadable or an unreachable source (SourceLost, which names the target). |
| INFO | planned failover: named fenced source for re-assert | fg, runId, site | A run ending SourceLeftFenced with its own, attributable fence named the source as the topology manager's last active site, so the primary re-assert can restore it once every site is reachable and read-only (engine and topology). The naming happens after the run's terminal status record (or terminal marker) has committed, before its claim is released. Also logged by a starting topology manager that re-derives the naming from the terminal record or a terminal marker (topology). A run whose result or marker carries the FenceNotAttributable warning is never named, including a run that sent a fence and then ended because authority moved. |
| INFO | planned failover: could not name the unconfirmed target | fg, runId, error | A PromotedUnconfirmed run could not record the target as the topology manager's last failover target. |
| INFO | planned failover: a failover was recorded during the run; the unconfirmed target is not named | fg, runId, target | A PromotedUnconfirmed run names its target only while the durable failover record is still the run's baseline (or its own). Another actor's failover was recorded during the run, so the target is not named and that failover's authority stands. |
| INFO / WARN | planned failover: automation held pending acknowledgement | fg, runId, sites | A run ended ManualRecoveryRequired and its claim became an automation hold (engine at INFO; topology at WARN, also when a starting manager rebuilds the hold from status.plannedFailoverHold, a terminal marker or a run-less True AutomaticFailoverHeld condition, or when the engine installs a hold at runtime). sites is a comma-separated string: the hold's ambiguous sites (status.plannedFailoverHold.ambiguousSites) when known, empty for a hold rebuilt from a marker or a bare condition. runId is empty for a hold from a run-less condition. Automatic cross-site actions stay held until bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>. Page on it. |
| INFO | planned failover: hold acknowledged | fg, runId | The acknowledgement names the held run; the operator clears status.plannedFailoverHold (the AutomaticFailoverHeld condition turns False/PlannedFailoverHoldAcknowledged), the marker, the hold and then the acknowledgement, in that order. An acknowledgement that names another run, finds no standing hold, or finds the claim slot held for another run emits the PlannedFailoverHoldAcknowledgmentIgnored Warning Event instead, and resumes nothing; the annotation is removed when no hold stands. |
| ERROR | planned failover: status stalled; ended the run with a terminal marker | fg, runId, outcome, reason, error | The run's status writes failed for 2 minutes from the durable stall-since time (10 seconds for a run that already released its claim after its promotion was recorded and published: only its Publishing write was left); the planned-failover-terminal annotation now ends it (outcome/reason are the marker's). The claim is released, or kept as a hold for ManualRecoveryRequired. error is usually empty. |
| INFO | planned failover: status stalled, but the fenced set is too large for a durable out-of-band watermark; keeping the claim | fg, runId | The fenced set exceeds 64 KiB, so the stall release is not available: the claim stays held until a status write lands. |
| INFO | planned failover: promotion already recorded; publishing without the claim | fg, runId, target | A process that does not hold the run's claim (after a restart or a topology-manager replacement) found a Promoting block whose own failover record (its target and fenced set as the watermark, not older than the promotion intent) is the durable record. The confirmed tail had released the claim before the Publishing write landed, so the claim is not hydrated back and nothing is deferred: the run writes Publishing and Succeeded without it, and the topology manager reconciles DNS. Logged once per run and process. |
| INFO | planned failover: status stalled, but the promotion's failover record is not durable yet; retrying | fg, runId | The run's promotion is confirmed but its Publishing write has failed for the stall deadline (2 minutes from the durable stall-since time, or 10 seconds when the confirmed tail had already released the claim, in which case the run no longer defers automation), and neither the out-of-band failover annotations nor status carry the run's own failover record (its time, target and fenced set as the watermark) yet. The terminal marker would end the run with no durable watermark, so the operator keeps retrying instead; the live Promoting block still records the run (and, unless the claim was already released, owns the sites), and a restart re-derives the confirmation from it. Logged once per run. Repeats mean both the status subresource and the annotation write are being rejected. |
| INFO | planned failover: target replaced before the promotion was recorded; rolling back | fg, runId, target, targetServerUUID | A new operator process found the target pod replaced (@@server_uuid is not the run's) before any promotion statement it can prove. Because the run's own failover record is not durable (DNS never pointed at the target), the source still holds the whole fenced set under the run's fence and every other promotable site reads read-only, the run rolls back (TargetChanged) instead of ending ManualRecoveryRequired. Writes only the replaced pod could have held were made after the fence, which is the asynchronous-replication RPO. |
| INFO | planned failover: legacy status refused | fg, phase | A non-terminal block written by an older operator (no runId, or a removed phase) was refused with reason LegacyRunRefused: NotStarted for a block that owned nothing (Pending, Validating, Admitting, Deferred), PromotedUnconfirmed for Promoting or Resuming (the older operator's promotion may have made the target writable), SourceLeftFenced for the other phases. Nothing was hydrated or sent to MySQL. |
| INFO | planned failover: draining the source failed | fg, runId, sourceReachable, error | A drain query (processlist or KILL) failed. The engine probes whether the source still answers (sourceReachable); only a source that does not answer counts toward the 10s source-loss window (SourceLost). A source that answers is drained up to drainTimeout and drain.remainingConnections is left unset, with the message the connections remaining on <site> are unknown (<error>). |
| INFO | planned failover: the fenced GTID set is invalid; rolling back | fg, runId, error | status.plannedFailover.fence.gtidSet does not parse, so no lag probe can ever see the target catch up. The run rolls back with reason InvalidGTID at once. |
| INFO | planned failover: lag probe failed; counting the target as unreachable | fg, runId, error | The lag probe returned an error; the target counts as unreachable for this pass (the 10s target-loss window, then TargetLost). |
| WARN | planned failover: could not restore the target's super_read_only | fg, runId, site, error | Topology. A rollback (or the Promoting re-proof's re-fence) found the target read-only but with super_read_only=OFF, and setting it back to ON failed or did not read back. The rollback retries with the source still fenced, and ends SourceLeftFenced at the rollback deadline if it keeps failing. |
| WARN | planned failover: rollback found another writable site beside the source | fg, runId, sites | Topology. The rollback found the source already writable and another site writable too; sites is a comma-separated list. When that site is the target and the run may have written to it, the run ends ManualRecoveryRequired (ForeignWritable) and holds automation; any other writer is left to split-brain handling and the run stays RolledBack. |
| WARN | planned failover: the group was recreated; its claim slot starts empty | fg, previousUid, uid | Topology runner. A MysqlFailoverGroup was deleted and recreated under the same name while the old object's claim slot was held (a run or an automation hold). The new object does not inherit that slot, nor the retired manager's failover record. |
| ERROR | planned failover: pass on an early-returning reconcile failed | error | Engine (controller-runtime stream). An earlier reconcile step returned an error or a requeue, and the planned-failover pass that still runs for a run in flight (or a hold, acknowledgement, terminal marker or deadline annotation) failed too. The earlier step's error is returned. Such a pass never admits a new request. |
Retry and bookkeeping records, all engine at INFO. Each means the pass did not complete a step and requeues; repeats for minutes deserve a look:
msg | Fields |
|---|---|
planned failover: claim attempt failed; retrying | fg, runId, error |
planned failover: cannot confirm the claim; retrying | fg, runId, error |
planned failover: the claim is not held by this run; waiting | fg, runId |
planned failover: re-proof failed; retrying | fg, runId, error |
planned failover: status write did not commit; retrying | fg, runId, outcome, error |
planned failover: terminal status write did not commit; retrying | fg, runId, outcome, error |
planned failover: promotion attempt failed; retrying | fg, runId, error |
planned failover: recording the promotion failed; retrying | fg, runId, error |
planned failover: rollback attempt failed; retrying | fg, runId, error |
planned failover: could not remove the request annotation; a later pass retries | fg, why, outcome, error |
planned failover: the request annotation changed since this pass read it; left for the next pass | fg, why (the engine removes only the request value it evaluated) |
planned failover: could not remove a finished run's annotation; a later pass retries | fg, annotation, error (a planned-failover-stall-since, -manager-absent-since or -confirmed annotation left by a finished run) |
planned failover: could not record the promotion's confirmation; keeping the claim | fg, runId, outcome, error (the planned-failover-confirmed annotation; the confirmed tail keeps the claim and retries) |
planned failover: could not record the admission wait; retrying | fg, runId, waitReason, outcome, error |
planned failover: the provisional claim's Fencing write is still unsettled; holding | fg, runId, error (once the write is proven not to have committed, the grant is released with why=notCommitted) |
planned failover: could not record the stall-since annotation | fg, runId, outcome, error |
planned failover: could not clear the stall-since annotation; the next commit retries | fg, runId, error |
planned failover: could not record the manager-absent-since annotation; retrying | fg, runId, outcome, error |
planned failover: could not clear the manager-absent-since annotation; retrying | fg, runId, error |
planned failover: could not delete the terminal marker of a finished run | fg, runId, error |
planned failover: could not clear the hold condition; retrying | fg, runId, outcome, error |
planned failover: could not delete the hold's terminal marker; retrying | fg, runId, outcome, error |
planned failover: could not delete the hold acknowledgement; retrying | fg, runId, error |
planned failover: admission waiting | fg, runId, waitReason, detail (a deployment-lease refusal: an ordered update or in-place restore holds the sites, or a topology change is not persisted yet. runId is empty for a new request, which is then ended NotStarted at once for a refusal that depends on the request alone, or admitted into an Admitting wait with that admission.waitReason; a Deferred run logs its own runId and stays Deferred, recording admission.waitReason and the message deferred; admission waiting (<reason>): <detail>. An Admitting run's refusal is recorded as admission.waitReason and not logged) |
outcome on these is the write outcome (Committed, Conflict, NotCommitted, Unknown, Gone), not a run outcome. A status write that conflicts on all three attempts of a pass counts toward the 2-minute status stall like an Unknown or NotCommitted one. Earlier releases logged no planned failover: … records, so none was renamed.
One reconciler record belongs to planned failover without the prefix:
| Level | msg | Fields | Notes |
|---|---|---|---|
| ERROR | planned failover: pass failed; retrying on the planned-failover cadence | fg, error, requeueAfter | Controller-runtime stream. A planned-failover pass failed transiently (its uncached read of the group, a topology-facade error) and asked for a bounded retry. The reconcile requeues after requeueAfter (2s for a failed read) instead of returning the error, which would hand the retry to the controller's exponential backoff and could delay a fenced run's deadlines. A persistent read failure therefore logs this every couple of seconds. |
| ERROR | reconcile failed while a planned failover is in flight; retrying on the planned-failover cadence | error, requeueAfter | Controller-runtime stream. An earlier reconcile step (credentials, encryption, ConfigMaps, Deployments, Services, PDB, Dragonfly snapshot upgrade, restore Job, in-place restore) failed while a planned failover owns the sites, is Publishing, or holds a run claim. The planned-failover pass still ran, and the reconcile requeues after requeueAfter (the run's cadence) instead of returning the error, whose exponential backoff would starve the run's deadlines. |
Divergence and recovery
Fired when the operator inspects a returning old primary (after an emergency failover, or an old active site an ordered update refused to repoint), and by the replica GTID audit, which checks every read-only follower with a replication channel against the confirmed primary about every 30 seconds (and on the first poll after start, after the primary changes or is replaced under the same name, and after a completed promotion, handoff or re-assert; a refused divergent handoff candidate is audited on the next poll), whatever its replication threads report.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | initiating old primary recovery | oldPrimary, newPrimary, fg | Recovery sequence starting. |
| INFO | no GTID divergence, auto-recovering old primary as replica | site, fg | Old primary's GTID set is a subset of the new primary's — safe to attach as replica. |
| WARN | divergence detected | site, divergentTransactions, divergentGtid, oldPrimaryGtid, newPrimaryGtid, detectedBy, primary, fg | site holds committed transactions the primary never had. detectedBy is OldPrimaryRecovery (a returning old primary: oldPrimaryGtid is its set, newPrimaryGtid the new primary's) or ReplicaAudit (a follower, often replicating normally: oldPrimaryGtid carries the follower's set, newPrimaryGtid the primary's). When the audit re-verifies a block it did not create (an old primary whose replication was reattached by hand), it keeps that block's origin, OldPrimaryRecovery or empty for a block restored without provenance; primary names the site compared against. The site gets recoveryState=RecoveryBlocked and is not promoted by an ordered update or a planned failover. The operator does not auto-recover — the admin must reclone the site or make the primary contain the divergent set (containment then clears it). Emitted when divergence is first detected and again whenever the periodic re-verification finds the set CHANGED — an unchanged report is not re-logged each cycle. Mirrored by the bloodraven_divergent_transactions gauge, Degraded reason DivergentTransactions, and the DataLossDetected (old primary) or DivergentReplicaDetected (replica audit) Kubernetes Event. See playbook 33 for the replica case. |
| INFO | divergence cleared: replica holds no transactions the primary lacks | site, primary, fg | A fresh follower-first audit proved the site's set is contained in the primary's (after a reclone, or after the primary was made to contain the set). The block is dropped and bloodraven_divergent_transactions{site} returns to 0. This is the only way a RecoveryBlocked follower with a replication channel is cleared; healthy threads alone never clear it. Mirrored by the DivergenceResolved Event (replica-audit reports) or OldPrimaryRecovered (old-primary reports). |
| WARN | replica GTID audit unverified | site, primary, error, fg | The audit could not confirm the primary or could not read or parse a GTID set. Nothing changes: an existing block stays and none is invented. Retried on the next audit interval. When the primary could not be confirmed, site lists every follower that was due, comma-separated. |
| INFO | replica GTID audit discarded: a server changed while it was read | site, primary, fg | @@server_uuid of the follower or the primary changed between the audit's reads (a pod replaced by a clone or restore), so the sets do not describe one pair of servers. Retried on the next poll. |
| INFO | replica GTID audit discarded: authority changed while it ran | site, primary, fg | A promotion, handoff or update began while the audit was reading. Retried once authority settles. |
| INFO | old primary recovery complete | site, source, fg | Old primary is now replicating from the new primary. source is the new primary's host. |
| ERROR | old primary recovery failed | site, error, fg | One step of the recovery sequence (fence / GTID query / CHANGE REPLICATION SOURCE / START REPLICA) returned an error. |
| INFO | recovery state cleared (site is now replicating) | site, fg | The site is a healthy, source-converged replica again, so its RecoveryInProgress marker is dropped and bloodraven_divergent_transactions{site} returns to 0. A RecoveryBlocked report is never cleared here: healthy threads do not prove the site holds nothing the primary lacks (see divergence cleared above). |
| INFO | recovery state cleared (site is writable) | site, fg | The site is writable, so it is no longer "recovering as a replica". A RecoveryBlocked report is not dropped merely because the site turned writable — a rogue-writable site keeps its live divergence report (split-brain fencing re-fences it, and the periodic re-verification refreshes the report). When the site is the operator's own failover target and the unique writable primary, the report is converted into the DivergentPromotion condition (see the next row) rather than dropped. |
| ERROR | divergent site promoted; its extra transactions are now part of the primary's history | site, divergentTransactions, divergentGtid, comparedTo, fg | A site carrying a divergence report became the confirmed primary (an emergency failover with no clean candidate, or a manual promotion). Its extra transactions are now authoritative. Recorded durably as the DivergentPromotion status condition and the DivergentSitePromoted Warning Event; the condition stays until an administrator removes it. |
Kubernetes Event reasons, status conditions and planned-failover reasons for divergence:
| Name | Kind | When |
|---|---|---|
DataLossDetected | Warning Event | A returning old primary holds transactions the new primary never received (divergenceDetectedBy=OldPrimaryRecovery), on first detection and when the count changes. |
DivergentReplicaDetected | Warning Event | The replica GTID audit found a follower holding transactions the primary lacks (divergenceDetectedBy=ReplicaAudit), on first detection and when the count changes. |
DivergenceResolved | Normal Event | A replica-audit report was cleared by a fresh containment proof. |
DivergentSitePromoted | Warning Event | A site carrying a divergence report became the confirmed primary; the same text is the DivergentPromotion condition's message. |
Degraded reason DivergentTransactions | Condition | At least one site has recoveryState=RecoveryBlocked. Outranks the per-site replication reasons (ReplicationLagging, ReplicationBroken, ReplicationError, ReplicationSourceMismatch); the cross-site alerts (SplitBrain, NoPrimary, TotalLoss, CrossSiteAlert), BootstrapFailed and SiteUnhealthy still win, while Initializing after an operator start does not. Ready is not changed, and the deployment API does not treat this reason (or a RecoveryPending=DivergentTransactions that only covers replica-audit reports) as unstable; an old primary's report still is. |
DivergentPromotion reason DivergentSitePromoted | Condition | Written when a divergent site is promoted and never cleared by the operator. A later divergent promotion is appended (newline-separated, newest last); the five newest records are kept, with a leading (earlier divergent promotions omitted; …) line once older ones are dropped. Every promotion also has its own DivergentSitePromoted Event and ERROR log. Remove it after reviewing the transactions: kubectl patch mysqlfailovergroup <group> --subresource=status --type=json -p '[{"op":"remove","path":"/status/conditions/<index>"}]'. |
status.plannedFailover.result.reason=DivergentTarget | Planned failover | The requested target has a divergence report (refused before fencing, outcome NotStarted), or held transactions outside the fenced source's set during the lag wait (rollback.reason=DivergentTarget, then outcome RolledBack, or SourceLeftFenced if another site was writable). |
Replication source convergence
After topology changes, Bloodraven verifies that every follower replicates
directly from the uniquely confirmed active primary. These events cover
candidate, DisasterRecovery, and ReadOnly followers; they are separate from the
old-primary recovery events above.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | replication source convergence started | site, activeSite, currentSource, expectedSource, fg | A follower needs a source or thread-state correction and passed the initial mutation gates. |
| INFO | replication source convergence complete | site, source, fg | The canonical source is the active primary and both replication threads are running. |
| WARN | replication source convergence blocked | site, activeSite, stage, followerGtid, activeGtid, fg | GTID containment failed before or after stopping replication. No source change is issued. |
| ERROR | replication source convergence failed | site, activeSite, stage, error, fg | A bounded source mutation or verification attempt failed. The next poll can retry safely. |
Stable stage values include pre-stop-gtid, post-stop-gtid, stop,
change-source, start, and verify. A follower awaiting its post-restore
reclone (status.restoreReclones.pending) is not converged: its status reads
Pending with reason RestorePeerHold and no convergence event is logged. Use the status
sourceConvergenceState and sourceConvergenceReason for current state; use
these logs for the detailed failure and GTID evidence.
Bootstrap and reclone
starting bootstrap is the single canonical event for "we are about to clone a replica". The source field disambiguates why:
source value | Meaning |
|---|---|
fresh-deploy | Initial bootstrap of a new failover group; donor is the seed site. |
auto-clone | Operator detected an empty replica during steady-state and is recovering it without an admin trigger. |
reclone | The bloodraven.shipstream.io/reclone-site annotation was set (by an admin, or by the in-place restore's reclone queue), the safety interlock passed, and the operator is wiping the named site. This is the reclone-started event. |
| Level | msg | Fields |
|---|---|---|
| INFO | starting bootstrap | source, donor, recipient, donorHost, fg |
| INFO | bootstrap deferred: another topology change holds the sites | recipient, source, reason, fg |
| INFO | cloning from primary | donor, fg |
| INFO | clone completed successfully | replica, fg |
| INFO | setting up replication | source, fg |
| INFO | replication started successfully | source, fg |
| INFO | bootstrap completed successfully | source, fg |
| ERROR | bootstrap failed | source, error, fg |
| INFO | clone returned expected connection drop, waiting for restart | error, fg |
| INFO | replica already has primary data (prior clone detected), skipping clone phase | fg |
A bootstrap is not started while another actor holds the sites (a planned failover, an ordered update, a promotion, an in-place restore, or a replaced topology manager still changing them); it is retried on a later poll. bootstrap deferred: another topology change holds the sites appears only when that actor claimed the sites while the bootstrap was being prepared; reason names it.
clone returned expected connection drop, waiting for restart is not a success signal. The client connection drops both when a clone succeeds (the recipient restarts) and when it is killed or aborted after MySQL has already dropped the recipient's user data. Before configuring replication the operator reads the recipient's performance_schema.clone_status and requires a new Completed row and a non-empty gtid_executed; otherwise it re-fences the recipient and logs bootstrap failed with the clone error.
A reclone-only narrative is therefore: filter msg="starting bootstrap" AND source="reclone" for the trigger event, then watch for bootstrap completed successfully (source="reclone") or bootstrap failed (source="reclone"). The Bootstrapping condition mirrors them: True while the clone runs, then False with reason NotBootstrapping after bootstrap completed successfully, or False with reason BootstrapFailed after bootstrap failed, which also sets Degraded=True with reason BootstrapFailed until the site replicates or a new clone starts.
State transitions
Every per-site state change emits one record. Use this to replay the topology timeline.
| Level | msg | Fields |
|---|---|---|
| INFO | state transition | site, from, to, fg |
from and to values: Unknown, Unreachable, ReadOnly, Writable (the status.sites[].state values). Mirrored by the bloodraven_state_transitions_total counter, whose labels carry the kebab-case form (unknown, unreachable, read-only, writable).
| Level | msg | Fields | Notes |
|---|---|---|---|
| WARN | site not classified after its probe budget; probes alternate between answers and errors | site, probes, fg | Once per site per topology manager: the site is still Unknown after failureThreshold + recoveryThreshold probes. It no longer holds the group in Initializing, which becomes NoPrimary naming the site. See Initializing after an operator start. |
Topology decisions
| Level | msg | Fields | Notes |
|---|---|---|---|
| WARN | ALERT | message, fg | A cross-site EvalCrossSite action returned an alert string (split brain, no primary, total loss). The same conditions emit SplitBrainDetected / NoPrimaryDetected / TotalLossDetected Kubernetes Events. A split brain whose second writable site the debounce has not confirmed yet ends its message with ; <sites> not yet confirmed. The Initializing state after an operator start has no alert and logs nothing here. |
| WARN | split-brain auto-resolve: fencing non-preferred site per spec.splitBrainPolicy.sitePriorities | (context) | Opt-in splitBrainPolicy is fencing the lower-priority site. The poll-driven retry of a persistent split-brain re-emits this at WARN at most every 30s (DEBUG in between). |
| INFO | fencing returning old primary (split brain after failover) | site, fg | A second site came back writable after a failover; the operator is fencing every writable site except the one holding live primary authority. Emitted at INFO on the first fence of a given split-brain and at DEBUG on the poll-driven retries in between. |
| ERROR | failed to fence returning old primary | site, error, fg | The super_read_only=ON write above failed. The next poll retries. DEBUG on retries, matching the INFO/DEBUG pairing of the fence itself. |
| INFO | post-fence application connections evicted | site, count, fg | Operator-side recovery removed one or more sessions that survived the sidecar's best-effort fence eviction. It retries on later topology polls until an empty pass or timeout, without blocking the poll loop between passes. No pass runs while a planned failover holds its claim (or a planned-failover automation hold stands), an ordered update or an unobserved promotion owns the sites; the drain resumes afterwards with a fresh budget. A Dragonfly snapshot upgrade does not defer it. |
| WARN | post-fence application connection drain retry failed | site, error, fg | A recovery drain pass could not enumerate or kill sessions. The operator retries within spec.connectionDrainTimeout. |
| INFO | post-fence application connection drain complete | site, fg | An empty eviction pass proved no application sessions remain. Old-primary recovery may now complete. |
| WARN | post-fence application connection drain timed out | site, timeout, fg | The dedicated drain budget expired. Time spent deferred behind a planned failover's claim or hold, or an ordered update, does not count against it. Recovery proceeds because super_read_only=ON still prevents writes; a surviving session is limited to stale reads. |
| WARN | re-asserting fenced promoted primary: no site is writable and the last failover target is GTID-complete; restoring writability | site, authority, fg | The last failover target was found fenced (read-only) with every site reachable and nothing writable — typically its own sidecar re-fenced it with a stale lease right after a promotion. The operator restores writability on the target. authority is lastFailoverTarget, or lastActiveSite for a group that has never failed over: the last site the operator saw as the group's unique writable primary (restored from status.activeSite after an operator restart), including a fresh-deploy seed and a primary that restarted read-only. While a spec.initFromBackup restore has not succeeded it is initFromBackupRestoreSeed: the site the restore loads into (restored from status.restore.targetSite, else status.activeSite, after an operator restart), re-asserted from any fence even with part of the load in it; its peers then only need to be read-only with no user schemas and no replication history. Mirrored by bloodraven_primary_reassert_total. Rate-limited to once per failoverCooldown. |
| WARN | primary re-assert refused: peer has transactions the target lacks — divergence needs human review | site, peerGtid, targetGtid, fg | The no-writable-site wedge was detected but restoring the last failover target would abandon peer transactions. The group stays read-only until an admin resolves the divergence. A peer's recorded divergentGtid (it carries a divergence report) is not counted: those transactions are already reported and stay on the peer; peerGtid is the peer's full set. |
| WARN | primary re-assert refused: the initFromBackup restore seed has replication configured | site, fg | The restore seed reads as a replica, so it is not the seed the restore authority describes; writability is not restored. |
| WARN | promotion refused: the initFromBackup restore has not succeeded and only its seed holds the restored data | site, seed, fg | Emergency failover or split-brain resolution would have promoted site while a spec.initFromBackup restore is unfinished. Nothing is cloned until the restore succeeds, so site is empty; promoting it would make the group serve an empty server once the restore succeeded. The group waits for seed. |
| WARN | primary re-assert refused: target no longer contains the recorded promotion GTID set (wiped or restored since promotion?) | site, promotionGtid, targetGtid, fg | The failover history no longer describes the target's data lineage; the operator will not restore writability automatically. |
| WARN | primary re-assert refused: recorded promotion GTID set failed to parse — status corrupted or manually edited? | site, promotionGtid, error, fg | status.lastFailover.promotionGtidExecuted is non-empty but malformed. The operator wrote this value from MySQL itself, so a parse failure means corruption or manual tampering — the re-assert safety argument depends on it, so the operator refuses. |
| INFO | failover blocked by anti-flap cooldown | (context), retryAt | A failover decision was deferred because failoverCooldown has not elapsed since the last one. The operator re-evaluates at retryAt, when the cooldown expires, even if no site changes state. |
| WARN | skipping promotion candidate: site is mid-keyring-rotation | site, unsealReason, fg | Emergency failover chose another sealed candidate. Mirrored by Warning Event KeyringPromotionSkipped and bloodraven_keyring_promotions_blocked_total{outcome="skipped"}. |
| WARN | promotion refused: every remaining candidate is mid-keyring-rotation | sites, fg | No sealed candidate remained. The group stays without a writable primary until rotation finishes. Mirrored by Warning Event KeyringPromotionRefused, Degraded reason NoPrimary with an UnsealReason=Rotation message, and bloodraven_keyring_promotions_blocked_total{outcome="refused"}. |
| WARN | promotion refused: split-brain winner is mid-keyring-rotation | site, unsealReason, fg | Split-brain auto-resolve would have picked a rotating site. Losers are not fenced. |
| WARN | promotion refused: candidate is mid-keyring-rotation | site, unsealReason, fg | Last-line check refused failover.Execute on a rotating candidate. |
| WARN | primary re-assert refused: target is mid-keyring-rotation | site, unsealReason, fg | The last failover target is mid-rotation; writability is not restored. |
| WARN | re-asserting restore offline_mode fence | site, fg | A full-instance in-place restore holds offline_mode=ON on site, and the topology manager found it OFF (usually a mysqld restart mid-restore) and set it again. |
| ERROR | failed to re-assert restore offline_mode fence | site, error, fg | The re-assert above failed; the next poll retries. Application sessions may reach the primary until it succeeds. |
| WARN | re-stopping restore peer replication IO thread | site, fg | site awaits its post-restore reclone and its replication receiver must stay stopped (the restore is past Fencing, or failed and left status.restoreReclones Held), but it was running or connecting (started by hand, or by an older operator). The topology manager stopped it again. |
| ERROR | failed to re-stop restore peer replication IO thread | site, error, fg | The re-stop above failed; the next poll retries. The receiver may pull the restore's DROP statements until it succeeds. |
| WARN | could not confirm a restore protection change; applying additions only | fg, error | A restore guard change (a released hold, clone block, promotion exclusion, freeze or offline_mode fence, or any re-stop change) could not be confirmed by reading the group through the API server. Only additions from the operator's own view are applied; releases and re-stop changes wait for the next confirmed read. |
| WARN | bootstrap refused: the recipient is held by an in-place restore | recipient, source, fg | A clone into a site whose restore queue is Armed or Held, or while an in-place restore froze the topology, was refused, whoever asked (source is auto-clone, fresh-deploy or reclone), also when the restore arrived while the clone was being admitted. Logged once per site; mirrored by the BootstrapRefusedRestoreHold Warning Event. |
| WARN | could not read the group uncached; the new topology manager shares its predecessor's planned-failover claim | fg, error | A topology manager starting with a predecessor (or with a claim already in the group's slot) could not read the group through the API server to hydrate the planned-failover claim. The claim slot is shared by every manager generation, and the predecessor's named active site (for example a source a planned failover left fenced, not yet re-asserted) is carried into the new manager (carried the replaced topology manager's named active site), so nothing is lost; a brand-new manager with an empty slot is not started and the next sync retries. |
| INFO | carried the replaced topology manager's named active site | fg, site | Follows the record above: the new manager's own restore named no active site, so it took the predecessor's (a site it still has). |
| WARN | could not read the group uncached; the new topology manager keeps its predecessor's restore protection | fg, error | A replacement topology manager could not read the group through the API server before starting, so it kept the restore holds of the manager it replaces. A brand-new manager in that situation is not started (failed to start topology manager) and the next sync retries. |
| ERROR | read the primary's gtid_executed after a failed in-place restore | fg, site, error | Settling a failed full-instance restore's reclone queue could not read the primary's gtid_executed for two minutes after the first failed read (or no topology manager ran for the group within ten minutes of the run ending). Whether the restore changed the primary is unproven, so the peers are held. |
| INFO | holding the reclone queue: the failed restore's Job pods did not terminate in time | fg, job, error (only when the pods could not be listed) | A pod of a failed full-instance restore's Job was still running, or the pods could not be listed, ten minutes after the run ended. Whether the restore changed the primary is unproven (gtid_executed is not compared while a pod may still write), so the peers are held. |
| INFO | old primary recovery deferred: replication held by an in-place restore | site, fg | Recovery would have pointed a site awaiting its post-restore reclone at the primary and started its replication. |
| WARN | promotion refused: candidate is pending a post-restore reclone | sites, fg | Emergency failover, split-brain resolution or the primary re-assert would have promoted a site a full-instance in-place restore has not recloned yet (status.restoreReclones.pending). It still holds pre-restore data, so it is not promoted. Mirrored by Warning Event PromotionRefusedReclonePending and, when no other candidate remains, Degraded reason NoPrimary with a pending post-restore reclone message. |
| ERROR | promotion watermark too large for the out-of-band failover record; only the status copy carries it | fg, target, bytes | A promotion's promotionGtidExecuted exceeds 64 KiB, so the bloodraven.shipstream.io/promotion-gtid-executed annotation is not written; only status.lastFailover.promotionGtidExecuted carries it. A planned failover with such a fenced set cannot use the status-stall release. |
| WARN | ignored out-of-order local failover record | target, lastFailover, currentTarget, currentLastFailover, fg | Two local promotion paths reported out of timestamp order; the newer in-process anti-flap pair remains authoritative. |
| INFO | carried a retired topology manager's failover record | fg, target, lastFailover | A topology manager replaced by a configuration or credential change held a failover record newer than its replacement's (typically an ordered-update handoff that finished on the old manager, whose status writes are dropped once it is replaced). The record moved into the replacement under the timestamp-monotonic rule, at registration or once the old manager's update workers exited, so its cooldown and last failover target apply; the replacement's next status write and out-of-band flush persist it. |
| INFO | not carrying a retired topology manager's failover record: its target is no longer a site | fg, target, lastFailover | As above, but the record's target was removed from (or renamed in) spec.sites, so the record is dropped rather than naming a site nothing can reach. |
| ERROR | out-of-band anti-flap state write failed; retrying every poll | fg, target, lastFailover, error | The annotation write was rejected. The newest record stays pending and is retried on each poll. |
| INFO | cross-site action deferred: in-place restore in progress | fg | Decisions are paused while restoreInPlace runs. |
| INFO | cross-site action deferred: planned failover in progress | fg | Emergency failover, split-brain resolution and every other cross-site decision are paused while a planned failover holds its claim (from Fencing until its promotion is confirmed, recorded and published, or its terminal record is durable) or a planned-failover automation hold stands (AutomaticFailoverHeld=True). A request that is only waiting (Admitting without a grant, or Deferred) defers nothing. Logged on every poll while it lasts. |
Sidecar fencing
The per-MySQL sidecar emits these in its operational stream. SELF-FENCING: is a stable prefix — msg strings that begin with it indicate the sidecar wrote super_read_only=ON to its local MySQL without operator instruction.
| Level | msg | Fields | Notes |
|---|---|---|---|
| ERROR | SELF-FENCING: topology mismatch — operator-authoritative active site disagrees with our site, setting super_read_only=ON | site, authoritativeActiveSite, observedAt, pod | The operator (or a peer relaying the operator's view) reports a different active site than this sidecar is on. Fired even when the operator is reachable. |
| INFO | fencing: clone in progress on this site; not fencing | pod | A CLONE INSTANCE is executing on this server. The operator makes a clone recipient writable because CLONE requires it, so the monitor skips both fencing rules while the clone runs and never kills the CLONE session. The recipient restarts after the clone and the startup safety net fences it. Repeats every monitor tick for the length of the clone. |
| ERROR | SELF-FENCING: Bloodraven and every peer unreachable beyond lease timeout, setting super_read_only=ON | bloodravenLastOk, latestPeerOk, peers, leaseTimeout, pod | Backstop rule: nothing is reachable, so we can't be sure we're still primary. |
| INFO | SELF-FENCING: killed app connections | count, pod | Application sessions evicted after fencing succeeded. Server-internal threads are spared, so a fenced replica keeps applying from the authoritative primary. |
| WARN | SELF-FENCING: failed to kill connections after fencing | error, count, pod | The fence write landed but eviction was incomplete. count is how many sessions were killed — 0 when the process-list query itself failed, so no session was ever enumerated. error joins every cause that applied: the listing query failed, iteration ended early, some rows would not scan, some KILLs were refused, or the fence hit its 20s deadline. super_read_only=ON still holds, so surviving sessions cannot write. The sidecar does not retry because it cannot safely distinguish a pooled operator promotion session. After promotion, the operator drains the fenced former primary until a pass finds no sessions or spec.connectionDrainTimeout expires. The sidecar's /status reports self_fenced, scoped to the current sidecar process. |
| WARN | SELF-FENCING: super_read_only write failed but the fence is in place; skipping connection eviction | error, pod | The SET GLOBAL returned an error, but a follow-up @@super_read_only read proved the write landed. The fence counts as established: /status reports self_fenced=true and SELF-FENCED follows. Eviction is skipped because the fence budget is spent; operator-side recovery performs the bounded follow-up drain. |
| ERROR | SELF-FENCING: super_read_only write outcome is unconfirmed; will probe again | error, probeError, pod | Both the SET GLOBAL result and its independent confirmation probe were unavailable. The monitor retains a process-local pending marker and retries @@super_read_only on later ticks instead of getting stuck behind read_only=ON with self_fenced=false. |
| WARN | fencing: could not resolve pending super_read_only write | error, pod | A later tick still could not read @@super_read_only; the pending result remains armed for another tick. |
| WARN | SELF-FENCING: previously unconfirmed super_read_only write is now confirmed | pod | A later tick proved the ambiguous write established super_read_only=ON; /status now reports self_fenced=true. |
| ERROR | SELF-FENCING FAILED: could not set super_read_only | error, pod | The fence write failed and a follow-up @@super_read_only read confirmed the instance was not super-fenced. The sidecar retries on the next tick. |
| WARN | fencing: could not confirm whether the super_read_only write landed | error, pod | The immediate @@super_read_only confirmation read failed. The pending outcome is retained and retried on later ticks. |
| ERROR | SELF-FENCED: super_read_only=ON has been set, only Bloodraven can restore | pod | Final status; the sidecar will not unfence on its own. The next operator promotion clears it. |
| INFO | fencing: MySQL is writable after prior self-fence; rearming monitor | pod | An actor with SUPER privileges (the operator, per the restore contract) made MySQL writable again after a self-fence. The monitor re-arms with a fresh lease window — it will not re-fence until a full leaseTimeout passes with the operator and every peer unreachable again. |
| INFO | fencing: MySQL turned writable; discarding active-site view from before the promotion | pod | This sidecar's MySQL went from read-only to writable with no failed probe in between: the operator promoted the site. The monitor drops its cached active-site view, which may still name the old primary, and starts a fresh lease window, as after a self-fence restore. Without this, an operator that becomes unreachable right after a failover leaves the new primary to fence itself on the pre-promotion view. |
| INFO | fencing: adopted active-site view from peer | peer, activeSite, observedAt, pod | Peer sidecar relayed a fresher view than what this sidecar had cached. Drives the topology-mismatch rule. |
Safety-net events. The startup safety net runs before the fencing monitor starts, and each of its MySQL writes times out after 3 s. If its fence (or, on the active site, its clear) does not complete, the fencing monitor finishes the job on the first tick where MySQL answers. The monitor also re-arms the safety net after MySQL has been unreachable for 30 s or more. Rows marked monitor come only from that deferred path. The two rows marked both can come from either path.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | safety net: set super_read_only=ON as precaution on startup | pod | |
| WARN | safety net: could not set initial super_read_only; the fencing monitor will retry | error, timeout, pod | The startup fence failed or timed out (for example, a hung mysqld). Startup continues, and the fencing monitor applies the safety net once MySQL answers. |
| INFO | safety net: this is the active site, clearing super_read_only | site, pod | both |
| INFO | safety net: confirmed standby site, staying fenced | site, activeSite, pod | |
| INFO | safety net: no active site reported by operator, staying fenced | pod | |
| WARN | safety net: could not query active site, staying fenced | error, pod | |
| ERROR | safety net: failed to clear super_read_only on active site | error, pod | both. Startup hands the clear to the monitor, and the monitor retries it on the next tick while the operator still confirms this site. |
| WARN | safety net: MySQL was unreachable long enough to have restarted; re-arming | unreachableFor, threshold, pod | monitor. A restarted mysqld comes back read_only=ON (the operator starts every mysqld with that flag), which stops application users but not SUPER sessions, so the monitor re-applies the startup rule and super-fences it unless the operator names this site active. |
| WARN | safety net: could not read super_read_only; will retry | error, pod | monitor. Still armed. |
| INFO | safety net: operator confirms this is the active site; MySQL is already writable | site, pod | monitor. Settled without a write. |
| INFO | safety net: operator confirms this is the active site; leaving the self-fence for the operator to clear | site, pod | monitor. The safety net never lifts a fence this monitor set itself. |
| INFO | safety net: super_read_only=ON confirmed; staying fenced | site, activeSite, operatorAnswered, pod | monitor. activeSite is the operator's answer on this tick, empty if it gave none. |
| INFO | safety net: set super_read_only=ON after MySQL became reachable | site, activeSite, operatorAnswered, pod | monitor. Like the startup fence, this one does not evict sessions and does not set self_fenced. If the instance is writable and the authoritative view names another active site, the topology-mismatch rule fences instead, with eviction. |
| ERROR | safety net: could not set super_read_only; will retry | error, probeError, pod | monitor. The fence write failed, and a confirming @@super_read_only read either failed or showed it did not land. Still armed. |
PITR archiver
Emitted by the sidecar's BinlogArchiver and the supervisor that builds
its archive store.
| Level | msg | Fields |
|---|---|---|
| ERROR / WARN | failed to init archive store | error, attempt, retryIn, pod |
| INFO | archive store initialized after retries | attempts, failingFor, pod |
| INFO | binlog archiver starting | storageType, binlogDir, binlogIndex, pollInterval, pod |
| INFO | archived sealed binlogs | count, pod |
| INFO | retention sweep complete | (sweep stats), pod |
| WARN | archive binlog | file, error, pod |
| WARN | retention: delete object | key, error, pod |
| INFO / WARN | retention deferred | reason, pod; optional error for failed issuance |
| WARN | fetch retention cutoff | error, pod |
| WARN | retention: coverage unavailable | file, error, pod |
| WARN | retention: save coverage | error, pod |
| WARN | retention: save delete acknowledgements | error, pod |
| WARN | retention: compact retirement records | error, pod |
| INFO | archive descriptor created | archiveID, destination, site, pod |
| WARN | archive descriptor changed | archiveID, destination, error, site, pod |
| INFO | promotion recorded | serverUuid, serverId, gtidExecuted, site, pod |
| WARN | uploads wait for the promotion record | error, site, pod |
| WARN | server incarnation reset recorded | serverUuid, serverId, gtidExecuted, observedAt, probeInterval, site, pod |
| WARN | server incarnation changed while the sidecar was not running | latestRecordAt, serverUuid, serverId, site, pod |
| WARN | archive history unsafe | violation, detail, site, pod |
| WARN | persist history integrity marker | error, site, pod |
| WARN | remove pending history integrity marker | path, error, site, pod |
archive descriptor created is logged once, by the site whose atomic create
wrote <prefix>/binlogs/archive.json; a sibling that lost the race verifies
the winner's descriptor silently. archive descriptor changed means a
re-verification before a manifest save or retention delete found a different
or missing descriptor; the site stops writing and lastError carries the
refusal token. promotion recorded follows each confirmed promotion record
(the first writable observation of an episode); gtidExecuted is the
server's set verbatim. uploads wait for the promotion record is logged on
each scan of a writable site whose current episode has no saved promotion
record yet (the record could not be written, or the role probe is
incomplete); the site uploads nothing until it is saved, and lastError
carries the error. server incarnation reset recorded follows each saved
incarnation-reset record: the server's server_uuid or server_id
changed, or its gtid_executed no longer contains the site's latest
record's set (a reclone, reset or replaced datadir). probeInterval bounds
how long before observedAt the change happened. server incarnation changed while the sidecar was not running is logged at sidecar start, before
that record, when the first observation already shows such a change against
the site's latest record (latestRecordAt); uploads wait until the record is
saved. archive history unsafe is logged once per sidecar
process for the first violation (violation is ReplicationFilter,
BinlogFilter, BinlogFormat, BinlogRowImage or CommitOrder).
persist history integrity marker means the marker could not be saved yet;
uploads stay blocked and the save is retried every scan. A violation found
while the archive is not yet verified (or is refused) is held as a pending
marker in the sidecar's pod-lifetime state directory and saved at the first
verified session; remove pending history integrity marker means that file
(path) could not be deleted after the save. See
Archive identity.
archive binlog also reports missing/divergent history and reset guards through
error; /archiver/status.lastError retains the problem and
bloodraven_archiver_upload_failures increases while it persists. The original
message and field keys are unchanged. Audited divergent identities persist in
the manifest and block affected transactions, including discovered aliases and
an otherwise unchecked last row. Bootstrap/clone observations cannot establish
divergence; eligible recovery on a fresh UUID can clear foreign-source flags.
Known same-source loss stays unresolved on numeric return until verified or
explicit canonical recovery. Reuse guards contain verified conflicting issuance;
publication order does not set them. Quarantine or a canonical-backup
acknowledgement provides explicit recovery. Recovery steps are in
Repair PITR history safely.
failed to init archive store is logged once per failed archive store
construction attempt. It used to be a single ERROR line followed by the
sidecar exiting; the sidecar now keeps running and retries with jittered
backoff from 5 s doubling to at most 5 minutes, so the line repeats at
most once per backoff interval. It is ERROR when the error text differs
from the previous attempt's (including the first) and WARN while the same
cause repeats. attempt counts from 1; retryIn is the delay before the
next attempt (a duration). archive store initialized after retries
follows the first success after at least one failure; failingFor is the
time since the first failed attempt. A first-attempt success logs neither.
The operator's archiver poller emits two lines about the PITRArchiveReady
condition. A poller that cannot read the group through the uncached API
reader (when its topology manager starts, or later) leaves the persisted
condition as it is until it can:
| Level | msg | Fields | Notes |
|---|---|---|---|
| WARN | PITR archive condition read failed; leaving the condition as persisted | fg, error | Once per episode per topology manager, however long the reads keep failing. error also reports a group recreated under the same name (UID mismatch); that condition is not written by this manager. |
| INFO | PITR archive condition read recovered | fg | The read that ends the episode; the condition is reconciled and written again from the next poll round. |
Per-upload success/failure is also reflected in the bloodraven_archiver_upload_failures and bloodraven_archiver_last_upload_timestamp_seconds metrics — prefer those for alerting.
Kubernetes Event reasons for the PITR archive:
| Reason | Type | Object | When |
|---|---|---|---|
PITRArchiveNotReady / PITRArchiveRecovered | Warning / Normal | MysqlFailoverGroup | PITRArchiveReady entered False ("PITRArchiveReady is False (True ("PITRArchiveReady is True again: every site reports …"). |
PITRArchiveUnsafeHistoryDetected | Warning | MysqlFailoverGroup | Edge-triggered on the transition into PITRArchiveReady=False / ArchiveUnsafeHistory, from a sidecar's history-integrity marker, in addition to PITRArchiveNotReady. spec.mysqlConf is not previewed. |
PITRMysqlConfRefused | Warning | MysqlFailoverGroup | While PITR is enabled, spec.mysqlConf sets an option that breaks replay (replication or binlog filter, binlog_format≠ROW, binlog_row_image≠FULL); it was left out of my.cnf. Once per refused set; the condition PITRMysqlConfAccepted=False/PITRUnsafeSettingRefused lists each as <site>: <field> sets <key>[=<value>] (<why>). |
PITRArchiveReissueRiskDetected | Warning | MysqlFailoverGroup | The active site changed to a site whose last archiver status reports divergent or quarantined rows in its own manifest: "site <s> was promoted while its archive manifest holds N divergent/quarantined rows. If <s> was recloned after a lossy failover, it reissues the lost GTIDs, and every restore from this archive refuses PITRArchiveHistoryReset. Start a new archive prefix and take a full backup." |
PITRArchiveRefused | Warning | MysqlStandbyCluster | The standby refused the source archive (descriptor, encryption, keyCheck or a manifest); emitted when the SourceConfigKnown reason changes. The message is the refusal, token first. |
PITRSharedArchivePrefix was removed with pointInTime.sites. The PITR
replay Events (PITRReplayTruncated, PITRDivergentHistoryExcluded,
PITROnHistoryTransactionsExcluded) are listed in
Monitoring.
retention deferred logs reason="noRetainedBackups" or
reason="backupInFlight" at INFO for normal waits. Bounded backfill with
eligible work remaining also logs binlogGTIDCoverageUnknown at INFO while
it schedules the next batch. archiveIssuanceUnverified at INFO means
legacy/overlapping issuance checks are pending and new retirements wait.
A failed issuance check logs the same reason at WARN with error, naming
site/name/ID and the repair/quarantine remedy. The failure remains in
/archiver/status.lastError throughout cooldown until verification succeeds
or that identity is removed/quarantined. All new retirements for that site
can remain deferred; monitor archive capacity. It does not itself schedule
rapid retention retries. Other WARN reasons are
backupMetadataUnknown (missing/malformed backup GTIDs or start times),
backupGTIDCoverageUnknown (no compatible protocol/proof), and
binlogGTIDCoverageUnknown (eligible files lack complete, content-bound
transaction coverage with no immediately eligible backfill). Those coverage
and backup deferrals retain files without themselves indicating an upload
failure. Failed issuance audits do retain lastError and increase the archiver
failure metric as described above.
retention: coverage unavailable warns once per failed inspection attempt;
that identity waits the normal retention interval before retry. Healthy
identities can continue in 30-second batches. fetch retention cutoff now
logs lookup errors, including HTTP 429, at WARN with the existing error
field. Throttling is per transport source IP; it cannot consume another
source's group admission. A 429 has no delete authority and retries after
30 seconds; it is distinct from a successful safe-deferral response.
retention: delete object leaves a durable pending record for retry.
retention: save delete acknowledgements means successful deletes could
not be acknowledged; the next sweep safely repeats them. retention sweep complete keeps its cutoff, deleted, and manifest_pruned fields.
For a sweep that only finishes older pending deletes without a current
operator proof, cutoff is an empty string and manifest_pruned is zero.
deleted counts successful, idempotent delete calls.
retention: compact retirement records means the manifest update that
removes obsolete acknowledged records failed. Their durable records
remain available for later scans; this does not re-upload retired data.
Each site's maintenance can emit retention events while that site is a
replica. Active archive counts exclude retirement records.
Data-at-rest encryption
Emitted when spec.encryptionAtRest.enabled=true. See
Data-at-rest encryption for the lifecycle
these events trace.
Operator events:
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | keyring phase transition | site, from, to, reason, version, fg | The per-site keyring lifecycle advanced. The normal sequence is Pending → Unsealed → Escrowed → Sealed. Any transition to Unsealed from Sealed means the site is deliberately running with a writable keyring (clone or rotation). |
| INFO | keyring escrowed | namespace, group, site, version, secret, digest | The operator accepted a sidecar escrow push and stored a new immutable Secret version. Also emitted by the sidecar with site, version, secret, digest, pod. |
| INFO | refusing to enable encryption at rest on a live failover group | activeSite, fg | spec.encryptionAtRest.enabled was turned on for a group that is already serving, with spec.encryptionAtRest.existingData: Refuse (the default). Rendering is unchanged; existing tablespaces would have stayed plaintext. |
| WARN | keyring escrow rejected | namespace, group, site, reason | A push to /keyring/escrow failed authentication. The response to the caller is deliberately vague; the reason is only here. |
| WARN | keyring escrow digest mismatch | namespace, group, site, claimed, computed | An escrow push carried a digest that did not match its payload. Nothing was stored. |
| ERROR | keyring escrow: store failed | namespace, group, site, error | The operator could not write the escrow Secret. The site stays unsealed and the sidecar keeps retrying. |
| ERROR | keyring rotation: cannot confirm whether held rotation stamps were persisted; keeping the promotion block | fg, sites, error | An encryption pass ended without a successful status write, or wrote a status that would end a hold, and the uncached re-read that decides whether its (or an earlier pass's) UnsealReason=Rotation stamp landed failed. The listed sites stay promotion-blocked (emergency failover skips them, ordered updates and planned failovers will not promote them) until a later pass reaches the API server. Persistent repeats mean the operator cannot read its own CR. |
| ERROR | keyring rotation: cannot rule out a rotation stamp still in flight; keeping the promotion block | fg, sites, error | The re-read showed no stamp for these sites, but the conditional write that makes an in-flight status write unable to commit (the bloodraven.shipstream.io/encryption-status-fence annotation bump) failed, typically a conflict or an API error. The sites stay promotion-blocked; the requeued pass retries. |
Sidecar events:
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | keyring agent enabled | path, escrowArmed, rotate, pod | Sidecar startup. escrowArmed=false means this pod renders a sealed keyring and only reports its digest. |
| INFO | rotated innodb master key | site, pod | ALTER INSTANCE ROTATE INNODB MASTER KEY succeeded. |
| INFO | encrypted the mysql system tablespace | site, pod | ALTER TABLESPACE mysql ENCRYPTION='Y' succeeded on the writable site. |
| WARN | keyring escrow push failed, will retry | error, site, digest, pod | The push was rejected or the operator echoed a digest that did not match. Escrow is retried until it succeeds; the site is not sealed in the meantime. |
| WARN | could not encrypt the mysql system tablespace | error, site, pod | Retried on the next tick. |
| ERROR | keyring rotation failed | error, site, pod | The rotation statement failed. The agent still escrows whatever is on disk, because a failed rotation may already have written a key. |
Kubernetes Event reasons emitted on the MysqlFailoverGroup:
| Reason | Type | When |
|---|---|---|
KeyringPhaseChanged | Normal | Any keyring phase transition. A transition to Failed also has its own Warning (KeyringEscrowTimedOut, KeyringEscrowMissing, KeyringEscrowCorrupt, KeyringDigestMismatch, KeyringNotReadOnly). |
KeyringEscrowTimedOut | Warning | A site's keyring was not escrowed within keyring.escrowTimeout; the site is Failed. The operator keeps retrying and the site stays unsealed. |
KeyringUnsealed | Normal | A site was unsealed so a CLONE INSTANCE can rewrap tablespace keys. |
KeyringRotationRejected | Warning | A rotation was requested for the active primary, or while an ordered update or planned failover was in flight. Also emitted, once per reason, while a planned failover or promotion is in flight, the operator has not yet observed a unique writable primary with the target read-only, no topology manager runs for the group yet, or a manager being replaced still has an ordered update in flight. Every refusal except the active-primary one (by status or as the operator observes it) keeps the rotate-keyring annotation and retries every 10s. |
KeyringPromotionSkipped | Warning | Emergency failover skipped a mid-rotation site and promoted another sealed candidate. |
KeyringPromotionRefused | Warning | Emergency failover refused to promote because every remaining candidate is mid-keyring-rotation. Finish the rotation, then the next poll can promote. |
KeyringEscrowMissing | Warning | A sealed site's escrow Secret no longer exists. That site cannot restart. See the DR runbook. |
KeyringEscrowCorrupt | Warning | A sealed site's escrow Secret no longer hashes to the recorded digest. |
KeyringDigestMismatch | Warning | A pod rendered sealed is running a keyring that does not match its escrow; the operator refuses to call it sealed. |
KeyringNotReadOnly | Warning | MySQL reports a writable keyring on a site the operator considers sealed. |
EncryptionAdoptionRefused | Warning | Encryption was enabled on a group that is already serving, and spec.encryptionAtRest.existingData is Refuse. |
Dragonfly
Bloodraven optionally co-manages per-site Dragonfly instances and emits the following events when spec.dragonfly.enabled=true. Mirrored by the bloodraven_dragonfly_site_up gauge and the bloodraven_dragonfly_promotions_total{result} counter, plus the matching Dragonfly* Kubernetes Events on the MysqlFailoverGroup.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | dragonfly: configured replica | site, host, port, fg | Operator issued REPLICAOF against a non-active site to align it with the active master. |
| WARN | dragonfly: stale master on non-active site | site, active, fg | A site reports role=master but is not the active site. Auto-rejoin is attempted only when the stale instance has connected_slaves=0 AND master_repl_offset=0 (provably never accepted writes); otherwise the stale master is shed from the active Service via the traffic-label gate and left for human intervention. |
| INFO | stale-master reconfigure: REPLICAOF applied | site, host, port, fg | Auto-rejoin succeeded: the stale master is now linked as a replica of the active master. |
| WARN | stale-master reconfigure: REPLICAOF failed | site, host, port, error, fg | Auto-rejoin attempt failed; the next tick retries. |
| INFO | client-kill: evicted clients from old master | site, fg | After a planned-failover Dragonfly promotion, the operator issued CLIENT KILL TYPE NORMAL against the demoted source so application clients reconnect through the active Service. |
| INFO | dragonfly: source offset unavailable; cannot prove sync | error | A planned failover could not read the source Dragonfly's replication offset when it entered WaitingForDragonflySync (status.plannedFailover.dragonfly.sourceOffsetState: Unavailable). Session preservation cannot be proven, so the run applies onSyncTimeout at once: proceed continues with sessions: Lost, fail rolls back. Emitted through the reconciler's logger. |
| INFO | dragonfly/mysql active-site drift: promoting Dragonfly replica to match MySQL | oldSource, target, mysqlActiveSite, fg | MySQL active site and Dragonfly master diverged; the manager is promoting the synced Dragonfly replica on the MySQL active site. |
| INFO | dragonfly-only emergency: active master unreachable; promoting replica | oldSource, target, fg | Dragonfly master failed without a MySQL failover; the manager is promoting the single healthy replica and leaving MySQL status.activeSite unchanged. |
| INFO | dragonfly emergency: REPLTAKEOVER succeeded | site, fg | After an emergency MySQL failover, Dragonfly was promoted with sessions preserved. |
| WARN | dragonfly emergency: REPLTAKEOVER failed; falling back | site, error, fg | Emergency promote could not preserve sessions; falling back to REPLICAOF NO ONE. |
| WARN | dragonfly emergency: old master unreachable; skipping REPLTAKEOVER | site, oldSource, fg | The old Dragonfly master did not answer a dial and PING within 1s. A takeover cannot complete without the master, so the operator goes straight to REPLICAOF NO ONE (sessions lost). |
| WARN | dragonfly emergency: re-dial for REPLICAOF NO ONE failed | site, error, fg | After a failed REPLTAKEOVER, the fresh connection for the fallback could not be opened. The outcome is unconfirmed: a takeover can complete on the server after the client gives up, and a lost reply looks like a failure. The operator waits until the server can no longer be acting on the takeover (its budget plus 10s, plus 1s), then reads the target's role. One of the two promotion outcome unconfirmed / promotion failed lines below follows with the result. |
| INFO | dragonfly emergency: target promoted via REPLICAOF NO ONE (sessions lost) | site, fg | Emergency promote completed via the empty-master fallback after REPLTAKEOVER failed or was skipped. Mirrored by the Warning Event DragonflySessionsLost (not DragonflyPromotionSucceeded) and by bloodraven_dragonfly_promotions_total{result="sessions_lost"}. |
| WARN | dragonfly: REPLTAKEOVER not advertised | fg | Capability probe found REPLTAKEOVER missing from the running command table. |
| INFO | dragonfly: REPLTAKEOVER capability restored | fg | A previously missing REPLTAKEOVER command is now advertised (typically after an image upgrade). |
| WARN | dragonfly emergency: REPLICAOF NO ONE failed | site, error, fg | The fallback returned an error. A timeout or connection error is unconfirmed (the server may have applied it, or an earlier takeover may have completed), so the operator waits for the command to settle and reads the target's role. One of the two lines below follows with the result. MySQL emergency failover was not affected. |
| WARN | dragonfly emergency: promotion outcome unconfirmed; target observed as master | site, cause, fg | A promotion whose reply was lost or whose fallback failed was resolved by observation: the target reports role=master. It is recorded as a completed promotion (labels, CLIENT KILL, source recreate, status stamp), with the Warning Event DragonflySessionsLost (not DragonflyPromotionSucceeded) and bloodraven_dragonfly_promotions_total{result="sessions_lost"}, since session continuity cannot be confirmed. |
| WARN | dragonfly emergency: promotion failed; target not observed as master | site, role, cause, fg | The target did not report role=master once the commands could no longer be in effect (role is the observed role, or unknown: <error>). The old source's traffic label is restored so the active Service is not left empty, and the failure is stamped (DragonflyPromotionFailed, result="failed"). The next Dragonfly manager tick promotes again if MySQL and Dragonfly still disagree. |
| INFO | dragonfly emergency: restore old source traffic failed | site, error | Restoring the old source's traffic label after a promotion that did not happen failed. The reconciler's label sweep re-converges it on a later pass. |
| INFO | dragonfly promotion: request coalesced | site, replacedSite, fg | A MySQL emergency failover queued a Dragonfly promotion while an earlier one was still waiting to run. Only the latest request (site) is kept, since it reflects the latest MySQL decision; replacedSite was the target of the dropped one. |
| INFO | dragonfly promotion: manager stopping; request dropped | site, fg | A promotion was requested, or was about to run, while the group's Dragonfly manager was stopping (config or credential restart, group deleted, leadership lost). The replacement manager's Tick promotes if MySQL and Dragonfly still disagree. |
| INFO | dragonfly promotion: yielding to another owner; skipping | site, fg | A planned failover or rollout started waiting for the group's Dragonfly gate, or the manager began stopping, before this promotion sent any role-changing command. The promotion stops without sending one (the old source's traffic label is restored) and hands over the gate. A promotion that has already sent REPLTAKEOVER or REPLICAOF NO ONE never yields; it finishes and records its outcome first. |
| INFO | dragonfly promotion: planned failover in flight; skipping | site, fg | A queued emergency promotion found a planned failover that owns the sites or holds its claim (or an automation hold), or a Dragonfly snapshot upgrade in progress, which owns Dragonfly. During an upgrade nothing is stored for later: the upgrade reconcile re-derives the follow-up from MySQL authority. |
| INFO | dragonfly promotion: request superseded by a newer decision; skipping | site, reason, fg | A queued emergency promotion was dropped because a decision recorded after it was made points elsewhere: a planned failover claimed the group at or after the request's second (status.plannedFailover.claimTime; a run that is only waiting has decided nothing), MySQL failed over to another site (status.lastFailover.target), or Dragonfly was promoted to another site (status.dragonfly.lastPromotionTarget) while MySQL's status.activeSite does not name the request's target. reason says which. Status times have one-second resolution, so in a rare race (a promotion to the other site stamped in or after the request's second, before MySQL has published the request's target) a valid request is dropped; the Dragonfly manager's drift check promotes the MySQL site once pendingPromotionActiveSiteTTL (30s) after that promotion has passed. |
| INFO | dragonfly rollout: promotion no longer valid; skipping | site, reason, fg | A Dragonfly rollout promotion, re-checked against an uncached read once it held the gate, found its choice obsolete: a planned failover or snapshot upgrade is running, or the Dragonfly master is no longer the site it would take over from. The rollout requeues and decides again. |
| INFO | dragonfly rollout: target no longer in sync; skipping | site, error, fg | The rollout promotion re-proves the target's sync under the gate; it was no longer in sync (or could not be checked). The rollout requeues. |
| INFO | dragonfly rollout: site became the active Dragonfly master; holding its restart | site, fg | A StatefulSet rollout was about to restart a site it had chosen as a replica, but a promotion has since made it the Dragonfly master. The restart is held; the next reconcile treats the site as the active one. |
| INFO | dragonfly rollout: site is the planned failover's Dragonfly target; holding its restart | site, fg | A StatefulSet template rollout would have restarted the Dragonfly target of a planned failover that owns Dragonfly. The restart is held while the run owns Dragonfly. |
| INFO | dragonfly snapshot-upgrade deferred | fg, reason | A snapshot-restore upgrade run that spec.dragonfly.image drift asks for (upgradeStrategy: SnapshotRestore) cannot start: a site's Dragonfly StatefulSet does not carry the current spec.dragonfly settings other than the image yet, or the active site's Dragonfly pod does not run its current template (the render rolls them out first), status.activeSite is empty, a planned failover run or automation hold owns the sites, the effective Dragonfly master is not status.activeSite (for example after a planned takeover failed once MySQL had moved, or a Dragonfly-only promotion; the run would restart a replica and re-point the real master at it), or a site's Dragonfly StatefulSet is refused for WorkloadSelectorMismatch. reason names the cause. Logged when the cause changes, not on every 10s retry; no site moves meanwhile. Mirrored by the DragonflyUpgradeDeferred Event. Replaces dragonfly snapshot-upgrade deferred: the Dragonfly master differs from the MySQL active site (fields dragonflyMaster, activeSite). |
| INFO | dragonfly promotion: target already promoted since the request; skipping | site, fg | The queued request's target was already promoted: either a promotion to it was stamped after the request was made (for example by a Tick that saw MySQL move first), or status already names it and it still reports role=master. Nothing is repeated, so no duplicate events or metrics. |
| INFO | dragonfly gate busy; retrying | fg | A planned-failover Dragonfly phase, a rollout promotion, a replica StatefulSet rollout or a snapshot-upgrade step waited 5s for the group's Dragonfly gate, which another writer (the Dragonfly manager's current Tick or promotion, or a stopping manager's last step) still held. The reconcile requeues after 1s. A planned failover also reports the wait (DragonflyPromotionDeferred). |
| DEBUG | dragonfly label sweep skipped: gate busy | fg | The reconciler's Dragonfly pod-label sweep skipped this pass because another writer held the group's Dragonfly gate. The next reconcile sweeps again. |
| WARN | dragonfly emergency: target unreachable; skipping promotion | site, error, fg | The target could not be dialled within the 5s preliminary step budget. |
Kubernetes Event reasons emitted on the MysqlFailoverGroup (visible via kubectl describe):
| Reason | When |
|---|---|
DragonflyPromotionStarted | Planned-failover state machine entered PromotingDragonfly. |
DragonflyPromotionSucceeded | Dragonfly target was promoted by a planned failover or by an emergency REPLTAKEOVER. The emergency REPLICAOF NO ONE fallback, and an outcome confirmed only by observation, emit DragonflySessionsLost instead. |
DragonflyPromotionFailed | Promotion command failed; behavior depends on spec.dragonfly.plannedFailover.onSyncTimeout (planned) or is best-effort (emergency). A planned failover also emits it with the takeover did not finish within <d> of its intent when PromotingDragonfly passed its deadline (40s + maxSyncWait + 12s from takeover.intentTime) without a takeover; no REPLTAKEOVER is sent then. |
DragonflySessionsLost | Emergency promotion fell back to REPLICAOF NO ONE after REPLTAKEOVER failed or was skipped, or its outcome was only confirmed by observing the target as master. Session continuity is not guaranteed. Emitted instead of DragonflyPromotionSucceeded. |
DragonflyReplTakeoverUnsupported | Runtime probe found REPLTAKEOVER missing from the running Dragonfly command table. |
DragonflyReplTakeoverSupported | Runtime probe transitioned from unsupported to supported (typically after an image upgrade). |
DragonflyStaleMasterDetected | A non-active site reports master role. Logged + dedup'd in 5-minute windows. Auto-rejoin is attempted in reconcileReplication when connected_slaves=0 AND master_repl_offset=0. |
DragonflyOldSiteReconfigured | A stale master passed the auto-rejoin gate and was attached as a replica of the active master via REPLICAOF. |
DragonflySyncTimedOut | WaitingForDragonflySync exhausted spec.dragonfly.plannedFailover.maxSyncWait. |
DragonflyUpgradeStarted | A snapshot-restore Dragonfly upgrade run started (upgradeStrategy: SnapshotRestore and a site's live StatefulSet runs an image other than spec.dragonfly.image); status.dragonfly.upgrade was initialized. |
DragonflyUpgradeDeferred | Normal. A needed snapshot-restore upgrade run waits: no status.activeSite, a planned failover run or automation hold (including a durable one after a restart before the topology manager registered) owns the sites, the Dragonfly master differs from the MySQL active site, or a Dragonfly StatefulSet is refused for WorkloadSelectorMismatch. Emitted when the cause changes; retried every 10s. |
DragonflyUpgradeSnapshotStarted | Active Dragonfly traffic was shed and the operator is about to issue SAVE. |
DragonflyUpgradeSnapshotCompleted | SAVE completed against the active Dragonfly master using spec.dragonfly.snapshot.dir. |
DragonflyUpgradeSucceeded | Active and replica Dragonfly pods are on the target image, active traffic is restored, and replicas are linked. |
DragonflyUpgradeFailed | Snapshot-restore upgrade reached a terminal failure; the operator best-effort restored active traffic. With reason ActiveSiteChanged the upgrade was abandoned because a promotion moved the Dragonfly master away from its retained site; routing labels are left to that promotion. With reason WorkloadSelectorMismatch a site's StatefulSet was found refused before the next step, and nothing was saved, restarted or updated by that step. |
DragonflyPromotionDeferred | Warning. A planned failover reached PromotingDragonfly while another Dragonfly owner (usually an emergency promotion that had already sent its role change) held the group's Dragonfly gate. MySQL writes stay paused on the fenced source until it finishes: about 12-13s in practice, about 40s in the worst case. Emitted once per wait; status.plannedFailover.dragonfly.message says the same. |
License
Offline license observation. These events never change failover or reconcile
behavior. updatesUntil is an RFC3339 UTC timestamp string.
Per-group records (always include fg as namespace/name):
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | license verified | fg, organization, edition, updatesUntil, issuedFor, kid | A group's resolved token newly verified. |
| INFO | license updates period ended | fg, organization, edition, updatesUntil | A valid token's updatesUntil is in the past. Perpetual state, not an error. |
| WARN | license invalid | fg, reason, kid | A presented token failed verification. The group continues as Community. |
Process-level records for the operator default (no fg):
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | operator license verified | organization, edition, updatesUntil, issuedFor, kid | --license / BLOODRAVEN_LICENSE verified at process start. |
| INFO | operator license updates period ended | organization, edition, updatesUntil | Operator default token is valid but the update period has ended. |
| WARN | operator license invalid | reason, kid | Operator default token failed verification. Process still starts. |
Kubernetes Event reasons emitted on the MysqlFailoverGroup:
| Reason | Type | When |
|---|---|---|
LicenseVerified | Normal | A presented token newly verified, including the perpetual ended-updates state. |
LicenseInvalid | Warning | A presented token failed offline verification. Advisory only. |
Backup stamp
Emitted by bloodraven backup-stamp, which runs as the backup-stamp init
container of every backup Job and again from dump.py on plaintext Jobs:
with --check before the dump starts and with --end after it. Records carry subcommand="backup-stamp", and every record
after the store is built carries key (the stamp's object key or path),
backup and site.
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | backup stamp written | key, backup, site, reused, serverID, pitrArchiveID, serverClockStart | The init container wrote the stamp before the dump. reused is true when a retried pod adopted the stamp an earlier attempt wrote (its earlier serverClockStart is kept). pitrArchiveID is empty when PITR was off. |
| INFO | backup stamp finished | key, backup, site, serverClockStart, serverClockEnd | serverClock.end was added after a successful dump. On encrypted Jobs backup-encrypt-upload logs it after the upload, with its own subcommand and record fields. |
| ERROR | read dumped server clock | error | UTC_TIMESTAMP(6) and @@global.server_id could not be read from the server the dump uses. |
| ERROR | check dumped server | error | The server answered with a server_id other than the source site's pinned one (its spec.sites[].serverID). |
| ERROR | write backup stamp | error | The stamp could not be written, or a different stamp (another backup's identity, or one that already records an end) is at the key. |
| INFO | backup stamp checked | key, backup, site | --check: the location still holds this backup's unfinished stamp, so a plaintext dump may start. |
| ERROR | finish backup stamp | error | The end step failed. Also logged by backup-encrypt-upload. |
| ERROR | check backup stamp | error | The location no longer holds this backup's unfinished stamp (another backup took it over): backup-stamp --check before a plaintext dump, which then does not start, or backup-encrypt-upload before uploading, which uploads nothing. |
Configuration failures log read backup stamp template, resolve backup stamp storage, init backup stamp store, parse serverClock.end or
install backup stamp tool, also with error. Every failure fails the backup,
and the container's termination message starts with backup stamp: .
Times are µs RFC 3339 UTC strings.
Backup retention and verification holds
Emitted by the MysqlBackup reconciler. namespace and backup name the
MysqlBackup; fg and profile are its failover group and profile names
(not namespace/name, unlike the topology records above).
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | backup prune held by verification | namespace, backup, backupUID, fg, profile, verifications, verificationUIDs | Retention would prune a successful backup, but one or more MysqlBackupVerifications may still be reading it. verifications and verificationUIDs are parallel lists, sorted by name. The backup is kept and gets a RetentionDeferred Event; it is pruned on the first retention pass after the runs' pods stop. Logged on every retention pass while the hold lasts. |
| INFO | removed dead verification holds | namespace, backup, holds | A MysqlBackup being deleted carried verification-hold.bloodraven.shipstream.io/<uid> annotations whose holder can no longer read it: the verification is gone (for example its finalizer was removed by hand) or was replaced by a same-name one, with no pods of that name running; or it still exists but is finished, its Job has a terminal condition (or is gone) and its pods have stopped. holds lists the removed annotation keys. |
PITR archive bindings
Emitted by the MysqlFailoverGroup reconciler for the group's
archive encryption bindings.
fg is the group's name. No record carries passphrase bytes, an unkeyed
digest of them, or a keyCheck. keySecretName
(mysql-<group>-pitr-key-<h10>) does end in a truncated keyed digest: h10
is an HMAC, under the group's digest key, over the destination and the
passphrase bytes. Without that key it reveals nothing about the passphrase,
but treat it as key-derived rather than key-free (see
Key snapshots).
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | pitr archive binding changed | fg, bindingID, destination, archiveID, state, reason, keySecretName, cause | A binding entry is created, or its state, reason, archiveID or keySecretName changes. The last two can change with state and reason unchanged: a Pending/Authenticating check that rejects one candidate key and starts the next logs a line whose keySecretName differs and whose state/reason repeat. Filter on state/reason yourself if you only want those transitions. cause is proof (a sidecar report), check (a pitr-archive-check Job result, including a rejected candidate), recovery (rebuilt from a key snapshot's acceptance annotation after a status loss), integrity (the snapshot no longer verifies), c3 (the snapshot is missing: re-created under the same name, or KeyMaterialLost) or input (any other spec, Secret or snapshot input change; for example a new target's first entry, the identity profile's key Secret or key missing, a Refused entry retried after an input change, a check or Establishing candidate superseded by an identity change, or an acceptance that could not be recorded). Every line carries a cause. |
| INFO | pitr archive binding released | fg, bindingID, destination | A release annotation removed a binding and deleted its snapshots. |
| WARN | pitr render annotation ignored | fg, namespace, site, error | A site Deployment's bloodraven.shipstream.io/pitr-render annotation is malformed or has an unknown schemaVersion; it counts as absent. |
| INFO | pitr render deferred | fg, namespace, sites | While any site Deployment has no decodable render annotation (and the write binding is not Bound or Establishing), existing site Deployments and their ConfigMaps are not updated until a check Job authenticates the live key; sites lists their sites. A site created meanwhile renders. |
The check Job (bloodraven pitr-archive-check) logs INFO pitr archive check
(destination, role, state, reason, archiveID) and writes one
termination line, which the operator parses:
BLOODRAVEN_PITR_ARCHIVE_CHECK v=2 state=<Absent|Verified|Refused> reason=<token or empty> archiveID=<id> sourceUID=<uid> sourceNamespace=<ns> sourceName=<name> destinationDigest=<64 hex>
role is Writer or Reader (BLOODRAVEN_PITR_CHECK_ROLE). Values are
percent-encoded like the PITR verdict. A transient storage error is retried 5
times; the Job then exits non-zero without a line. A verdict that cannot be
written to the termination log exits 4 with ERROR write the verdict to the termination log; the check fails (path, error); the operator then reports
"check Job <job> failed: ..." and retries. The operator likewise retries a
check that exits 0 with a line that does not parse ("check Job <job> reported
a verdict line that does not parse: <parser error>").
Lifecycle
| Level | msg | Fields |
|---|---|---|
| INFO | starting bloodraven manager | (none) |
| INFO | starting auxiliary HTTP server | addr |
| INFO | topology manager runner starting | (none) |
| INFO | starting topology manager | fg |
| INFO | topology manager stopped | fg |
| INFO | stopping topology manager | fg |
| INFO | config changed, restarting topology manager | fg |
| INFO | group recreated, restarting topology manager | fg, previousUid, uid |
| WARN | topology manager exited; restarting | fg |
| WARN | timed out waiting for the Dragonfly manager to stop; its last step finishes under the group's Dragonfly gate | fg |
| WARN | connection material unreadable; keeping the running topology manager | fg, error |
| ERROR | failed to start topology manager | fg, error |
| INFO | restored lastFailoverTarget from CR status | fg, target |
| INFO | restored lastFailover from CR status | fg, lastFailover |
| WARN | restored lastFailoverTarget from out-of-band annotations | fg, target, statusTarget |
| WARN | restored lastFailover from out-of-band annotations | fg, lastFailover, statusLastFailover |
| ERROR | out-of-band anti-flap annotation unreadable; falling back to CR status | fg, error |
| ERROR | CR status anti-flap state unreadable; ignoring unsafe copy | fg, error |
| INFO | starting graceful shutdown | fg |
| INFO | CR deleted — DNSEndpoint will be garbage-collected | (none) |
| INFO | sidecar image version differs from the operator | operatorImage, sidecarImage (the effective image: the pin, or the operator's default), operatorTag, sidecarTag |
| WARN | BLOODRAVEN_SIDECAR_IMAGE is not set; groups without spec.sidecar.image cannot render their Deployments | (none) |
| INFO | sidecar starting | listenAddr, peerAddresses, bloodravenAddress, leaseTimeout, peerCheckInterval, site, namespace, fg, pod |
| INFO | sidecar stopped | pod |
| INFO | received signal, shutting down | signal, pod |
Kubernetes Event reasons emitted on the MysqlFailoverGroup:
| Reason | Type | When |
|---|---|---|
SidecarVersionSkew | Warning | The group's sidecar image (a spec.sidecar.image pin, or the operator's default BLOODRAVEN_SIDECAR_IMAGE when it pins none) is tagged with a different release than the running operator. The two halves share a rendering contract, so a mismatch can break the sidecar silently — on a spec.tls group an older sidecar cannot reach MySQL at all. Advisory only: the operator reports skew but never blocks on it, because a re-tagged or locally built image is indistinguishable from a real mismatch. |
TopologyManagerStartFailed | Warning | The group has no running topology manager because starting one keeps failing; mirrors the ERROR failed to start topology manager. Emitted on the first failure after a running manager was torn down, otherwise once the failure outlives one 30s retry; then at most once per 5 minutes (a changed error) or every 10 minutes (unchanged), because Kubernetes throttles a group's Warning Events together, whatever their reason. |
TopologyManagerStarted | Normal | A topology manager started after a TopologyManagerStartFailed Warning; the message gives the number of failed attempts. Failure history is kept in memory only, so a recovery after an operator restart is not announced. |
Breaking changes
v1.7.0: PITR archive identity (#206)
- Removed messages:
pitr-download's WARNstop proof scan ended at unreadable successorand WARNquarantined binlog excluded, and theanchorReasontexts, which belonged to the removed per-site planner.download site binlogsis kept; itserrortokens changed (see PITR Job completion and failure metadata):PITRArchiveSiteOmitted,PITRCrossSiteOrder,PITRBackupBoundUnknownandPITRArchiveFilteredHistoryare gone; the archive identity, dump, frontier, divergent-history, gap and order tokens are new. - Added Job messages: INFO
dump metadataand ERRORread dump metadata(dump-meta), INFOpitr planand theBLOODRAVEN_PITR_DIVERGENT_SETline (pitr-download).pitr-downloadno longer logs INFOpitr-download complete, ERRORmissing required env varor ERRORparse stop-datetime; its configuration failures loginvalid pitr-download config. - Verdict:
BLOODRAVEN_PITR_VERDICT v=2, followed byBLOODRAVEN_PITR_RESULT v=1and an optionalBLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONSline;v=1is gone and is refused. - restore.py sentinels: added
BLOODRAVEN_PITR_SEGMENT,BLOODRAVEN_PITR_RESULT v=1,BLOODRAVEN_PITR_DIVERGENT count=,BLOODRAVEN_PITR_DIVERGENT_SET gtids=andBLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS gtids=; removedBLOODRAVEN_PITR_NOOP;BLOODRAVEN_PITR_STARTandBLOODRAVEN_PITR_COMPLETEchanged their fields;BLOODRAVEN_RESTORE_COMPLETEdropped itspitr*keys. - Removed sentinel and Events:
BLOODRAVEN_PITR_CROSS_SITE_UNVERIFIED, thePointInTimeCrossSiteUnverifiedEvent and reason, and thePITRSharedArchivePrefixEvent. - Added messages: sidecar
archive descriptor created,archive descriptor changed,promotion recorded,uploads wait for the promotion record,archive history unsafeandpersist history integrity marker,remove pending history integrity marker,server incarnation reset recordedandserver incarnation changed while the sidecar was not running(PITR archiver), and thebackup-stamprecords, includingbackup-encrypt-upload'scheck backup stamp(Backup stamp). - Removed line: verify.sh's
BLOODRAVEN_VERIFY_REPLAY_COMPLETE(file,position,timestamp) is no longer printed; a verification's replay detail comes only fromstatus.pointInTime(for examplestatus.pointInTime.lastReplayedBinlogandstatus.pointInTime.newestReplayedCommitTime). - Added Events:
PITRArchiveUnsafeHistoryDetected,PITRMysqlConfRefused,PITRArchiveReissueRiskDetected,PITRDivergentHistoryExcludedandPITROnHistoryTransactionsExcluded(restored group or verification) and, onMysqlStandbyCluster,PITRArchiveRefused. - Environment: restore and verification Jobs no longer render
BLOODRAVEN_PITR_SITES,BLOODRAVEN_PITR_ALLOW_UNVERIFIED_CROSS_SITEorBLOODRAVEN_PITR_START_GTIDS; they renderBLOODRAVEN_PITR_DESTINATION,BLOODRAVEN_PITR_ENCRYPTION_MODE,BLOODRAVEN_PITR_MODEandBLOODRAVEN_PITR_DUMP_META,BLOODRAVEN_PITR_STOP_DATETIMEis always the canonical µs UTC form, and adump-metainit container runs first. Sidecars also getBLOODRAVEN_FAILOVER_GROUP_UIDandBLOODRAVEN_PITR_ARCHIVE_ID. - Status fields:
/archiver/statusgainsarchive,divergentRows,quarantinedRows,promotionRecordedAtandhistoryIntegrity. ThePITRArchiveReadycondition gains the archive identity reasons andSidecarOutdated, and no longer previewsspec.mysqlConf; the newPITRMysqlConfAcceptedcondition reports refusedmysqlConfoptions. Manifest records gainkind(Resetfor incarnation-reset records),probeIntervalandreadOnlyGtidExecuted.status.restore.pitr*is replaced bystatus.restore.provenance.pointInTime, andstatus.restoreInPlace.pitr*is removed (status.restoreInPlace.provenance.pointInTimestays unset); verifications addstatus.pointInTime.
v1.7.0: PITR archive encryption bindings (#203)
Additions only; no documented msg or field was renamed or removed.
- Added operator messages:
pitr archive binding changed,pitr archive binding released,pitr render annotation ignored,pitr render deferred(PITR archive bindings). - Added Job line and message:
BLOODRAVEN_PITR_ARCHIVE_CHECK v=2(its last key isdestinationDigest, the hex SHA-256 of the normalized destination, which bounds the line under the 4096-byte termination message limit) and INFOpitr archive checkfrombloodraven pitr-archive-check. - Added Events on the
MysqlFailoverGroup:PITRArchiveBound,PITRArchiveBindingReleased,PITRArchiveEncryptionChangeRefused,PITRArchiveKeyChangeRefused,PITRArchiveBindingRefused,PITRArchiveKeySourceRemoved,PITRArchiveKeySnapshotRecreated,PITRArchiveKeyMaterialLost,PITRArchiveKeySnapshotReadFailed,PITRArchiveBindingReleaseRefused,PITRArchiveBindingReleaseRejected,PITRArchiveBindingLimitReached,BackupScheduleRefused; on aMysqlBackup, NormalPITRArchiveStampSkipped(the backup carries no archive stamp because the site renders disagree or none is readable). New conditionPITRArchiveBindingReady; newMysqlBackupreasonsProfileReadOnly,PITRArchiveRolloutPending,DumpKeySnapshotConflict,PassphraseSecretMissing; new restore/verification tokensPITRArchiveBindingNotReadyandPITRBackupDumpKeyMissing. - Environment: sidecars get
BLOODRAVEN_PITR_KEY_REVISION(the key snapshot name orplaintext) and no longerBLOODRAVEN_PITR_PROFILE_NAME; the retention query sendsarchive=instead ofprofile=.
Stability commitments
| What | Stability |
|---|---|
msg strings listed in the Event reference | Stable. Changes go through a deprecation note in the release notes. |
Field names listed alongside a stable msg | Stable. New fields may be added to existing events; existing fields will not be renamed or removed without a deprecation note. |
| Field value shapes (strings, numbers, durations) | Stable for the values listed. GTIDs are passed through verbatim from MySQL — their shape is whatever MySQL emits. |
time, level, msg field names themselves | Stable. Tied to log/slog defaults. |
DEBUG-level records | Unstable. May appear, disappear, or change shape without notice. Disabled by default. |
Ad-hoc INFO/WARN/ERROR records not listed above (e.g. retry warnings, transient probe errors) | Best-effort. Field set is intended to be useful but not contractual. Don't build alerts that key on the exact msg string. |
Controller-runtime (zap) stream | Inherited from upstream. Bloodraven does not redefine this stream's shape. |
Pipeline integration tips
Filtering operational vs. controller-runtime
Most aggregators (Loki, Elasticsearch, Vector) let you split streams by JSON shape. A reliable predicate:
$.time && $.msg // operational (slog)
$.ts && $.logger // controller-runtime (zap)
Per-event alerts
Because every key event has a stable msg, pipeline alerts can be expressed as exact-match filters rather than fragile regexes. Examples for Loki:
# Failover started
{app="bloodraven"} | json | msg = "initiating failover"
# Failover failed (escalate)
{app="bloodraven"} | json | level = "ERROR" and msg = "failover failed"
# Divergence requires manual reclone
{app="bloodraven"} | json | msg = "divergence detected"
# Reclone triggered (track who/what asked for it via fg + recipient)
{app="bloodraven"} | json | msg = "starting bootstrap" and source = "reclone"
# Sidecar self-fenced — page on this
{app="bloodraven-sidecar"} | json | msg =~ "^SELF-FENCING:"
# Planned failover held automatic failover — page on this
{app="bloodraven"} | json | msg = "planned failover: automation held pending acknowledgement"
Correlating with metrics and Kubernetes Events
Several stable log events are mirrored by other observable signals — when one fires, the others fire too:
| Log event | Metric | Kubernetes Event |
|---|---|---|
failover complete | bloodraven_failovers_total{target_site} | FailoverExecuted |
divergence detected | bloodraven_divergent_transactions{site} > 0 | DataLossDetected (old primary) or DivergentReplicaDetected (replica audit) |
divergence cleared: replica holds no transactions the primary lacks | bloodraven_divergent_transactions{site} returns to 0 | DivergenceResolved or OldPrimaryRecovered |
divergent site promoted; its extra transactions are now part of the primary's history | (none) | DivergentSitePromoted |
old primary recovery complete | bloodraven_divergent_transactions{site} returns to 0 | OldPrimaryRecovered |
ordered update: stalled; site pod not Ready | (none; the Updating condition carries it) | OrderedUpdateStalled, once per stall |
failed to start topology manager | (none) | TopologyManagerStartFailed, rate-limited: at once when a running manager was torn down, otherwise after one 30s retry, then at most every 5 minutes |
planned failover: automation held pending acknowledgement | (none; status.plannedFailoverHold carries it, summarised by the AutomaticFailoverHeld condition) | PlannedFailoverManualRecoveryRequired, then PlannedFailoverHoldPersisted every 10 minutes |
planned failover: status stalled; ended the run with a terminal marker | bloodraven_planned_failovers_total{result} | PlannedFailoverStatusStalled |
planned failover: claim released (why=publishing) | bloodraven_planned_failovers_total{result="succeeded"} once Succeeded lands | PlannedFailoverSucceeded |
state transition | bloodraven_state_transitions_total{site, from, to} | (none — too noisy for events) |
Prefer metrics for alert thresholds and Kubernetes Events for human notification routing; logs are richest for forensics and timeline reconstruction.
Useful structured fields to index
If your pipeline supports indexing specific fields, the high-value ones are:
fg— partitions everything by failover groupsite(andoldPrimary/newPrimary/promotedSite/donor/recipient) — for per-site timelineslevel— for severity routingsource— for bootstrap/reclone disambiguationerror— full error string from the operator'serrorchain
PITR Job completion and failure metadata
PITR restore and verification Jobs run dump-meta, pitr-download,
optionally decrypt-download, and then restore.py (see
The restore Job).
The init containers log JSON records. dump-meta records also carry
subcommand="dump-meta".
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | dump metadata | location, backupUID, sourceUID, pitrArchiveID, site, dumpEnd | dump-meta read and accepted the dump's @.json, @.done.json and stamp, and wrote dump.json. dumpEnd is µs RFC 3339 UTC; pitrArchiveID is empty when the backup was taken with PITR off. |
| ERROR | read dump metadata | error | dump-meta refused the dump (error starts with its token), or could not read it from the store. |
| INFO | pitr plan | archiveID, mode, outcome, witness, groundTruth, segments, files, sites, replayed, divergent, onHistoryExcluded, downloaded, indexedTransactions | pitr-download found a plan and staged it. witness is the witness GTID and groundTruth is Origin or Anchored (both empty without a witness); sites is a comma-separated list; replayed is the replayed transaction count; divergent and onHistoryExcluded are GTID counts; downloaded (archived files fetched) and indexedTransactions size the planning work. |
| ERROR | download site binlogs | error | Kept. pitr-download refused (error starts with its token) or failed to plan or stage. |
Configuration and setup failures log, also with error: missing required env, invalid dump-meta config, resolve dump storage, init dump store
and write dump metadata (dump-meta); invalid pitr-download config,
invalid archive config and init archive store (pitr-download).
dump-meta and pitr-download write the same refusal, stable token first
and bounded to about 3 KiB, to their termination messages, and the operator
copies it into restore and verification status. The tokens and their order
are in Restore refusals.
A timestamp-mode PITRArchiveDivergentHistory names the transactions to
exclude, up to 1 KiB, and pitr-download prints the whole set to its log as
BLOODRAVEN_PITR_DIVERGENT_SET gtids=<set, up to 64 KiB>; it prints the same
line on success when the plan excluded divergent transactions. coverage.json
in the staging directory records the full plan, refusals included.
restore.py prints these text sentinels to its log:
| Sentinel | When |
|---|---|
BLOODRAVEN_PITR_TARGET_LOCAL gtids=<set, up to 64 KiB> | Before the load, when the target already holds its own transactions (image initialization, first-boot users); they are kept and are part of the certified executed set. |
BLOODRAVEN_PITR_START mode=<mode> stop=<target or none> segments=<n> transactions=<n> | Before the first segment. |
BLOODRAVEN_PITR_SEGMENT k=<index> file=<site>/<name> gtids=<count> transactions=<n> | Before each segment (file is percent-encoded). |
BLOODRAVEN_PITR_COMPLETE segments=<n> | After the replay; segments=0 when the backup already is the target state. |
BLOODRAVEN_PITR_FAILED: mysqlbinlog exit=<rc> segment=<k> / BLOODRAVEN_PITR_FAILED: mysql exit=<rc> segment=<k> (…) | A replay process failed; on a mysqlbinlog failure mysql is killed first. The mysql form's parenthesised note says <k> is where the failure was detected (the failing statement may be from an earlier segment, since mysql reads ahead) and appends a pipe or read error; a mysql still running then is killed (exit=-9). Also written to the termination message. |
BLOODRAVEN_PITR_FAILED: cannot open a session to … / BLOODRAVEN_PITR_FAILED: cannot read the replayed gtid_executed to prove the replay (R6): … | The fresh session for the baseline proof (R2) or the replay proof (R6) could not be opened or read: an operational failure, not a mismatch. |
BLOODRAVEN_PITR_PARTIAL requested=<target> newest=<time or none> | Outcome Partial. |
BLOODRAVEN_PITR_DIVERGENT count=<n> and BLOODRAVEN_PITR_DIVERGENT_SET gtids=<set, up to 64 KiB> | The replay excluded transactions outside the target history. A set cut at 64 KiB ends with .... |
BLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS gtids=<set, up to 64 KiB> | An excluded transaction on the target history precedes a replayed one. A set cut at 64 KiB ends with .... |
BLOODRAVEN_PITR_RESULT v=1 executed=<set> or BLOODRAVEN_PITR_RESULT v=1 executedSha256=<hex> | The proven post-replay @@global.gtid_executed; only its digest when the set is longer than 2 KiB. |
A PITR Job's outcome comes from its main container's termination message, not
its log. On success restore.py (through verify.sh on a verification)
writes, as its last action:
BLOODRAVEN_PITR_VERDICT v=2 mode= stop= outcome= archive= encryptionMode= sourceUID= sourceNamespace= sourceName= segments= files= transactions= lastFile=<site>/<name> newest= divergent=<n> onHistoryExcluded=<n> executedSha256= sites=
BLOODRAVEN_PITR_RESULT v=1 executed=<set>
BLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS gtids=<set>
- Line 1, the verdict, is at most 1024 bytes.
modeistimestamporlatest;stopis the µs UTC target (empty in latest mode);outcomeisTargetCovered,PartialorArchiveHead;archiveandencryptionModeidentify the archive andsource*its source group;filescounts the distinct files the segments read;transactionsthe replayed transactions;divergentandonHistoryExcludedare GTID counts;executedSha256is the hex SHA-256 of the post-replaygtid_executed;siteslists at most eight staged sites plus a+Ncount.stop,source*,lastFile,newestandsitesare percent-encoded. - Line 2 carries the post-replay
gtid_executed, or onlyexecutedSha256=<hex>when it is longer than 2 KiB. - Line 3 is present only when the on-history exclusions are non-empty and short enough to fit; otherwise status records only their count.
- The message fits the main container's share of the kubelet's termination
messages (12 KiB split among the pod's containers, at most 4 KiB each),
which the operator renders as
BLOODRAVEN_TERMINATION_MESSAGE_LIMIT(default 3072): line 3 is dropped first, then line 2 gives way toexecutedSha256=<hex>. No line is ever cut;verify.shrefusesPITRCoverageInvalid: restore.py's verdict is N bytes, over …rather than publish one that does not fit.
The operator accepts the verdict only from a Succeeded pod the Job owns, exit
code 0, matching the rendered mode, target, encryption and source group,
with a line 2 whose set hashes to executedSha256; anything else reports
PITRCoverageInvalid. It fills pointInTime in status from it. restore.py
and verify.sh write their own refusals there too (PITRCoverageInvalid: …,
PITRArchiveIdentityMismatch: …, PITRExcludeIntersectsBackup: …,
PITRNoArchivedReplay: …, PITRReplayMismatch: …,
PITRReplayUnsupported: …, BLOODRAVEN_PITR_FAILED: …,
SanityCheckFailed: …, SanityCheckTimeout: …). Other failures fall back
to the log tail. The failure detail recorded in status is the earliest
failing container's message. The served reason for BLOODRAVEN_PITR_FAILED
is PITRReplayFailed; the message keeps the sentinel.
The Python restore Job's text sentinel BLOODRAVEN_RESTORE_COMPLETE carries
only the source and target GTID and binlog coordinates, which the operator
reads best-effort; the pitr* keys are gone. These Job messages are not in
the Event reference contract above.
Artifact cleanup Job
The finalizer's cleanup Job runs bloodraven artifact-cleanup. It logs
artifact cleanup complete (objects) or artifact cleanup failed
(exitCode, error), with subcommand, backup, storageType and
location on every record. These Job messages are best-effort, not part of
the Event reference contract. The operator reads the outcome from the
container's termination message instead:
BLOODRAVEN_CLEANUP_COMPLETE objects=<n> on success, or
BLOODRAVEN_CLEANUP_FAILED: <reason> on failure, with exit code 64 for a
refusal or configuration error and 75 for a transient storage error. An
abandoned (never-succeeded) backup's cleanup that finds no stamp but objects at
the location deletes nothing, exits 0 with
BLOODRAVEN_CLEANUP_SKIPPED: <reason> and logs WARN artifact cleanup skipped
(reason), with the same common fields.

