Observability
Log schema contract
Bloodraven emits structured JSON logs from both the operator (bloodraven) and the per-MySQL sidecar (bloodraven-sidecar). This page is the contract that downstream log pipelines key off of: which fields are stable, what the msg values are for the events you care about, and what guarantees we make about changing them.
If you only need one rule of thumb: filter on msg for the event vocabulary in the Event reference below — those strings are stable. Everything else is best-effort.
Streams
Both binaries write to stdout. There are two independent JSON streams; you can tell them apart by the presence of certain keys.
| Stream | Source | Identifies as | What's in it |
|---|---|---|---|
Operational (slog) | Operator and sidecar | Has time, level, msg | Failover, promotion, bootstrap, recovery, fencing, archiver, sidecar startup, divergence detection — every event a human operator or alerting pipeline would care about |
Controller-runtime (zap) | Operator only | Has ts, level, msg, logger, controller, controllerKind, reconcileID | Reconcile-loop bookkeeping from controller-runtime: CR fetches, status updates, watch events. Useful for debugging, not a stable interface |
The contract on this page applies to the operational stream. The controller-runtime stream is emitted as-is by upstream sigs.k8s.io/controller-runtime and inherits whatever shape that library produces — we don't redefine it.
To filter to operational logs in most pipelines, key on the presence of the time field (slog) or the absence of the logger field (zap).
Common fields
Every record in the operational stream carries:
| Field | Type | Description |
|---|---|---|
time | RFC3339Nano timestamp (string) | Event time, normalized to UTC by the binary's slog handler regardless of pod timezone. Always ends in Z. |
level | string | One of DEBUG, INFO, WARN, ERROR. |
msg | string | The event identifier. Stable for events listed in the Event reference; may change for ad-hoc debug logs. |
Records emitted under a specific failover group also carry:
| Field | Type | Description |
|---|---|---|
fg | string | The MysqlFailoverGroup namespaced name (namespace/name). Present on every operator log scoped to a group. On the sidecar, this is the bare group name passed via BLOODRAVEN_FAILOVER_GROUP. |
Sidecar records additionally carry:
| Field | Type | Description |
|---|---|---|
pod | string | The pod name (set via BLOODRAVEN_POD_NAME). Disambiguates per-replica logs when shipping multiple sites' sidecars to one stream. |
Levels
| Level | When |
|---|---|
DEBUG | Per-poll bookkeeping (status no-ops, transient probe errors, archiver tick events). Off by default — the operator's slog handler is set to INFO. |
INFO | State changes the operator deliberately took: failover, promotion, bootstrap, recovery, sidecar lifecycle. Most of the event vocabulary lives here. |
WARN | Degraded but not fatal: a single retry, a peer briefly unreachable, a non-critical operation that failed (connection kill, taint patch). The operator continues. |
ERROR | Operator-affecting failure: failover failed, self-fence triggered, status update rejected by the API server, CronJob-pod startup validation failed. Always paired with an error field (a string carrying either the underlying error or, for validation failures, a description of what was missing). |
DEBUG records may appear or disappear without notice. INFO/WARN/ERROR msg strings listed below are stable.
Field naming convention
- Keys are
camelCase. Common keys:site,fg,error,peer,count,source,donor,recipient. - Site identifiers (
site,oldPrimary,newPrimary,promotedSite,donor,recipient,activeSite,authoritativeActiveSite) all carry the bare site name as defined inspec.sites[].name. - GTID fields (
promotionGtid,divergentGtid,oldPrimaryGtid,newPrimaryGtid,followerGtid,activeGtid) carry MySQL GTID-set strings exactly as MySQL returns them — never parsed or canonicalised. - Counts (
count,divergentTransactions,attempt,maxRetries) are JSON numbers, not strings. - Durations (
leaseTimeout,pollInterval,delay,timeout) are emitted byslog's defaulttime.Durationrendering — currently a string like"30s". Treat as opaque if you need to parse, prefer the metric of the same name.
Event reference
This is the stable vocabulary. msg strings here will not change without a deprecation note in CHANGELOG.md.
Failover
The four events that trace one failover, in order:
| Level | msg | Fields | Fired when |
|---|---|---|---|
| INFO | initiating failover | candidate, oldPrimary, fg | Operator has chosen a promotion target and is about to run the promotion sequence. DNS flips only after promotion succeeds and the target is verified writable. |
| INFO | failover complete | promotedSite, promotionGtid, fg | Execute finished: candidate is writable. promotionGtid is the candidate's gtid_executed snapshot taken just before clearing super_read_only — the upper bound on data that survived. |
| INFO | promotion confirmed: site is writable | site, fg | Next poll observes the promoted site is writable. The internal post-failover guard clears here. |
| ERROR | failover failed | error, fg | The promotion sequence returned an error. The operator does not retry automatically; the next eligible state-transition tick will re-evaluate. |
| ERROR | promotion succeeded but writable confirmation failed; DNS not flipped | site, error, fg | Execute returned successfully but the promoted site did not report writable within the confirmation window. DNS is not flipped and no failover state is recorded — the promotion is treated as unconfirmed and re-evaluated on the next tick. |
| ERROR | DNS flip failed after successful promotion | site, error, fg | Promotion and writable confirmation both succeeded, so the failover state (cooldown, split-brain target, promotionGtidExecuted) and bloodraven_failovers_total are already recorded; only the DNS update failed. bloodraven_dns_flips_total is left unincremented. The poll loop reconciles DNS against the current active site (see DNS reconciled to active site below), so a transient failure such as an RBAC denial self-heals once the write is permitted again — MySQL has already promoted regardless. |
| WARN | DNS reconcile failed | site, target, error, fg | The poll-driven DNS reconcile tried to point the record at the current active site and the write was rejected. Logged once per failing episode, not once per poll: while the failure persists the retry continues silently (DEBUG DNS reconcile still failing) and MySQL is unaffected. |
| INFO | DNS reconciled to active site | site, target, fg | The DNS record diverged from the current active site and was repaired — a promotion-time flip that had failed, a record left stale by an operator restart, or an out-of-band edit. bloodraven_dns_flips_total{site} increments here, and only when the record's value actually changed. No promotion is re-run and MySQL is not touched. |
Supporting events emitted inside Execute:
| Level | msg | Fields |
|---|---|---|
| INFO | fenced old primary with super_read_only=ON | fg |
| WARN | failed to fence old primary (may be unreachable) | error, fg |
| INFO | killed app connections on old primary | count, fg |
| WARN | failed to kill app connections on old primary | error, fg |
| INFO | relay log drain complete | fg |
| WARN | relay log drain did not complete cleanly, proceeding with promotion | error, fg |
Divergence and recovery
Fired after an emergency failover when the operator inspects the returning old primary.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | initiating old primary recovery | oldPrimary, newPrimary, fg | Recovery sequence starting. |
| INFO | no GTID divergence, auto-recovering old primary as replica | site, fg | Old primary's GTID set is a subset of the new primary's — safe to attach as replica. |
| WARN | divergence detected | site, divergentTransactions, divergentGtid, oldPrimaryGtid, newPrimaryGtid, fg | Old primary has committed transactions the new primary never saw. Operator does not auto-recover — the admin must reclone the site or replay the divergent set onto the new primary (containment then auto-rejoins). Emitted when divergence is first detected and again whenever the periodic re-verification finds the set CHANGED — an unchanged report is not re-logged each cycle. Mirrored by the bloodraven_divergent_transactions gauge and the DataLossDetected Kubernetes Event. |
| INFO | old primary recovery complete | site, source, fg | Old primary is now replicating from the new primary. source is the new primary's host. |
| ERROR | old primary recovery failed | site, error, fg | One step of the recovery sequence (fence / GTID query / CHANGE REPLICATION SOURCE / START REPLICA) returned an error. |
| INFO | recovery state cleared (site is now replicating) | site, fg | The site is a healthy, source-converged replica again, so its recovery marker is dropped and bloodraven_divergent_transactions{site} returns to 0. The normal terminal state for both RecoveryInProgress and an externally resolved RecoveryBlocked. |
| INFO | recovery state cleared (site is writable) | site, fg | The site is writable, so it is no longer "recovering as a replica". A RecoveryBlocked report is not dropped merely because the site turned writable — a rogue-writable site keeps its live divergence report (split-brain fencing re-fences it, and the periodic re-verification refreshes the report). The one case a blocked report clears here is when the site is the operator's own failover target and the unique writable primary, which makes the report definitionally stale. |
Replication source convergence
After topology changes, Bloodraven verifies that every follower replicates
directly from the uniquely confirmed active primary. These events cover
candidate, dr-only, and read-only followers; they are separate from the
old-primary recovery events above.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | replication source convergence started | site, activeSite, currentSource, expectedSource, fg | A follower needs a source or thread-state correction and passed the initial mutation gates. |
| INFO | replication source convergence complete | site, source, fg | The canonical source is the active primary and both replication threads are running. |
| WARN | replication source convergence blocked | site, activeSite, stage, followerGtid, activeGtid, fg | GTID containment failed before or after stopping replication. No source change is issued. |
| ERROR | replication source convergence failed | site, activeSite, stage, error, fg | A bounded source mutation or verification attempt failed. The next poll can retry safely. |
Stable stage values include pre-stop-gtid, post-stop-gtid, stop,
change-source, start, and verify. Use the status
sourceConvergenceState and sourceConvergenceReason for current state; use
these logs for the detailed failure and GTID evidence.
Bootstrap and reclone
starting bootstrap is the single canonical event for "we are about to clone a replica". The source field disambiguates why:
source value | Meaning |
|---|---|
fresh-deploy | Initial bootstrap of a new failover group; donor is the seed site. |
auto-clone | Operator detected an empty replica during steady-state and is recovering it without an admin trigger. |
reclone | Admin set the bloodraven.shipstream.io/reclone-site=<name> annotation, the safety interlock passed, and the operator is wiping the named site. This is the reclone-started event. |
| Level | msg | Fields |
|---|---|---|
| INFO | starting bootstrap | source, donor, recipient, donorHost, fg |
| INFO | cloning from primary | donor, fg |
| INFO | clone completed successfully | replica, fg |
| INFO | setting up replication | source, fg |
| INFO | replication started successfully | source, fg |
| INFO | bootstrap completed successfully | source, fg |
| ERROR | bootstrap failed | source, error, fg |
| INFO | clone returned expected connection drop, waiting for restart | error, fg |
| INFO | replica already has primary data (prior clone detected), skipping clone phase | fg |
A reclone-only narrative is therefore: filter msg="starting bootstrap" AND source="reclone" for the trigger event, then watch for bootstrap completed successfully (source="reclone") or bootstrap failed (source="reclone").
State transitions
Every per-site state change emits one record. Use this to replay the topology timeline.
| Level | msg | Fields |
|---|---|---|
| INFO | state transition | site, from, to, fg |
from and to values: unknown, unreachable, read-only, writable. Mirrored by the bloodraven_state_transitions_total counter.
Topology decisions
| Level | msg | Fields | Notes |
|---|---|---|---|
| WARN | ALERT | message, fg | A cross-site EvalCrossSite action returned an alert string (split brain, no primary, total loss). The same conditions emit SplitBrainDetected / NoPrimaryDetected / TotalLossDetected Kubernetes Events. |
| WARN | split-brain auto-resolve: fencing non-preferred site per spec.splitBrainPolicy.sitePriorities | (context) | Opt-in splitBrainPolicy is fencing the lower-priority site. The poll-driven retry of a persistent split-brain re-emits this at WARN at most every 30s (DEBUG in between). |
| INFO | fencing returning old primary (split brain after failover) | site, fg | A second site came back writable after a failover; the operator is fencing every writable site except the one holding live primary authority. Emitted at INFO on the first fence of a given split-brain and at DEBUG on the poll-driven retries in between. |
| ERROR | failed to fence returning old primary | site, error, fg | The super_read_only=ON write above failed. The next poll retries. DEBUG on retries, matching the INFO/DEBUG pairing of the fence itself. |
| INFO | post-fence application connections evicted | site, count, fg | Operator-side recovery removed one or more sessions that survived the sidecar's best-effort fence eviction. It retries on later topology polls until an empty pass or timeout, without blocking the poll loop between passes. |
| WARN | post-fence application connection drain retry failed | site, error, fg | A recovery drain pass could not enumerate or kill sessions. The operator retries within spec.connectionDrainTimeout. |
| INFO | post-fence application connection drain complete | site, fg | An empty eviction pass proved no application sessions remain. Old-primary recovery may now complete. |
| WARN | post-fence application connection drain timed out | site, timeout, fg | The dedicated drain budget expired. Recovery proceeds because super_read_only=ON still prevents writes; a surviving session is limited to stale reads. |
| WARN | re-asserting fenced promoted primary: no site is writable and the last failover target is GTID-complete; restoring writability | site, fg | The last failover target was found fenced (read-only) with every site reachable and nothing writable — typically its own sidecar re-fenced it with a stale lease right after a promotion. The operator restores writability on the target. Mirrored by bloodraven_primary_reassert_total. Rate-limited to once per failoverCooldown. |
| WARN | primary re-assert refused: peer has transactions the target lacks — divergence needs human review | site, peerGtid, targetGtid, fg | The no-writable-site wedge was detected but restoring the last failover target would abandon peer transactions. The group stays read-only until an admin resolves the divergence. |
| WARN | primary re-assert refused: target no longer contains the recorded promotion GTID set (wiped or restored since promotion?) | site, promotionGtid, targetGtid, fg | The failover history no longer describes the target's data lineage; the operator will not restore writability automatically. |
| WARN | primary re-assert refused: recorded promotion GTID set failed to parse — status corrupted or manually edited? | site, promotionGtid, error, fg | status.promotionGtidExecuted is non-empty but malformed. The operator wrote this value from MySQL itself, so a parse failure means corruption or manual tampering — the re-assert safety argument depends on it, so the operator refuses. |
| INFO | failover blocked by anti-flap cooldown | (context) | A failover decision was deferred because failoverCooldown has not elapsed since the last one. |
| WARN | ignored out-of-order local failover record | target, lastFailover, currentTarget, currentLastFailover, fg | Two local promotion paths reported out of timestamp order; the newer in-process anti-flap pair remains authoritative. |
| ERROR | out-of-band anti-flap state write failed; retrying every poll | fg, target, lastFailover, error | The annotation write was rejected. The newest record stays pending and is retried on each poll. |
| INFO | cross-site action deferred: in-place restore in progress | fg | Decisions are paused while restoreInPlace runs. |
| INFO | cross-site action deferred: planned failover in progress | fg | Decisions are paused while a planned-failover annotation is being processed. |
Sidecar fencing
The per-MySQL sidecar emits these in its operational stream. SELF-FENCING: is a stable prefix — msg strings that begin with it indicate the sidecar wrote super_read_only=ON to its local MySQL without operator instruction.
| Level | msg | Fields | Notes |
|---|---|---|---|
| ERROR | SELF-FENCING: topology mismatch — operator-authoritative active site disagrees with our site, setting super_read_only=ON | site, authoritativeActiveSite, observedAt, pod | The operator (or a peer relaying the operator's view) reports a different active site than this sidecar is on. Fired even when the operator is reachable. |
| ERROR | SELF-FENCING: Bloodraven and every peer unreachable beyond lease timeout, setting super_read_only=ON | bloodravenLastOk, latestPeerOk, peers, leaseTimeout, pod | Backstop rule: nothing is reachable, so we can't be sure we're still primary. |
| INFO | SELF-FENCING: killed app connections | count, pod | Application sessions evicted after fencing succeeded. Server-internal threads are spared, so a fenced replica keeps applying from the authoritative primary. |
| WARN | SELF-FENCING: failed to kill connections after fencing | error, count, pod | The fence write landed but eviction was incomplete. count is how many sessions were killed — 0 when the process-list query itself failed, so no session was ever enumerated. error joins every cause that applied: the listing query failed, iteration ended early, some rows would not scan, some KILLs were refused, or the fence hit its 20s deadline. super_read_only=ON still holds, so surviving sessions cannot write. The sidecar does not retry because it cannot safely distinguish a pooled operator promotion session. After promotion, the operator drains the fenced former primary until a pass finds no sessions or spec.connectionDrainTimeout expires. The sidecar's /status reports self_fenced, scoped to the current sidecar process. |
| WARN | SELF-FENCING: super_read_only write failed but the fence is in place; skipping connection eviction | error, pod | The SET GLOBAL returned an error, but a follow-up @@super_read_only read proved the write landed. The fence counts as established: /status reports self_fenced=true and SELF-FENCED follows. Eviction is skipped because the fence budget is spent; operator-side recovery performs the bounded follow-up drain. |
| ERROR | SELF-FENCING: super_read_only write outcome is unconfirmed; will probe again | error, probeError, pod | Both the SET GLOBAL result and its independent confirmation probe were unavailable. The monitor retains a process-local pending marker and retries @@super_read_only on later ticks instead of getting stuck behind read_only=ON with self_fenced=false. |
| WARN | fencing: could not resolve pending super_read_only write | error, pod | A later tick still could not read @@super_read_only; the pending result remains armed for another tick. |
| WARN | SELF-FENCING: previously unconfirmed super_read_only write is now confirmed | pod | A later tick proved the ambiguous write established super_read_only=ON; /status now reports self_fenced=true. |
| ERROR | SELF-FENCING FAILED: could not set super_read_only | error, pod | The fence write failed and a follow-up @@super_read_only read confirmed the instance was not super-fenced. The sidecar retries on the next tick. |
| WARN | fencing: could not confirm whether the super_read_only write landed | error, pod | The immediate @@super_read_only confirmation read failed. The pending outcome is retained and retried on later ticks. |
| ERROR | SELF-FENCED: super_read_only=ON has been set, only Bloodraven can restore | pod | Final status; the sidecar will not unfence on its own. The next operator promotion clears it. |
| INFO | fencing: MySQL is writable after prior self-fence; rearming monitor | pod | An actor with SUPER privileges (the operator, per the restore contract) made MySQL writable again after a self-fence. The monitor re-arms with a fresh lease window — it will not re-fence until a full leaseTimeout passes with the operator and every peer unreachable again. |
| INFO | fencing: adopted active-site view from peer | peer, activeSite, observedAt, pod | Peer sidecar relayed a fresher view than what this sidecar had cached. Drives the topology-mismatch rule. |
Safety-net events (sidecar startup):
| Level | msg | Fields |
|---|---|---|
| INFO | safety net: set super_read_only=ON as precaution on startup | pod |
| INFO | safety net: this is the active site, clearing super_read_only | site, pod |
| INFO | safety net: confirmed standby site, staying fenced | site, activeSite, pod |
| INFO | safety net: no active site reported by operator, staying fenced | pod |
| WARN | safety net: could not query active site, staying fenced | error, pod |
| ERROR | safety net: failed to clear super_read_only on active site | error, pod |
PITR archiver
Emitted by the sidecar's BinlogArchiver.
| Level | msg | Fields |
|---|---|---|
| INFO | binlog archiver starting | storageType, binlogDir, binlogIndex, pollInterval, pod |
| INFO | archived sealed binlogs | count, pod |
| INFO | retention sweep complete | (sweep stats), pod |
| WARN | archive binlog | file, error, pod |
| WARN | retention: delete object | key, error, pod |
Per-upload success/failure is also reflected in the bloodraven_archiver_upload_failures and bloodraven_archiver_last_upload_timestamp_seconds metrics — prefer those for alerting.
Data-at-rest encryption
Emitted when spec.encryptionAtRest.enabled=true. See
Data-at-rest encryption for the lifecycle
these events trace.
Operator events:
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | keyring phase transition | site, from, to, reason, version, fg | The per-site keyring lifecycle advanced. The normal sequence is Pending → Unsealed → Escrowed → Sealed. Any transition to Unsealed from Sealed means the site is deliberately running with a writable keyring (clone or rotation). |
| INFO | keyring escrowed | namespace, group, site, version, secret, digest | The operator accepted a sidecar escrow push and stored a new immutable Secret version. Also emitted by the sidecar with site, version, secret, digest, pod. |
| INFO | refusing to enable encryption at rest on a live failover group | activeSite, fg | spec.encryptionAtRest.enabled was turned on for a group that is already serving. Rendering is unchanged; existing tablespaces would have stayed plaintext. |
| WARN | keyring escrow rejected | namespace, group, site, reason | A push to /keyring/escrow failed authentication. The response to the caller is deliberately vague; the reason is only here. |
| WARN | keyring escrow digest mismatch | namespace, group, site, claimed, computed | An escrow push carried a digest that did not match its payload. Nothing was stored. |
| ERROR | keyring escrow: store failed | namespace, group, site, error | The operator could not write the escrow Secret. The site stays unsealed and the sidecar keeps retrying. |
Sidecar events:
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | keyring agent enabled | path, escrowArmed, rotate, pod | Sidecar startup. escrowArmed=false means this pod renders a sealed keyring and only reports its digest. |
| INFO | rotated innodb master key | site, pod | ALTER INSTANCE ROTATE INNODB MASTER KEY succeeded. |
| INFO | encrypted the mysql system tablespace | site, pod | ALTER TABLESPACE mysql ENCRYPTION='Y' succeeded on the writable site. |
| WARN | keyring escrow push failed, will retry | error, site, digest, pod | The push was rejected or the operator echoed a digest that did not match. Escrow is retried until it succeeds; the site is not sealed in the meantime. |
| WARN | could not encrypt the mysql system tablespace | error, site, pod | Retried on the next tick. |
| ERROR | keyring rotation failed | error, site, pod | The rotation statement failed. The agent still escrows whatever is on disk, because a failed rotation may already have written a key. |
Kubernetes Event reasons emitted on the MysqlFailoverGroup:
| Reason | Type | When |
|---|---|---|
KeyringPhase | Normal / Warning | Any keyring phase transition. Warning when the new phase is Failed. |
KeyringUnsealed | Normal | A site was unsealed so a CLONE INSTANCE can rewrap tablespace keys. |
KeyringRotationRefused | Warning | A rotation was requested for the active primary, or while an ordered update or planned failover was in flight. |
KeyringEscrowMissing | Warning | A sealed site's escrow Secret no longer exists. That site cannot restart. See the DR runbook. |
KeyringEscrowCorrupt | Warning | A sealed site's escrow Secret no longer hashes to the recorded digest. |
KeyringDigestMismatch | Warning | A pod rendered sealed is running a keyring that does not match its escrow; the operator refuses to call it sealed. |
KeyringNotReadOnly | Warning | MySQL reports a writable keyring on a site the operator considers sealed. |
EncryptionAdoptionRefused | Warning | Encryption was enabled on a group that is already serving. |
Dragonfly
Bloodraven optionally co-manages per-site Dragonfly instances and emits the following events when spec.dragonfly.enabled=true. Mirrored by the bloodraven_dragonfly_site_up gauge and the bloodraven_dragonfly_promotions_total{result} counter, plus the matching Dragonfly* Kubernetes Events on the MysqlFailoverGroup.
| Level | msg | Fields | Notes |
|---|---|---|---|
| INFO | dragonfly: configured replica | site, host, port, fg | Operator issued REPLICAOF against a non-active site to align it with the active master. |
| WARN | dragonfly: stale master on non-active site | site, active, fg | A site reports role=master but is not the active site. Auto-rejoin is attempted only when the stale instance has connected_slaves=0 AND master_repl_offset=0 (provably never accepted writes); otherwise the stale master is shed from the active Service via the traffic-label gate and left for human intervention. |
| INFO | stale-master reconfigure: REPLICAOF applied | site, host, port, fg | Auto-rejoin succeeded: the stale master is now linked as a replica of the active master. |
| WARN | stale-master reconfigure: REPLICAOF failed | site, host, port, error, fg | Auto-rejoin attempt failed; the next tick retries. |
| INFO | client-kill: evicted clients from old master | site, fg | After a planned-failover Dragonfly promotion, the operator issued CLIENT KILL TYPE NORMAL against the demoted source so application clients reconnect through the active Service. |
| INFO | dragonfly/mysql active-site drift: promoting Dragonfly replica to match MySQL | oldSource, target, mysqlActiveSite, fg | MySQL active site and Dragonfly master diverged; the manager is promoting the synced Dragonfly replica on the MySQL active site. |
| INFO | dragonfly-only emergency: active master unreachable; promoting replica | oldSource, target, fg | Dragonfly master failed without a MySQL failover; the manager is promoting the single healthy replica and leaving MySQL status.activeSite unchanged. |
| INFO | dragonfly emergency: REPLTAKEOVER succeeded | site, fg | After an emergency MySQL failover, Dragonfly was promoted with sessions preserved. |
| WARN | dragonfly emergency: REPLTAKEOVER failed; falling back | site, error, fg | Emergency promote could not preserve sessions; falling back to REPLICAOF NO ONE. |
| INFO | dragonfly emergency: target promoted via REPLICAOF NO ONE (sessions lost) | site, fg | Emergency promote completed with empty cache. |
| WARN | dragonfly emergency: REPLICAOF NO ONE failed | site, error, fg | Both promotion paths failed; cache is unavailable. MySQL emergency failover was not affected. |
| WARN | dragonfly emergency: target unreachable; skipping promotion | site, error, fg | Bounded budget expired before the operator could reach the target. |
Kubernetes Event reasons emitted on the MysqlFailoverGroup (visible via kubectl describe):
| Reason | When |
|---|---|
DragonflyPromotionStarted | Planned-failover state machine entered PromotingDragonfly. |
DragonflyPromotionCompleted | Dragonfly target was promoted (planned or emergency). |
DragonflyPromotionFailed | Promotion command failed; behavior depends on spec.dragonfly.plannedFailover.onSyncTimeout (planned) or is best-effort (emergency). |
DragonflyStaleMasterDetected | A non-active site reports master role. Logged + dedup'd in 5-minute windows. Auto-rejoin is attempted in reconcileReplication when connected_slaves=0 AND master_repl_offset=0. |
DragonflyOldSiteReconfigured | A stale master passed the auto-rejoin gate and was attached as a replica of the active master via REPLICAOF. |
DragonflySyncTimeout | WaitingForDragonflySync exhausted spec.dragonfly.plannedFailover.maxSyncWait. |
DragonflyUpgradeStarted | Snapshot-restore Dragonfly upgrade annotation was accepted and status.dragonfly.upgrade was initialized. |
DragonflyUpgradeRejected | Snapshot-restore upgrade request was invalid or another coordinated operation was running. |
DragonflyUpgradeSnapshotStarted | Active Dragonfly traffic was shed and the operator is about to issue SAVE. |
DragonflyUpgradeSnapshotCompleted | SAVE completed against the active Dragonfly master using spec.dragonfly.snapshot.dir. |
DragonflyUpgradeCompleted | Active and replica Dragonfly pods are on the target image, active traffic is restored, and replicas are linked. |
DragonflyUpgradeFailed | Snapshot-restore upgrade reached a terminal failure; the operator best-effort restored active traffic. |
Lifecycle
| Level | msg | Fields |
|---|---|---|
| INFO | starting bloodraven manager | (none) |
| INFO | starting auxiliary HTTP server | addr |
| INFO | topology manager runner starting | (none) |
| INFO | starting topology manager | fg |
| INFO | topology manager stopped | fg |
| INFO | stopping topology manager | fg |
| INFO | config changed, restarting topology manager | fg |
| INFO | restored lastFailoverTarget from CR status | fg, target |
| INFO | restored lastFailover from CR status | fg, lastFailover |
| WARN | restored lastFailoverTarget from out-of-band annotations | fg, target, statusTarget |
| WARN | restored lastFailover from out-of-band annotations | fg, lastFailover, statusLastFailover |
| ERROR | out-of-band anti-flap annotation unreadable; falling back to CR status | fg, error |
| ERROR | CR status anti-flap state unreadable; ignoring unsafe copy | fg, error |
| INFO | starting graceful shutdown | fg |
| INFO | CR deleted — DNSEndpoint will be garbage-collected | (none) |
| INFO | sidecar image version differs from the operator | operatorImage, sidecarImage, operatorTag, sidecarTag |
| INFO | sidecar starting | listenAddr, peerAddresses, bloodravenAddress, leaseTimeout, peerCheckInterval, site, namespace, fg, pod |
| INFO | sidecar stopped | pod |
| INFO | received signal, shutting down | signal, pod |
Kubernetes Event reasons emitted on the MysqlFailoverGroup:
| Reason | Type | When |
|---|---|---|
SidecarVersionSkew | Warning | spec.sidecarImage is tagged with a different release than the running operator. The two halves share a rendering contract, so a mismatch can break the sidecar silently — on a spec.tls group an older sidecar cannot reach MySQL at all. Advisory only: the operator reports skew but never blocks on it, because a re-tagged or locally built image is indistinguishable from a real mismatch. |
Stability commitments
| What | Stability |
|---|---|
msg strings listed in the Event reference | Stable. Changes go through a deprecation note in CHANGELOG.md. |
Field names listed alongside a stable msg | Stable. New fields may be added to existing events; existing fields will not be renamed or removed without a deprecation note. |
| Field value shapes (strings, numbers, durations) | Stable for the values listed. GTIDs are passed through verbatim from MySQL — their shape is whatever MySQL emits. |
time, level, msg field names themselves | Stable. Tied to log/slog defaults. |
DEBUG-level records | Unstable. May appear, disappear, or change shape without notice. Disabled by default. |
Ad-hoc INFO/WARN/ERROR records not listed above (e.g. retry warnings, transient probe errors) | Best-effort. Field set is intended to be useful but not contractual. Don't build alerts that key on the exact msg string. |
Controller-runtime (zap) stream | Inherited from upstream. Bloodraven does not redefine this stream's shape. |
Pipeline integration tips
Filtering operational vs. controller-runtime
Most aggregators (Loki, Elasticsearch, Vector) let you split streams by JSON shape. A reliable predicate:
$.time && $.msg // operational (slog)
$.ts && $.logger // controller-runtime (zap)
Per-event alerts
Because every key event has a stable msg, pipeline alerts can be expressed as exact-match filters rather than fragile regexes. Examples for Loki:
# Failover started
{app="bloodraven"} | json | msg = "initiating failover"
# Failover failed (escalate)
{app="bloodraven"} | json | level = "ERROR" and msg = "failover failed"
# Divergence requires manual reclone
{app="bloodraven"} | json | msg = "divergence detected"
# Reclone triggered (track who/what asked for it via fg + recipient)
{app="bloodraven"} | json | msg = "starting bootstrap" and source = "reclone"
# Sidecar self-fenced — page on this
{app="bloodraven-sidecar"} | json | msg =~ "^SELF-FENCING:"
Correlating with metrics and Kubernetes Events
Several stable log events are mirrored by other observable signals — when one fires, the others fire too:
| Log event | Metric | Kubernetes Event |
|---|---|---|
failover complete | bloodraven_failovers_total{target_site} | FailoverExecuted |
divergence detected | bloodraven_divergent_transactions{site} > 0 | DataLossDetected |
old primary recovery complete | bloodraven_divergent_transactions{site} returns to 0 | RecoveryComplete |
state transition | bloodraven_state_transitions_total{site, from, to} | (none — too noisy for events) |
Prefer metrics for alert thresholds and Kubernetes Events for human notification routing; logs are richest for forensics and timeline reconstruction.
Useful structured fields to index
If your pipeline supports indexing specific fields, the high-value ones are:
fg— partitions everything by failover groupsite(andoldPrimary/newPrimary/promotedSite/donor/recipient) — for per-site timelineslevel— for severity routingsource— for bootstrap/reclone disambiguationerror— full error string from the operator'serrorchain