Bloodraven in Production Emergency failover, end to end / The post-failover audit report

The post-failover audit report

Goal

(Optional — see the brief for the shorter jq route and what skipping it costs you.) Build a tool that turns a post-failover status into an incident record — when it happened, from which site to which, the promotion GTID, the divergent set and its transaction count, and a verdict of RPO 0 or N transactions lost — so that the number you report after an outage is measured rather than assumed.

Unit 3 — Emergency failover, end to end · Project · code-notebook · Python

Optional. Skip the Python and you still need the number, so read Without Python — the jq route at the end of this brief before you decide: it produces the same audit in four commands by letting MySQL do the GTID arithmetic, which is what you would actually type during an incident. What the full project adds is the parser — tagged MySQL 9.x sets, interval splitting, and the reason divergentTransactionCount is a cross-check rather than an input — and a verdict you can put in a ticket without a MySQL to hand.

Goal

Build a tool that turns a post-failover status into an incident record — when it happened, from which site to which, the promotion GTID, the divergent set and its transaction count, and a verdict of RPO 0 or N transactions lost — so that the number you report after an outage is measured rather than assumed.

Running example: the playground failover group on the three-site playground — iad, pdx, and the reader. Grading runs against the captures in tests/fixtures/, so you do not need a live cluster to finish or to be marked.

How this works

playground failed over. pdx took writes about twelve seconds after you held iad down, and the incident review wants a number. braudit turns one captured status into an incident record: when the promotion happened, which site it moved to, the promotion GTID, the divergent set held by every other site, and a verdict — RPO 0, N transactions lost, or UNMEASURED.

Capture the artefact from your own playground:

kubectl -n bloodraven-playground get mysqlfailovergroup playground -o json > playground-after-failover.json
python3 braudit.py playground-after-failover.json

No cluster to hand? tests/fixtures/playground-recovery-blocked.json is that capture, taken after the promotion this unit drove, with iad back and blocked on four divergent transactions.

The arithmetic is the operator’s own. There is no divergence exactly when the new primary’s GTID set contains the old primary’s; what it does not contain is the difference, GTID_SUBTRACT(old, new), and the cardinality of that difference is your lost-transaction count. You compute it here, from the sets, for every site that is not the active one — including the reader, which is where a site that came up writable for a few seconds before the sidecar fenced it will show up.

Your tasks

TODO A — parse_gtid_set(text). Return {uuid: [(start, end), ...]}. Accept several UUIDs separated by commas, several intervals for one UUID separated by colons (uuid:1-19:25-30), a single-transaction interval written bare (uuid:7), and newlines anywhere, because a captured set is folded across lines. Empty string gives {}. Anything else raises ValueError — including a MySQL 9.x tagged set such as uuid:Domain_1:1-3, because a tag is part of the UUID’s identity and quietly dropping it would understate the count.

TODO B — transaction_count(gtid_set). The number of transactions in a parsed set. An interval (20, 23) is four transactions, not one and not three.

TODO C — gtid_subtract(minuend, subtrahend). GTID_SUBTRACT: the transactions in the first set that are not in the second. Subtracting from the middle of an interval splits it — {u: [(1, 32)]} minus {u: [(1, 19), (25, 30)]} is {u: [(20, 24), (31, 32)]}, which is seven transactions. A replica’s gtid_executed really does look like that mid-catch-up with a parallel applier, so this is not a contrived case.

TODO D — select_failover_record(obj, now). The operator writes the failover record twice, to the status subresource and to the bloodraven.shipstream.io/last-failover / bloodraven.shipstream.io/last-failover-target annotation pair, because the two paths fail independently. Rehydrate it the way the operator does after a restart: discard a copy stamped more than five minutes ahead of now, discard a copy whose target is not a site in spec.sites, take the later of what survives, and give a tie to status — equal instants describe the same promotion. Return {"at": datetime|None, "target": str, "source": "status"|"annotations"|"none"}.

TODO E — audit(obj, now). Build the record. Apply the verdict rules in this order and stop at the first that matches:

  1. The active site is missing from status.sites, or its gtidExecuted is empty → UNMEASURED.
  2. status.promotionGtidExecuted is empty → UNMEASURED. The failover step that records it is non-fatal, so a promotion can complete without it, and a capture without it cannot be audited.
  3. The active site’s gtidExecuted does not contain promotionGtidExecuted → UNMEASURED. The site writing now is not the site this record describes.
  4. Any non-active site has an empty gtidExecuted → UNMEASURED. An unknown site is not a zero.
  5. Total is 0 → RPO 0.
  6. Otherwise → N transactions lost.

What the scaffolding is for

Argument parsing, capture loading, parse_time/fmt_time, render_gtid_set, render and the exit plumbing are wired. So is gtid_contains, written as not gtid_subtract(subset, superset) — it is there to show that containment is subtraction asked the other way round, which is why TODO C buys you both halves of the operator’s test.

Expected output

$ python3 braudit.py tests/fixtures/playground-recovery-blocked.json --now 2026-04-30T21:00:00Z
group: bloodraven-playground/playground
failoverAt: 2026-04-30T20:55:52Z
recordSource: status
from: iad
to: pdx
promotionGtidExecuted: a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:1-19,a3c3f9e8-5f9d-11f1-bf37-568bfb8d0365:1-7
site: iad divergent=a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:20-23 lost=4
site: reader divergent=- lost=0
lost: 4
verdict: 4 transactions lost
$ echo $?
1

Exit 0 for RPO 0, 1 when transactions were lost, 2 for UNMEASURED.

Rules

Steps

How this is graded

Four automated test cases (weights 40 / 20 / 25 / 15) run against the fixtures in tests/; run them yourself at any point with python3 tests/run_tests.py. A human marks the four criteria in rubric.md.

Two of the test cases are adversarial. One capture in tests/fixtures/ was taken before the operator had computed any divergence, and one has a status write that failed while its annotation write succeeded. A tool that reads the easy field passes the canonical case and fails both.

Without Python — the jq route

The GTID arithmetic does not have to be yours. MySQL ships the two functions the operator itself uses, so with a live cluster you can produce the same audit in four commands — and this is what you would actually type during an incident, because it needs nothing you had to write in advance.

NS=bloodraven-playground; FG=playground
CAP=$(kubectl -n $NS get mysqlfailovergroup $FG -o json)

# 1. What the operator recorded at the moment of promotion.
echo "$CAP" | jq -r '"activeSite=\(.status.activeSite)  target=\(.status.lastFailoverTarget)  at=\(.status.lastFailover)"'
PROMO=$(echo "$CAP" | jq -r '.status.promotionGtidExecuted // "" | gsub("\\s";"")')

# 2. Every non-active site's executed set, straight from status.
echo "$CAP" | jq -r '.status.activeSite as $a | .status.sites[] | select(.name != $a)
  | "\(.name)\t\(.gtidExecuted // "")"'

# 3. Let MySQL do the subtraction — for the old primary, against the promotion set.
OLD=$(echo "$CAP" | jq -r '.status.sites[] | select(.name=="iad") | .gtidExecuted // "" | gsub("\\s";"")')
kubectl -n $NS exec deploy/mysql-$FG-pdx -c mysql -- \
  env MYSQL_PWD=playground-root-pw mysql -uroot -Nse \
  "SELECT GTID_SUBTRACT('$OLD','$PROMO')"

# 4. Cross-check against what the operator published — never as your input.
echo "$CAP" | jq -r '.status.sites[] | select(.divergentGtid != null)
  | "\(.name)\tdivergent=\(.divergentGtid)\tcount=\(.divergentTransactionCount)"'

GTID_SUBTRACT(old, new) is the divergence primitive and its cardinality is your lost-transaction count; GTID_SUBSET(old, new) asked the other way round is the containment test that decides whether the returning site rejoins at all. Both are MySQL’s, not Bloodraven’s, so they work on any pair of sets you can get your hands on.

Two cautions carry over from the full project unchanged. Audit before you reclone — a capture taken afterwards shows no divergence, because the divergent transactions are gone, and that is what a reclone is. And step 4 is a cross-check, not an input: divergentGtid and divergentTransactionCount are published only once the operator has fenced the returning site and run its own comparison, so a capture taken before that has neither, and a tool that reads them as its source of truth reports RPO 0 for an outage it simply arrived too early to measure.

What the Python project adds on top: a parser that refuses a tagged MySQL 9.x set rather than silently under-counting it, interval splitting so a mid-catch-up replica subtracts correctly, the two-copy failover-record rehydration, and a verdict you can produce from a JSON capture alone with no MySQL to hand. If you are on call and you have a cluster, the four commands above are enough.

Steps

How it is graded

CriterionWhat earns itWeight
The count is measured from the GTID setsEvery reported count comes from transaction_count(gtid_subtract(...)) over sets parsed out of the capture. status.sites[].divergentTransactionCount and status.sites[].divergentGtid are never read as inputs (printing them as a labelled cross-check is fine). Subtraction handles multi-interval sets and interval splitting; a set that cannot be parsed raises rather than counting wrong. Full marks require the early-capture fixture — no operator-computed divergence anywhere in it — to report 8.30
The failover record is rehydrated as the operator rehydrates itBoth durable copies are read; a copy more than five minutes ahead of now is discarded; a copy whose target is not in spec.sites is discarded; the later survivor wins; a tie goes to status. recordSource names which copy was used, so a reader can tell a status read from an annotation read. A submission that reads only status caps at half these marks even if it is otherwise correct.25
Three verdicts, applied in order, with matching exit codesRPO 0, N transactions lost and UNMEASURED are distinct outcomes with exit codes 0, 1 and 2. All four unmeasurable conditions are implemented and are checked in the stated order: no active-site gtidExecuted, no promotionGtidExecuted, an active site that does not contain the promotion GTID, and any non-active site with no gtidExecuted. UNMEASURED carries a reason naming the field or site that was missing. No submission that prints a number for an unmeasurable capture can score above half here.25
Craft: legible under incident conditions, loud on bad inputThe report reads top to bottom as an incident record and needs no explanation at 3am: one fact per line, per-site lines showing the actual divergent set beside its count. Functions are small and single-purpose, with the parsing, the arithmetic, the record selection and the verdict separable. Malformed input produces an exception or an UNMEASURED verdict naming the problem — never a silent zero, never a traceback with no message. Comments explain the non-obvious rules (the tie, the skew guard, the field not read), not the syntax.20
Total100

Test cases

Test Checks Expected Weight
canonical_audit_of_playground Correctness on the canonical capture: the post-failover playground status with iad blocked on four divergent transactions. Checks the full record — group, failover instant and source, from/to, the normalised promotion GTID, the per-site divergent sets and counts, the total, the verdict string and the exit code — and checks the clean-rejoin capture reports RPO 0 with exit 0. PASS 40
gtid_arithmetic_and_unmeasurable_captures Generality on awkward input, plus the structural check that the four required functions exist by name and are independently callable with the documented shapes. Multi-interval sets, bare single-transaction intervals, folded newlines, disjoint UUIDs, a tagged set that must raise ValueError, and the three unmeasurable captures: no promotionGtidExecuted, a site with no gtidExecuted, and an active site that does not contain the promotion GTID. PASS 20
catches_trusting_the_operators_divergence_fields Catches a shortcut: reading status.sites[].divergentTransactionCount or divergentGtid instead of subtracting the GTID sets. The early capture was taken while the old primary was still unreachable, so the operator has published neither field — a shortcut reports RPO 0 for a capture holding eight lost transactions, seven on iad across a split interval and one on the reader from the seconds it came up writable. A second case plants a stale divergentTransactionCount of 99 beside sets that say 4. PASS 25
catches_status_only_failover_record Catches a shortcut: reading status.lastFailover/lastFailoverTarget alone, which is what the starter ships. In this capture the status write was the one that failed, so the annotation pair is newer and the status copy names the previous promotion — a status-only tool reports the wrong site and a seventeen-minute-old instant. Also checks the guards that make the other direction safe: a skewed future annotation and a target absent from spec.sites are discarded, and an exact tie goes to status. PASS 15

Starter files

1 files

braudit.py

                    #!/usr/bin/env python3
"""braudit — turn a captured MysqlFailoverGroup into a post-failover incident record.

Usage:
    python3 braudit.py <capture.json> [--now 2026-04-30T21:00:00Z]

The capture is whatever `kubectl get mysqlfailovergroup playground -o json` printed
after the promotion. Everything peripheral is already wired: argument parsing,
capture loading, GTID rendering, report rendering, exit-code plumbing.

Five gaps are marked TODO A .. TODO E. Fill them in the order the brief lists.
The file runs as given and prints a report that is confidently wrong.
"""

import argparse
import json
import sys
from datetime import datetime, timedelta, timezone

# --- Wired: the two durable locations of the failover record -----------------
# The operator writes the same fact twice, deliberately: once to the status
# subresource, once to these annotations. Second precision, RFC 3339.
LAST_FAILOVER_ANNOTATION = "bloodraven.shipstream.io/last-failover"
LAST_FAILOVER_TARGET_ANNOTATION = "bloodraven.shipstream.io/last-failover-target"

# The operator refuses a durable record stamped further than this ahead of its
# own clock, rather than trusting a future timestamp it cannot have written.
CLOCK_SKEW_GRACE = timedelta(minutes=5)


# --- TODO A ------------------------------------------------------------------
def parse_gtid_set(text):
    """Parse a MySQL GTID set into {uuid: [(start, end), ...]}.

    Accepts: 'uuid:1-19', 'uuid:1-19:25-30' (several intervals for one UUID),
    'uuid:7' (a single transaction), several UUIDs separated by commas, and
    newlines anywhere (a captured set is often folded across lines).

    Returns {} for an empty or whitespace-only string.

    Raise ValueError for anything else — including a MySQL 9.x tagged set such
    as 'uuid:Domain_1:1-3'. A tag is part of the UUID's identity, so ignoring it
    would understate the count, and an audit that understates is worse than one
    that refuses.
    """
    return {}  # TODO A


# --- TODO B ------------------------------------------------------------------
def transaction_count(gtid_set):
    """Number of transactions in a parsed GTID set."""
    return 0  # TODO B


# --- TODO C ------------------------------------------------------------------
def gtid_subtract(minuend, subtrahend):
    """GTID_SUBTRACT: the transactions in `minuend` that are not in `subtrahend`.

    Both arguments are parsed sets. Returns a parsed set. Subtracting an
    interval from the middle of another splits it in two: {u: [(1, 32)]} minus
    {u: [(1, 19), (25, 30)]} is {u: [(20, 24), (31, 32)]}.
    """
    return {}  # TODO C


# --- Wired: containment, expressed through subtraction ------------------------
def gtid_contains(superset, subset):
    """True when every transaction in `subset` is also in `superset`.

    This is GTID_SUBSET(subset, superset) with the arguments the way round the
    operator asks the question: does the new primary contain the old one's set?
    """
    return not gtid_subtract(subset, superset)


# --- TODO D ------------------------------------------------------------------
def select_failover_record(obj, now):
    """Pick the authoritative failover record, the way the operator does on restart.

    Read both durable copies — the status subresource and the annotation pair.
    Discard a copy whose timestamp is more than CLOCK_SKEW_GRACE ahead of `now`,
    and discard a copy whose target is not the name of a site in spec.sites.
    Of what survives, the later timestamp wins; a tie goes to status, because
    equal instants describe the same promotion.

    Returns {"at": <datetime or None>, "target": <str>, "source": <str>} where
    source is "status", "annotations", or "none".
    """
    status = obj.get("status") or {}
    # The naive read: trust status and stop. It is right until the status write
    # is the one that failed. Replace it.
    at = parse_time(status.get("lastFailover"))
    return {"at": at, "target": status.get("lastFailoverTarget", ""), "source": "status" if at else "none"}  # TODO D


# --- TODO E ------------------------------------------------------------------
def audit(obj, now):
    """Build the incident record. See the brief for the verdict rules in order."""
    status = obj.get("status") or {}
    meta = obj.get("metadata") or {}
    record = select_failover_record(obj, now)

    return {
        "group": "%s/%s" % (meta.get("namespace", ""), meta.get("name", "")),
        "failoverAt": fmt_time(record["at"]),
        "recordSource": record["source"],
        "from": "-",
        "to": record["target"] or "-",
        "promotionGtidExecuted": "-",  # TODO E: render status.promotionGtidExecuted
        "sites": [],                   # TODO E: one entry per non-active site
        "lost": 0,                     # TODO E: measured total, or None
        "verdict": "RPO 0 — no transaction was lost",  # TODO E
        "exitCode": 0,                 # TODO E: 0 / 1 / 2
    }


# --- Wired: rendering and plumbing -------------------------------------------
def render_gtid_set(gtid_set):
    """Render a parsed set back to MySQL's notation. '-' when the set is empty."""
    if not gtid_set:
        return "-"
    parts = []
    for uuid in sorted(gtid_set):
        intervals = sorted(tuple(iv) for iv in gtid_set[uuid])
        if not intervals:
            continue
        rendered = [str(s) if s == e else "%d-%d" % (s, e) for s, e in intervals]
        parts.append(uuid + ":" + ":".join(rendered))
    return ",".join(parts) if parts else "-"


def parse_time(value):
    """Parse an RFC 3339 timestamp into an aware datetime, or None."""
    if not value:
        return None
    return datetime.fromisoformat(str(value).replace("Z", "+00:00")).astimezone(timezone.utc)


def fmt_time(value):
    if value is None:
        return "-"
    return value.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")


def render(record):
    lines = [
        "group: %s" % record["group"],
        "failoverAt: %s" % record["failoverAt"],
        "recordSource: %s" % record["recordSource"],
        "from: %s" % record["from"],
        "to: %s" % record["to"],
        "promotionGtidExecuted: %s" % record["promotionGtidExecuted"],
    ]
    for site in record["sites"]:
        lines.append("site: %s divergent=%s lost=%s"
                     % (site["name"], site["divergent"],
                        "-" if site["lost"] is None else site["lost"]))
    lines.append("lost: %s" % ("-" if record["lost"] is None else record["lost"]))
    lines.append("verdict: %s" % record["verdict"])
    return "\n".join(lines)


def main(argv=None):
    ap = argparse.ArgumentParser(description="Post-failover audit for a MysqlFailoverGroup capture.")
    ap.add_argument("capture", help="path to `kubectl get mysqlfailovergroup <name> -o json` output")
    ap.add_argument("--now", default=None,
                    help="RFC 3339 instant to audit against (default: now, UTC)")
    args = ap.parse_args(argv)

    with open(args.capture, "r", encoding="utf-8") as fh:
        obj = json.load(fh)

    now = parse_time(args.now) if args.now else datetime.now(timezone.utc)

    record = audit(obj, now)
    print(render(record))
    return record["exitCode"]


if __name__ == "__main__":
    sys.exit(main())

Projects are not auto-graded here. The rubric and the test cases above are the grading contract — run them yourself on your own machine.

Erase saved progress?

This erases all quiz scores, reading progress, project checklists, and your name on the certificate. It cannot be undone, and it affects only this course in this browser.