Bloodraven in Production Where failover meets your application / Make the writer survive

Make the writer survive

Goal

Instrument a writer against playground, run both a planned and an emergency failover underneath it, and produce a drill record with the measured write-gap for each — so the recovery time you claim for your application is one you observed rather than one you assumed.

Unit 4 project — Where failover meets your application. Running example: the playground failover group on the three-site bloodraven-playground.

Goal

Instrument a writer against playground, run both a planned and an emergency failover underneath it, and produce a drill record with the measured write-gap for each — so the recovery time you claim for your application is one you observed rather than one you assumed.

How this works

playground has moved its primary before. What you have never done is measure what that cost the application. This project produces that number twice — once for a planned move and once for an emergency one — and then makes you defend it.

The number is the write-gap: the interval between the last write your application completed against the demoted site and the first it completed against the promoted one. Not the first successful query. Not the first successful read. The first completed write. A read that succeeds during the outage tells you the demoted host is alive — which it usually is — not that you recovered.

You are handed four captures from bloodraven-playground, each a pair:

CaptureWhat happened
emergency-probe.jsonl + emergency-drill.jsoniad scaled to 0, operator promoted pdx. Fixed writer: 30 s connection lifetime, read/write split, retry on 1290/1792
planned-probe.jsonl + planned-drill.jsonbloodraven.shipstream.io/planned-failover=pdx, same fixed writer
planned-probe-unbounded.jsonl + planned-drill-unbounded.jsonthe same planned procedure a day later, pdx -> iad, run against the unfixed writer: one pool for reads and writes, no lifetime bound, no split, no error-class retry
unclosed-probe.jsonl + emergency-drill.jsonthe emergency drill again, with the probe process killed before the gap ever closed

If your playground is up, run both drills yourself and capture your own artefacts in the same shape — brdrill reads either. If it is not, the supplied captures are real enough to do the work, and they are what the grader uses.

Your tasks

Open starter/brdrill.py. Four functions are stubbed, marked TODO A .. TODO D. Each returns None today, which is why the starter prints not computed and exits 1.

TODO A — write_gap(samples, drill). Find the last successful write on demoted_site(drill) and the first successful write on promoted_site(drill) after it. Return oldSite, newSite, lastWriteOldSite, firstWriteNewSite, gapSeconds (rounded to three decimals) and closed. When either end is missing, closed is False and gapSeconds is None — never 0. A gap you did not observe is not a gap of zero.

TODO B — error_classes(samples). Count failed samples into readOnlyRefusal (error code 1290 or 1792), connection (no error code at all — the transport failed, the server never answered) and other. All three keys always present. This is the split your retry policy runs on: retry the refusals, not the lock-wait timeouts.

TODO C — stale_read_window(samples, drill). Count successful read samples served by the demoted site at or after promotion_instant(drill), and report count, first, last and seconds. Two traps: readOnly is true on the reader site by design, so read-only is not the same as stale; and a clean kill produces zero stale reads, because the host is gone.

TODO D — verdict(drill). Return the RPO verdict, in exactly the wording given in the docstring. A planned failover that reached Succeeded with transactionsLost: 0 may claim RPO 0 by construction — the operator fenced the source, snapshotted its GTID_EXECUTED, and promoted only once the target’s set contained that snapshot. An emergency failover may claim nothing of the kind; its RPO comes from divergentGtid on the old primary, which this drill did not look at.

Then run the comparison, and write starter/drill-record.md.

What the scaffolding is for

Everything that is not the measurement is already written: argparse wiring, JSONL and JSON loading, RFC3339 parsing (parse_ts) and rendering (iso), timestamp-ordering of samples, the demoted_site / promoted_site / promotion_instant accessors, record assembly in build_record, both output formats, the --baseline comparison, and the exit codes (0 complete, 1 something not computed, 2 gap never closed). Do not rewrite them; they are there so the four functions are the only thing being graded.

Expected output

$ python3 brdrill.py --probe tests/fixtures/emergency-probe.jsonl \
      --drill tests/fixtures/emergency-drill.json
DRILL RECORD — playground / emergency / iad -> pdx
  namespace            bloodraven-playground
  trigger              kubectl -n bloodraven-playground scale deployment mysql-playground-iad --replicas=0
  promotedAt           2026-08-11T09:14:18.000Z
  lastWriteOldSite     2026-08-11T09:14:05.500Z
  firstWriteNewSite    2026-08-11T09:14:19.500Z
  writeGapSeconds      14.0
  gapClosed            true
  staleReads           0
  staleReadSeconds     0.0
  errors               readOnlyRefusal=0 connection=28 other=0
  verdict              RPO not established by this drill — audit divergentGtid on the old primary
  samples              102 (writes 51, reads 51)

Rules

Steps

How this is graded

Three ways at once: these steps, the human rubric in rubric.md, and four machine tests run from tests/ against the captures in tests/fixtures/. The tests do not need a cluster.

TestWeight
emergency_drill_write_gap40
reads_do_not_close_a_write_gap30
unclosed_gap_is_not_zero15
error_classes_and_verdict15

See rubric.md for the criterion weights and what each one is looking for.

Steps

How it is graded

CriterionWhat earns itWeight
The write-gap is measured from writes, per site, and reports an unclosed gap as unclosedwrite_gap selects on op == "write" and ok being true, uses demoted_site(drill) and promoted_site(drill) rather than literal site names, and requires the closing write to come after the opening one. When either end is absent the result is closed: False with gapSeconds: None, and the CLI prints UNCLOSED and exits 2. Full marks require all three: writes only, sites from the drill, and no zero substituted for an unmeasured gap.30
Stale reads and error classes are counted on the right signalstale_read_window counts successful reads served by the demoted site at or after status.lastFailover, and returns a real zero for the emergency capture rather than a crash or a None. It does not use readOnly as the test, which would sweep in the reader site, which reports read_only=1 by design. error_classes returns all three keys always, puts both 1290 and 1792 in readOnlyRefusal, and puts a null error code in connection rather than other.20
The verdict separates what the drill proved from what it did notverdict claims RPO 0 only for a planned failover that reached Succeeded with transactionsLost: 0, and attributes it to the GTID superset gate rather than to the measured gap. A planned failover in any other phase, and every emergency failover, return a not-established string. Credit is lost if a write-gap is anywhere presented as an RPO, or if an emergency drill is allowed to claim zero loss.20
The drill record is a usable operational artefactdrill-record.md names both drills with their triggers and the direction each moved the primary, carries the measured gaps and the baseline delta, keeps ## Measured and ## Assumed genuinely separate, and states what the drill did not prove. The attribution line subtracts Bloodraven's 12.0 s promotion from the emergency gap and shows the arithmetic. No TODO left behind.15
Craft: clear code, honest failure handling, no invented dataThe four functions are short and readable, return the documented shapes exactly, and do not mutate the loaded samples. Missing ends, empty selections and absent plannedFailover blocks are handled without tracebacks and without silently substituting defaults. No timestamp, count or site name is hardcoded from the fixtures; every printed number is derived from the capture that was loaded.15
Total100

Test cases

Test Checks Expected Weight
emergency_drill_write_gap Correctness on the canonical capture: the clean-kill emergency drill must measure a 14.0 s write-gap from 09:14:05.500Z to 09:14:19.500Z, report zero stale reads (the host was gone), and refuse to claim an RPO. Checks the endpoints, not just the difference. PASS 40
reads_do_not_close_a_write_gap Catches a shortcut: counting any successful sample as recovery. In the baseline capture the unfixed writer keeps getting successful READS from the demoted primary for the whole outage while every write is refused with 1290, so an op-blind implementation reports 2.25 s instead of 63.5 s. The same test also moves the primary pdx -> iad, so a tool that hardcodes the site names from the emergency capture fails here too. PASS 30
unclosed_gap_is_not_zero Generality on an awkward capture: the probe process died two seconds before the promotion landed, so the gap never closed. The tool must report closed: False, gapSeconds: None, print UNCLOSED and exit 2 — a zero here would be a measurement nobody took. PASS 15
error_classes_and_verdict Structural: asserts the error taxonomy exists and is used. Both 1290 and 1792 land in readOnlyRefusal, a null code lands in connection, a lock-wait timeout lands in other, all three keys are always present, and verdict is derived from status.plannedFailover.phase and transactionsLost — a planned failover stalled in WaitingForLag may not claim RPO 0. PASS 15

Starter files

2 files

brdrill.py

                    #!/usr/bin/env python3
"""brdrill — turn a failover drill capture into a drill record.

You give it two artefacts from one drill against the `playground` failover group:

  --probe   a JSONL probe log, one line per write or read attempt your writer made
  --drill   a JSON capture of the drill: what you triggered, and the group status
            afterwards (activeSite, lastFailover, lastFailoverTarget, and for a
            planned move, status.plannedFailover)

It prints a drill record: the measured write-gap, the stale-read window, the
error classes your writer actually saw, and an RPO verdict.

Everything peripheral is wired already — argument parsing, JSONL and JSON
loading, timestamp parsing, record assembly, both output formats, and the exit
codes. Four functions are left for you. Each is marked TODO A .. TODO D and is
referenced by letter from the project instructions.

Usage:
  python3 brdrill.py --probe P.jsonl --drill D.json
  python3 brdrill.py --probe P.jsonl --drill D.json --json
  python3 brdrill.py --probe P.jsonl --drill D.json \
      --baseline B.jsonl --baseline-drill BD.json

Exit codes: 0 record complete and the gap closed; 1 something is still
`not computed`; 2 the gap never closed in this capture.
"""

from __future__ import annotations

import argparse
import json
import sys
from datetime import datetime, timezone

# The two MySQL error codes a write gets from a demoted primary. A pooled
# connection that survived a promotion fails here first — never on a health
# check, which passes, because the server is alive and merely read-only.
READ_ONLY_REFUSAL_CODES = (1290, 1792)


# --------------------------------------------------------------------------
# Wired: loading and parsing.
# --------------------------------------------------------------------------

def parse_ts(value: str) -> datetime:
    """Parse an RFC3339 timestamp from a probe log or a status field."""
    if not isinstance(value, str) or not value:
        raise ValueError("not a timestamp: %r" % (value,))
    text = value.strip()
    if text.endswith("Z"):
        text = text[:-1] + "+00:00"
    dt = datetime.fromisoformat(text)
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    return dt.astimezone(timezone.utc)


def iso(dt):
    """Render a datetime back as RFC3339 with millisecond precision."""
    if dt is None:
        return None
    return dt.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"


def load_probe(path):
    """Load a JSONL probe log. Blank lines are skipped; `ts` is parsed into
    `at`. Samples are returned in timestamp order, whatever order they were
    written in."""
    samples = []
    with open(path, "r", encoding="utf-8") as fh:
        for lineno, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                row = json.loads(line)
            except json.JSONDecodeError as exc:
                raise SystemExit("%s:%d: not JSON: %s" % (path, lineno, exc))
            try:
                row["at"] = parse_ts(row.get("ts"))
            except ValueError as exc:
                raise SystemExit("%s:%d: %s" % (path, lineno, exc))
            samples.append(row)
    if not samples:
        raise SystemExit("%s: no probe samples" % path)
    samples.sort(key=lambda r: r["at"])
    return samples


def load_drill(path):
    """Load a drill capture and check the fields every record needs."""
    with open(path, "r", encoding="utf-8") as fh:
        drill = json.load(fh)
    for section in ("drill", "status"):
        if section not in drill:
            raise SystemExit("%s: missing top-level %r" % (path, section))
    for field in ("mode", "group", "demotedSite"):
        if not drill["drill"].get(field):
            raise SystemExit("%s: missing drill.%s" % (path, field))
    for field in ("lastFailover", "lastFailoverTarget"):
        if not drill["status"].get(field):
            raise SystemExit("%s: missing status.%s" % (path, field))
    return drill


def demoted_site(drill):
    """The site the primary moved off. You record it when you run the drill."""
    return drill["drill"]["demotedSite"]


def promoted_site(drill):
    """The site the primary moved to, straight out of the group status."""
    return drill["status"]["lastFailoverTarget"]


def promotion_instant(drill):
    """When the operator stamped the promotion. `status.lastFailover` is
    RFC3339 at second precision, so it is coarser than your probe log."""
    return parse_ts(drill["status"]["lastFailover"])


# --------------------------------------------------------------------------
# TODO A — write_gap
#
# Return a dict describing the write-gap: the interval between the last write
# your application completed against the demoted site and the first it
# completed against the promoted one.
#
#   {"oldSite": str, "newSite": str,
#    "lastWriteOldSite": datetime|None, "firstWriteNewSite": datetime|None,
#    "gapSeconds": float|None, "closed": bool}
#
# `lastWriteOldSite` is the last sample with op == "write", ok true, and
# site == demoted_site(drill). `firstWriteNewSite` is the earliest sample with
# op == "write", ok true, site == promoted_site(drill), and a timestamp after
# `lastWriteOldSite`. `gapSeconds` is the difference in seconds, rounded to
# three decimals. `closed` is True only when both ends were found; when it is
# False, `gapSeconds` is None — do not fall back to 0.
#
# Reads are not writes. A read that succeeds during the outage tells you the
# site is alive, not that your application recovered.
# --------------------------------------------------------------------------

def write_gap(samples, drill):
    return None  # TODO A


# --------------------------------------------------------------------------
# TODO B — error_classes
#
# Return counts of the failed samples by error class, with all three keys
# always present:
#
#   {"readOnlyRefusal": int, "connection": int, "other": int}
#
# Classify each sample with ok false by its error code:
#   * code in READ_ONLY_REFUSAL_CODES  -> "readOnlyRefusal"
#   * code is null/missing             -> "connection" (the client never got a
#                                          server error; the transport failed)
#   * anything else                    -> "other"
#
# This is the split that decides what your retry policy may retry. Blanket
# retry-everything replays statements that failed for reasons a retry cannot fix.
# --------------------------------------------------------------------------

def error_classes(samples):
    return None  # TODO B


# --------------------------------------------------------------------------
# TODO C — stale_read_window
#
# Return the window during which reads kept succeeding against the demoted
# site after the promotion had already been stamped:
#
#   {"count": int, "first": datetime|None, "last": datetime|None,
#    "seconds": float}
#
# Count samples with op == "read", ok true, site == demoted_site(drill), and a
# timestamp at or after promotion_instant(drill). `seconds` is last - first
# rounded to three decimals, or 0.0 when there are fewer than two.
#
# Two traps. `readOnly` is true on the `reader` site by design, so a reader
# that reports read_only=1 is not stale — the site is. And a clean kill
# produces no stale reads at all: the host is gone, so the reads fail. Zero
# here is a real answer, not a missing one.
# --------------------------------------------------------------------------

def stale_read_window(samples, drill):
    return None  # TODO C


# --------------------------------------------------------------------------
# TODO D — verdict
#
# Return the RPO verdict for this drill as a string. Exactly these three:
#
#   mode "planned", status.plannedFailover.phase == "Succeeded" and
#   transactionsLost == 0:
#     "RPO 0 by construction (target GTID_EXECUTED contained sourceGtidAtFence
#      before promotion)"            <- one line, single spaces
#
#   mode "planned", any other phase:
#     "planned failover did not reach Succeeded (phase=<phase>) — RPO not established"
#
#   mode "emergency":
#     "RPO not established by this drill — audit divergentGtid on the old primary"
#
# A write-gap is not an RPO. The gap says how long your application could not
# write; the RPO says how much committed work the promotion threw away. Only
# the planned path can claim zero, and it claims it from the GTID superset
# gate, not from this capture.
# --------------------------------------------------------------------------

def verdict(drill):
    return None  # TODO D


# --------------------------------------------------------------------------
# Wired: record assembly and output.
# --------------------------------------------------------------------------

def build_record(samples, drill):
    gap = write_gap(samples, drill)
    errors = error_classes(samples)
    stale = stale_read_window(samples, drill)
    writes = sum(1 for s in samples if s.get("op") == "write")
    reads = sum(1 for s in samples if s.get("op") == "read")
    record = {
        "group": drill["drill"]["group"],
        "namespace": drill["drill"].get("namespace"),
        "mode": drill["drill"]["mode"],
        "oldSite": demoted_site(drill),
        "newSite": promoted_site(drill),
        "promotedAt": iso(promotion_instant(drill)),
        "trigger": drill["drill"].get("trigger") or drill["drill"].get("injection"),
        "samples": {"total": len(samples), "writes": writes, "reads": reads},
        "lastWriteOldSite": iso(gap["lastWriteOldSite"]) if gap else None,
        "firstWriteNewSite": iso(gap["firstWriteNewSite"]) if gap else None,
        "writeGapSeconds": gap["gapSeconds"] if gap else None,
        "gapClosed": gap["closed"] if gap else None,
        "staleReads": stale["count"] if stale else None,
        "staleReadSeconds": stale["seconds"] if stale else None,
        "staleReadFirst": iso(stale["first"]) if stale else None,
        "staleReadLast": iso(stale["last"]) if stale else None,
        "errors": errors,
        "verdict": verdict(drill),
        "_complete": all(x is not None for x in (gap, errors, stale)) and verdict(drill) is not None,
    }
    return record


def _cell(value):
    if value is None:
        return "not computed"
    if value is True:
        return "true"
    if value is False:
        return "false"
    return str(value)


def format_record(record):
    lines = []
    lines.append("DRILL RECORD — %s / %s / %s -> %s" % (
        record["group"], record["mode"], record["oldSite"], record["newSite"]))
    rows = [
        ("namespace", record["namespace"]),
        ("trigger", record["trigger"]),
        ("promotedAt", record["promotedAt"]),
        ("lastWriteOldSite", record["lastWriteOldSite"]),
        ("firstWriteNewSite", record["firstWriteNewSite"]),
        ("writeGapSeconds", "UNCLOSED" if record["gapClosed"] is False else record["writeGapSeconds"]),
        ("gapClosed", record["gapClosed"]),
        ("staleReads", record["staleReads"]),
        ("staleReadSeconds", record["staleReadSeconds"]),
    ]
    errors = record["errors"]
    if errors is None:
        rows.append(("errors", None))
    else:
        rows.append(("errors", "readOnlyRefusal=%d connection=%d other=%d" % (
            errors.get("readOnlyRefusal", 0), errors.get("connection", 0), errors.get("other", 0))))
    rows.append(("verdict", record["verdict"]))
    rows.append(("samples", "%d (writes %d, reads %d)" % (
        record["samples"]["total"], record["samples"]["writes"], record["samples"]["reads"])))
    if "baselineGapSeconds" in record:
        rows.append(("baselineGapSeconds", record["baselineGapSeconds"]))
        rows.append(("gapDeltaSeconds", record["gapDeltaSeconds"]))
    for key, value in rows:
        lines.append("  %-20s %s" % (key, _cell(value)))
    return "\n".join(lines)


def main(argv=None):
    parser = argparse.ArgumentParser(description="turn a failover drill capture into a drill record")
    parser.add_argument("--probe", required=True, help="JSONL probe log from the writer")
    parser.add_argument("--drill", required=True, help="JSON drill capture (trigger + group status)")
    parser.add_argument("--baseline", help="probe log from an earlier run to compare against")
    parser.add_argument("--baseline-drill", help="drill capture belonging to --baseline")
    parser.add_argument("--json", action="store_true", help="emit the record as JSON")
    args = parser.parse_args(argv)

    record = build_record(load_probe(args.probe), load_drill(args.drill))

    if args.baseline or args.baseline_drill:
        if not (args.baseline and args.baseline_drill):
            raise SystemExit("--baseline and --baseline-drill must be given together")
        base_gap = write_gap(load_probe(args.baseline), load_drill(args.baseline_drill))
        base_seconds = base_gap["gapSeconds"] if base_gap else None
        record["baselineGapSeconds"] = base_seconds
        if base_seconds is None or record["writeGapSeconds"] is None:
            record["gapDeltaSeconds"] = None
        else:
            record["gapDeltaSeconds"] = round(record["writeGapSeconds"] - base_seconds, 3)

    if args.json:
        print(json.dumps({k: v for k, v in record.items() if k != "_complete"}, indent=2))
    else:
        print(format_record(record))

    if not record["_complete"]:
        return 1
    if record["gapClosed"] is False:
        return 2
    return 0


if __name__ == "__main__":
    sys.exit(main())

drill-record.md

                    # Failover drill record — `playground`

Fill this in from `brdrill` output. Keep the drills separate: they are different
events making different claims. Replace every `TODO`.

## Drills

| Drill | Trigger | Moved | Capture |
| --- | --- | --- | --- |
| emergency | TODO | TODO -> TODO | `tests/fixtures/emergency-probe.jsonl` |
| planned | TODO | TODO -> TODO | `tests/fixtures/planned-probe.jsonl` |
| planned, baseline (unfixed pool) | TODO | TODO -> TODO | `tests/fixtures/planned-probe-unbounded.jsonl` |

## Measured

Only numbers these captures produced. Paste the `brdrill` lines.

- emergency: `writeGapSeconds` TODO
- planned: `writeGapSeconds` TODO
- planned, baseline: `writeGapSeconds` TODO, `gapDeltaSeconds` TODO
- stale reads, baseline: TODO reads over TODO s, first at TODO
- error classes, planned: TODO

## Assumed

Everything carried over rather than observed here — the RPO verdicts, and any
claim about what happens on a capture you did not take.

- verdict (emergency): TODO
- verdict (planned): TODO
- what this drill did **not** prove: TODO

## Attribution

Bloodraven's measured emergency promotion is 12.0 s to the `activeSite` flip
(detection is `pollInterval` × `failureThreshold` = 2 s × 3 = 6 s of it). Subtract
it from your emergency write-gap and the remainder belongs to your writer.

Not Bloodraven's: TODO s

## What would change the number

TODO — name the pool setting, the drain budget, or the strategy, and say which
direction it moves the gap.

Projects are not auto-graded here. The rubric and the test cases above are the grading contract — run them yourself on your own machine.

Erase saved progress?

This erases all quiz scores, reading progress, project checklists, and your name on the certificate. It cannot be undone, and it affects only this course in this browser.