A raven-headed operator in a navy henley sits at a sunlit wooden desk typing on a laptop, with an empty pale wall on the left and a code editor on the monitor behind.

Kubernetes

Bloodraven in Production

Stop treating the loss of a database site as an incident. You will hold a site down and watch Bloodraven promote another one in about 12 seconds, move the primary on purpose at an RPO of exactly zero, read the precise count of transactions an emergency failover cost you off the cluster itself, and roll a MySQL upgrade underneath live traffic without an incident.

  • 7 units
  • 27 topics
  • 34 quizzes and tests

0%

0 / 27

Topics read

0

Quizzes taken

Quiz average

Unit-test average

About this course

Most people meet a failover operator and assume the hard part is the promotion. It is not. Promotion is nine SQL statements and takes about twelve seconds — that number is measured, not estimated, across nine recorded runs against a real cluster. The hard part is everything around it: knowing whether the operator will act at all, proving what the failover cost you, and discovering that a textbook-perfect promotion leaves your application reading yesterday's data on one code path and hard-failing on the other, with nothing anywhere paging anyone.

You start by standing up a real three-site failover group on your laptop with an application reading and writing through it, and learning to read its status well enough to name the active site. Then you learn the decision loop — a two-second poll and a small table — until you can predict the operator's next move from a status dump alone. Unit 3 makes it real: you hold a site down, time the promotion, and audit the exact GTID set you lost. Unit 4 is where the database meets the application, and where the wall you hit at the end of Unit 3 gets explained. Unit 5 removes the assumption that anyone can see the truth: sidecars that fence themselves, split brain, five shapes of network partition, and the operator itself going away. Unit 6 adds backups, point-in-time recovery, encryption, alerting and a disaster-recovery drill, and finishes with a go-live checklist you would actually sign. Unit 7 is the two days the rest of the course skips: building a group from nothing, the TLS the CRD makes you have, and upgrading MySQL and the operator underneath live traffic.

You need Docker, kubectl, helm, and a machine that can run a three-worker k3d cluster — about two minutes of setup, no cloud account, no DNS provider. You do not need to know MySQL replication internals, though you will finish able to argue about GTID sets, super_read_only and relay logs with anyone who does. Every number in this course traces to a row on the sources page: operator source code at v1.0.0, the shipped CRDs, recorded chaos-run forensics, and the MySQL 9.7 manual. Where the official documentation disagrees with the code, this course teaches the code and shows you the discrepancy.

Where you start, and where you finish

Before: A primary is down: page a human, guess at the RPO, hope the application reconnects.

After: Unattended promotion in about 12 seconds, a planned switchover at RPO 0 by construction, and an exact lost-transaction count read from divergentGtid.

What you'll be able to do

Syllabus

Questions

What does this give me that the Bloodraven documentation doesn't?

The docs are the best reference in this field, and that is the honest comparison: their failure-mode matrix is the only one anywhere with all four columns — failure, signal, action, time to act, and what the operator will not do. Oracle's MySQL operator manual has no failover chapter at all. Vitess's reparenting page never mentions data loss. What reference material cannot do is make you produce the failure. The docs tell you the RPO; they never have you commit writes, kill the primary mid-flight, and count what did not survive. Their own divergent-recovery runbook opens with "investigate the lost transactions" and gives you no command that touches MySQL — and the next step runs CLONE INSTANCE, which destroys the evidence. This course fills in that step. It also teaches from the source code rather than the pages, which is how it found roughly a dozen places where the two disagree; you will be shown each one, because the day you need this knowledge is the day a stale page costs you an hour.

Can my organisation actually use Bloodraven? What is it licensed under?

Settle this before you plan a dependency, because it is not a technical question — and check it in the repository rather than here, because it is the fastest-moving fact in the course. The short version: Bloodraven is source-available, not open source. The full source is public and you may read, build and modify it; running it in production at a commercial company is a licensing question with a real answer. v1.0.0 ships the Business Source License 1.1: source-available, not OSI open source. Production use by a company over $1M annual revenue needs a commercial license; everything else in the additional-use grant stays free. Because terms can still move, they are recorded once — with the two commands that settle them (ls LICENSE*, and gh repo view ShipStream/bloodraven --json licenseInfo) — in section D of the version appendix on the sources page, rather than repeated through the units. This course is separately licensed; the notice is in every page footer.

Do I need a cloud account or a production cluster?

No. Everything runs on a local k3d cluster with three worker nodes — about two minutes of setup with Docker, kubectl and helm. You get a real three-site MySQL failover group, a live dashboard, an application writing through the primary Service, and a simulated external-dns pipeline. No cloud account, no DNS provider, no production infrastructure.

How much MySQL replication do I need to know already?

Enough to know that replicas follow primaries. You do not need to have configured GTID replication, and you will not be asked to memorise CRD fields. You will finish able to read a GTID set, explain why super_read_only is the fence and read_only is not, say what a relay log is and why draining one bounds a promotion, and use GTID_SUBTRACT to compute exactly what a failover lost.

Will this teach me every configuration field?

Deliberately not. Field shapes are what the CRD reference is for, and memorising them is the fastest way to learn nothing durable. This course teaches which class of knob to reach for and what it actually controls — including the several cases where a field does not mean what its name implies. maxLagSeconds is an alerting threshold and not a promotion gate. connectionDrainTimeout bounds how long the operator keeps trying, not how long your pool holds a socket. Knowing that is worth more than knowing forty defaults.

Is this a course about writing Kubernetes operators?

No. It is a course about running this one. There is no controller-runtime, no reconciler-writing, and no Go beyond reading the occasional line to settle an argument about what actually happens. The audience is the person who gets paged, not the person who ships the operator.

We already rehearse failover, and our chaos suite is green. What is left?

Two things, and they are the same thing seen twice. First, a green suite proves the happy path: Bloodraven ships 51 real-cluster chaos scenarios, two of which touch data integrity, and both assert that nothing was lost. That is the correct thing for CI to assert and it means CI has never shown you the sad path. Second, almost every rehearsal in this field — here and everywhere else — injects a process kill: delete the pod, kill the sandbox instance, power off the node. That is precisely the failure mode where fencing works best, because the node stops and everyone agrees it stopped. GitHub ran a deliberate promotion in February 2020 specifically to give their teams visibility, and it recreated the outage, because a silently clamped file-descriptor limit had never been exercised under load. Rehearsing the happy path is not rehearsing the failover.

Can Bloodraven give me zero data loss?

On a planned switchover, yes, by construction — the target is only promoted once its GTID set provably contains the fenced source's, so there is nothing left to lose. On an unplanned failover, no, and no asynchronous replication system can. What Bloodraven does instead is measure the loss exactly and hand you the GTID set and a transaction count. This course teaches you to read that number rather than to hope for zero, which is the difference between an RPO you can state in a meeting and one you are guessing at.

How is my progress tracked, and who grades the quizzes?

You do, and only this browser knows. The whole site is static: there is no account, no server, and nothing leaves the machine you are reading on. Progress, quiz scores, flashcard state and the certificate all live in a single key in this browser's own local storage — clear the site data, switch browsers or open it in a private window and you start from zero, with no way to recover it. Multiple-choice and true/false questions are marked automatically against the answer key shipped in the page. Short-answer questions are self-graded: you write an answer, then reveal a sample answer and a note saying what a full-credit response has to cover, and you decide. The certificate is a self-reported record of that self-assessment, verified by nobody. Treat it as a study aid, not a credential — and if you want an audit trail, print or export it, because nothing else is keeping one.

How much programming is this, and do I have to do every project?

Less than the project list suggests, and no. Four of the seven projects are the spine and are worth doing in order: brstatus (Unit 1) because status literacy is what every later unit reads, the failover drill (Unit 4) because your own application's write-gap is the only recovery number that means anything, the go-live pack (Unit 6) because it is the artefact you hand a rotation, and brprep (Unit 7), which is bash and jq with no Python at all. The other three — brdecide, the post-failover audit, and fencing forensics — are marked optional in their briefs: each is a deeper drill on a mechanism the reading already covers, and each brief opens by saying what you give up by skipping it. Two of them also ship a shorter route: brstatus and the audit report both have a 'without Python' section that does the same job with kubectl, jq and one MySQL function, which is closer to what you would actually type during an incident anyway. The Python that remains is standard-library only — no frameworks, no cluster at grading time, and every project runs against JSON fixtures so nothing depends on your laptop.

Erase saved progress?

This erases all quiz scores, reading progress, project checklists, and your name on the certificate. It cannot be undone, and it affects only this course in this browser.