Kubernetes
Bloodraven in Production
Stop treating the loss of a database site as an incident. You will hold a site down and watch Bloodraven promote another one in about 12 seconds, move the primary on purpose at an RPO of exactly zero, read the precise count of transactions an emergency failover cost you off the cluster itself, and roll a MySQL upgrade underneath live traffic without an incident.
- 7 units
- 27 topics
- 34 quizzes and tests
0%
0 / 27
Topics read
0
Quizzes taken
—
Quiz average
—
Unit-test average
About this course
Most people meet a failover operator and assume the hard part is the promotion. It is not. Promotion is nine SQL statements and takes about twelve seconds — that number is measured, not estimated, across nine recorded runs against a real cluster. The hard part is everything around it: knowing whether the operator will act at all, proving what the failover cost you, and discovering that a textbook-perfect promotion leaves your application reading yesterday's data on one code path and hard-failing on the other, with nothing anywhere paging anyone.
You start by standing up a real three-site failover group on your laptop with an application reading and writing through it, and learning to read its status well enough to name the active site. Then you learn the decision loop — a two-second poll and a small table — until you can predict the operator's next move from a status dump alone. Unit 3 makes it real: you hold a site down, time the promotion, and audit the exact GTID set you lost. Unit 4 is where the database meets the application, and where the wall you hit at the end of Unit 3 gets explained. Unit 5 removes the assumption that anyone can see the truth: sidecars that fence themselves, split brain, five shapes of network partition, and the operator itself going away. Unit 6 adds backups, point-in-time recovery, encryption, alerting and a disaster-recovery drill, and finishes with a go-live checklist you would actually sign. Unit 7 is the two days the rest of the course skips: building a group from nothing, the TLS the CRD makes you have, and upgrading MySQL and the operator underneath live traffic.
You need Docker, kubectl, helm, and a machine that can run a three-worker k3d cluster — about two minutes of setup, no cloud account, no DNS provider. You do not need to know MySQL replication internals, though you will finish able to argue about GTID sets, super_read_only and relay logs with anyone who does. Every number in this course traces to a row on the sources page: operator source code at v1.0.0, the shipped CRDs, recorded chaos-run forensics, and the MySQL 9.7 manual. Where the official documentation disagrees with the code, this course teaches the code and shows you the discrepancy.
Where you start, and where you finish
Before: A primary is down: page a human, guess at the RPO, hope the application reconnects.
After: Unattended promotion in about 12 seconds, a planned switchover at RPO 0 by construction, and an exact lost-transaction count read from divergentGtid.
What you'll be able to do
Predict the operator
Read a failover group's status and say what Bloodraven will do next, and why — before it does it.
Audit a failover
Time a real promotion and prove what it cost in transactions using promotionGtidExecuted and divergentGtid.
Explain every fence
Look at a read-only site and say which of the sidecar's rules put it there, and whether that was correct.
Connect an app that survives
Wire an application through the right Service with a pool that recovers from a promotion instead of serving stale reads.
Restore under pressure
Verify a backup, restore from it, and bootstrap a disaster-recovery group in a second cluster.
Run it on call
Build the alert set, map each alert to a first command, and hold a go-live gate that catches what the defaults miss.
Build one, and keep it alive
Provision a failover group into an empty namespace, wire the TLS the CRD demands, and roll MySQL and the operator underneath live traffic without turning a Tuesday into an incident.
Syllabus
Unit 1 — Meet the group
You are on call for a MySQL that has to survive losing an entire site. This unit puts a real three-site group on your laptop, then names the parts you can already see, then explains the bet Bloodraven is making. By the end you can point at any pod and say what role it holds and what it is allowed to become. Then the obvious question: who decides that, and how fast?
Unit 2 — How the operator decides
Under the failover machinery there is a two-second poll loop and a small table. The loop turns each site into one of four states with a debounce in front of it; the table turns the set of states into exactly one action. This unit teaches you to run that table in your head, then read the same decision back out of the logs and the metrics so you are never guessing what the operator is about to do. Knowing the table is not the same as knowing the timing — Unit 3 holds a site down and puts a clock on it.
Unit 3 — Emergency failover, end to end
You hold a site down and watch the whole sequence run — fence, kill, drain, promote, confirm, record, flip. You time it against a stopwatch rather than against the documentation, then prove exactly what the outage cost you in transactions, and decide what to do with the old primary when it comes back carrying writes nobody else has. It is a textbook-perfect promotion: pdx is writable in about twelve seconds and the counter application carries on as if nothing happened. That is the problem. It is still reading from iad, the site you just demoted.
Unit 4 — Where failover meets your application
Bloodraven's job ends at a label selector and a DNS record; your application's job starts there. This unit closes the gap Unit 3 opened — the promotion that worked perfectly while the counter app went on serving stale reads from the demoted site and failing every write against it, with nothing paging for either. You will fix a connection pool, move a primary on purpose at an RPO of zero, and decide what your cache and sessions owe you. Everything so far has assumed somebody can see the truth about playground; Unit 5 removes that assumption.
Unit 5 — When the world misbehaves
Everything so far assumed the operator can see the truth. This unit removes that assumption: sidecars fence themselves when they cannot confirm they are the active site, split brain gets a tiered response driven by sitePriorities, five documented partition shapes get five different answers, and the operator itself can be gone while playground carries on serving. You will learn which of those the system handles for you and which ones hand you a decision at 3am. None of it protects you from losing the whole cluster, the bucket, or last Tuesday — that is Unit 6.
Unit 6 — Backups, disaster recovery, and going live
A failover group protects you from losing a site. It does not protect you from losing the cluster, the bucket, or last Tuesday. This unit adds backups, PITR, verification, restore and encryption at rest, then the alerting and runbooks that make playground operable at 3am, and finishes with a go-live gate you would actually sign. By the end you can hand playground to an on-call rotation and state precisely what it will and will not do for them.
Unit 7 — Day 0 and day 2
Six units started from a group somebody else's script created. This one builds one from an empty namespace — credentials, storage, node labels, and the certificate the CRD made mandatory back in Unit 6 — and then keeps it alive: rolling MySQL and the operator underneath live traffic without turning a Tuesday into an incident. It closes with the one page you actually keep: every default, reason string, promotion step and metric label set on a single screen, beside a dated appendix for the facts that will not stay true.
Questions
What does this give me that the Bloodraven documentation doesn't?
The docs are the best reference in this field, and that is the honest comparison: their failure-mode matrix is the only one anywhere with all four columns — failure, signal, action, time to act, and what the operator will not do. Oracle's MySQL operator manual has no failover chapter at all. Vitess's reparenting page never mentions data loss. What reference material cannot do is make you produce the failure. The docs tell you the RPO; they never have you commit writes, kill the primary mid-flight, and count what did not survive. Their own divergent-recovery runbook opens with "investigate the lost transactions" and gives you no command that touches MySQL — and the next step runs CLONE INSTANCE, which destroys the evidence. This course fills in that step. It also teaches from the source code rather than the pages, which is how it found roughly a dozen places where the two disagree; you will be shown each one, because the day you need this knowledge is the day a stale page costs you an hour.
Can my organisation actually use Bloodraven? What is it licensed under?
Settle this before you plan a dependency, because it is not a technical question — and check it in the repository rather than here, because it is the fastest-moving fact in the course. The short version: Bloodraven is source-available, not open source. The full source is public and you may read, build and modify it; running it in production at a commercial company is a licensing question with a real answer. v1.0.0 ships the Business Source License 1.1: source-available, not OSI open source. Production use by a company over $1M annual revenue needs a commercial license; everything else in the additional-use grant stays free. Because terms can still move, they are recorded once — with the two commands that settle them (ls LICENSE*, and gh repo view ShipStream/bloodraven --json licenseInfo) — in section D of the version appendix on the sources page, rather than repeated through the units. This course is separately licensed; the notice is in every page footer.
Do I need a cloud account or a production cluster?
No. Everything runs on a local k3d cluster with three worker nodes — about two minutes of setup with Docker, kubectl and helm. You get a real three-site MySQL failover group, a live dashboard, an application writing through the primary Service, and a simulated external-dns pipeline. No cloud account, no DNS provider, no production infrastructure.
How much MySQL replication do I need to know already?
Enough to know that replicas follow primaries. You do not need to have configured GTID replication, and you will not be asked to memorise CRD fields. You will finish able to read a GTID set, explain why super_read_only is the fence and read_only is not, say what a relay log is and why draining one bounds a promotion, and use GTID_SUBTRACT to compute exactly what a failover lost.
Will this teach me every configuration field?
Deliberately not. Field shapes are what the CRD reference is for, and memorising them is the fastest way to learn nothing durable. This course teaches which class of knob to reach for and what it actually controls — including the several cases where a field does not mean what its name implies. maxLagSeconds is an alerting threshold and not a promotion gate. connectionDrainTimeout bounds how long the operator keeps trying, not how long your pool holds a socket. Knowing that is worth more than knowing forty defaults.
Is this a course about writing Kubernetes operators?
No. It is a course about running this one. There is no controller-runtime, no reconciler-writing, and no Go beyond reading the occasional line to settle an argument about what actually happens. The audience is the person who gets paged, not the person who ships the operator.
We already rehearse failover, and our chaos suite is green. What is left?
Two things, and they are the same thing seen twice. First, a green suite proves the happy path: Bloodraven ships 51 real-cluster chaos scenarios, two of which touch data integrity, and both assert that nothing was lost. That is the correct thing for CI to assert and it means CI has never shown you the sad path. Second, almost every rehearsal in this field — here and everywhere else — injects a process kill: delete the pod, kill the sandbox instance, power off the node. That is precisely the failure mode where fencing works best, because the node stops and everyone agrees it stopped. GitHub ran a deliberate promotion in February 2020 specifically to give their teams visibility, and it recreated the outage, because a silently clamped file-descriptor limit had never been exercised under load. Rehearsing the happy path is not rehearsing the failover.
Can Bloodraven give me zero data loss?
On a planned switchover, yes, by construction — the target is only promoted once its GTID set provably contains the fenced source's, so there is nothing left to lose. On an unplanned failover, no, and no asynchronous replication system can. What Bloodraven does instead is measure the loss exactly and hand you the GTID set and a transaction count. This course teaches you to read that number rather than to hope for zero, which is the difference between an RPO you can state in a meeting and one you are guessing at.
How is my progress tracked, and who grades the quizzes?
You do, and only this browser knows. The whole site is static: there is no account, no server, and nothing leaves the machine you are reading on. Progress, quiz scores, flashcard state and the certificate all live in a single key in this browser's own local storage — clear the site data, switch browsers or open it in a private window and you start from zero, with no way to recover it. Multiple-choice and true/false questions are marked automatically against the answer key shipped in the page. Short-answer questions are self-graded: you write an answer, then reveal a sample answer and a note saying what a full-credit response has to cover, and you decide. The certificate is a self-reported record of that self-assessment, verified by nobody. Treat it as a study aid, not a credential — and if you want an audit trail, print or export it, because nothing else is keeping one.
How much programming is this, and do I have to do every project?
Less than the project list suggests, and no. Four of the seven projects are the spine and are worth doing in order: brstatus (Unit 1) because status literacy is what every later unit reads, the failover drill (Unit 4) because your own application's write-gap is the only recovery number that means anything, the go-live pack (Unit 6) because it is the artefact you hand a rotation, and brprep (Unit 7), which is bash and jq with no Python at all. The other three — brdecide, the post-failover audit, and fencing forensics — are marked optional in their briefs: each is a deeper drill on a mechanism the reading already covers, and each brief opens by saying what you give up by skipping it. Two of them also ship a shorter route: brstatus and the audit report both have a 'without Python' section that does the same job with kubectl, jq and one MySQL function, which is closer to what you would actually type during an incident anyway. The Python that remains is standard-library only — no frameworks, no cluster at grading time, and every project runs against JSON fixtures so nothing depends on your laptop.