Security model

This page documents Bloodraven's security posture: the trust boundaries between its components, the authorizations each component holds, and the blast radius when a specific secret or identity is compromised. It complements Production hardening, which lists the settings to turn on; this page explains why they matter and what remains exposed when they are all correct.
Bloodraven is an in-cluster control plane for a tenant MySQL service. It is not a zero-trust system on its own. It relies on the hosting Kubernetes cluster for authentication, namespace isolation, and network policy. The guarantees below all assume a competent cluster operator is running it.
Security quick wins
- Use
spec.credentialswith five distinct role Secrets. - Enable
spec.tlsand require TLS in MySQL clients. - Keep the auxiliary Service reachable by sidecars for self-fencing and PITR retention; restrict ingress rather than disabling it without a replacement route.
- Add NetworkPolicy for operator, sidecar, MySQL, Prometheus, app, and backup paths.
- Pin and mirror operator, sidecar, MySQL, and backup images.
- If Dragonfly is enabled, protect its password Secret, active Service, and admin port with the same care as the MySQL data plane.
- Link Secret rotation to Credentials And TLS.
Threat-model assumptions
| Assumption | Consequence |
|---|---|
| Kubernetes API authn/authz is trusted | RBAC boundaries are meaningful. |
| CNI enforces NetworkPolicy correctly | Unauthenticated sidecar and auxiliary endpoints can be isolated. |
| Secret storage is protected | MySQL, Dragonfly, TLS, S3, and encryption secrets remain confidential. |
| Node administrators are trusted | A node admin can read mounted Secrets and MySQL data on that node. |
| Application workloads are less trusted than the operator | Apps should only receive app or read-only MySQL credentials. |
Trust model at a glance
Four identities participate in the required MySQL data path. Managed Dragonfly adds an optional cache/session data path when enabled. Everything else in the cluster is considered untrusted.
| Identity | What it is | Trust level |
|---|---|---|
| Operator | Single-replica Deployment running bloodraven, bound to a cluster-wide ServiceAccount. Owns the reconcile loop and DNS writes. | Fully trusted — it is the control plane. |
| Sidecar | Per-MySQL-pod container running bloodraven-sidecar. Serves read-only status HTTP, manages MySQL startup, writes binlog manifests, and self-fences its MySQL. | Trusted for the single pod it runs in. |
| MySQL | mysqld itself, with the per-role users Bloodraven creates (operator, app, readonly, monitor, backup). | Trusted for query execution; each MySQL user is trusted only for its role's grants. |
| Dragonfly | Optional per-site Dragonfly pods created when spec.dragonfly.enabled=true. The operator configures replication, role/traffic labels, and promotion. | Trusted for cache/session data only. Do not store durable application state here. |
| Application | Workloads that connect to the primary/replica Services using appSecretName / readOnlySecretName. | Untrusted by the operator — apps are tenants. |
All intra-cluster traffic between these identities flows over the pod network. Bloodraven does not authenticate HTTP calls between the operator and the sidecars, and it does not mTLS-protect the sidecar HTTP endpoint. Isolation is the hosting cluster's job (NetworkPolicy, namespace boundaries, CNI-level encryption if you require it).
Authorization — who can do what
Operator ServiceAccount
The operator runs with a cluster-scoped ClusterRole
(config/rbac/role.yaml, mirrored in
charts/bloodraven/templates/clusterrole.yaml). It holds the minimum
set of verbs to reconcile MysqlFailoverGroup and MysqlBackup
resources across any namespace. Notably:
- Full CRUD on
Secrets,ConfigMaps,Services,PersistentVolumeClaims,Jobs,CronJobs,Deployments,PodDisruptionBudgets— cluster-wide. The operator reads credential Secrets, renders MySQL config into ConfigMaps, and creates PVCs, Services, Pods, and Jobs per site. - Full CRUD on
dnsendpoints.externaldns.k8s.io— the failover path writesDNSEndpointCRs thatexternal-dnsthen pushes to the DNS provider. get/list/watch/patchonpodsandnodes, plusgetonpods/log— used for the tainter, the backup-sentinel log tail, and placement contract enforcement.- Full CRUD on
leases.coordination.k8s.io— leader election and durable deployment coordination. createontokenreviews.authentication.k8s.io— validates projected ServiceAccount tokens for the optional deployment API.create/get/update/patchonserviceaccounts,rolesandrolebindings— each group with backup or verification schedules gets amysql-<group>-schedule-triggerServiceAccount, bound to a Role that can only createMysqlBackupandMysqlBackupVerificationresources and read that group. The schedule CronJob pods run under it. Kubernetes only lets the operator grant verbs it already holds.
There is no namespaced partitioning. A single operator install
reconciles groups in every namespace. If you need tenant isolation
(one operator per tenant), deploy one operator per namespace scope and
use a Role instead — the code supports this but the shipped chart
does not.
MySQL user roles
When spec.credentials is used, the operator creates five MySQL users
with the following effective grants (see
CRD Reference → CredentialsSpec):
| Role | Effective authority |
|---|---|
operator | ALL PRIVILEGES ON *.* WITH GRANT OPTION — full admin. Used by the operator, the sidecar and restore Jobs (which drop and load schemas, and must keep working under a full-instance restore's offline_mode fence). |
app | DML, DDL, views, routines, triggers, events, temporary tables, LOCK TABLES and SHOW DATABASES on *.*, with every write and DDL privilege partially revoked on the mysql system schema — no administrative privileges (SUPER, CREATE USER, PROCESS, replication, ...) and no GRANT OPTION. Read/write for tenant workloads. |
readonly | SELECT, SHOW VIEW, SHOW DATABASES, PROCESS — read replicas. |
monitor | PROCESS, REPLICATION CLIENT + SELECT on performance_schema.* — mysqld_exporter and similar scrapers. |
backup | SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER, RELOAD, BACKUP_ADMIN, REPLICATION CLIENT — MySQL Shell dump jobs. |
Only operatorSecretName is required. A role you leave unset has no user, so
give each workload its own role Secret instead of the operator's: the
operator account is full admin.
Destructive annotations on the CR
patch on mysqlfailovergroups is the narrow privilege that controls
every operator-initiated destructive action. The annotations the
operator acts on are:
bloodraven.shipstream.io/reclone-site=<site>:confirm=<group>:uid=<uid>[:gtidPrefix=<prefix>]— TriggersCLONE INSTANCEon the named site, overwriting its data directory from the current primary. The group name and uid keep a request from applying to another group or a re-created one of the same name, and on a diverged site the GTID-prefix confirmation (≥ 8 chars of the source UUID ofstatus.sites[<site>].divergentGtid) prevents typos from wiping the wrong site. See Failover → Reclone flow.bloodraven.shipstream.io/planned-failover=<site>— Drains writes, honours the anti-flap cooldown, waits for the target to catch up, and promotes the named site. When Dragonfly is enabled, this also drives the Dragonfly sync andREPLTAKEOVERphases.bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>— Acknowledges an automation hold after a planned failover endedManualRecoveryRequired, which resumes automatic failover, split-brain resolution and DNS reconciliation for the group. Grant it with the same care as the planned-failover annotation, and only after the split brain is resolved.bloodraven.shipstream.io/revoke-deployment-leases- Revokes the group's deployment leases with reasonOperatorRevoked. This can unblock planned disruption and fences active deployment clients. See deployment coordination operations.bloodraven.shipstream.io/release-pitr-archive-binding=<id>:confirm=<group>:uid=<uid>— Releases one PITR archive encryption binding and deletes its key snapshot Secrets, which may be the cluster's only copy of an archive's passphrase. Refused while the binding is in use; never commit it in GitOps. See Releasing a binding.bloodraven.shipstream.io/last-failoverandbloodraven.shipstream.io/last-failover-target— Operator-owned durable anti-flap history. A writer can extend or clear the effective cooldown and change the remembered fencing target. The operator rejects malformed, unpaired, unknown-site, and implausibly future-dated values, but these keys must still be protected from manual or third-party mutation.bloodraven.shipstream.io/promotion-gtid-executedandbloodraven.shipstream.io/promotion-gtid-executed-for— Operator-owned out-of-band copy of the promotion watermark and the event it belongs to. The watermark gates which candidates may be promoted and how a returning old primary is judged, so a writer can weaken the promotion-data-regression check. The operator ignores a watermark whose binding does not name the eventlast-failoverandlast-failover-targetrecord.bloodraven.shipstream.io/planned-failover-terminal— Operator-owned terminal marker of a planned failover whose status writes stalled. With outcomeManualRecoveryRequiredit is an automation-hold record on its own: deleting it (together withstatus.plannedFailoverHoldand theAutomaticFailoverHeldcondition, which needmysqlfailovergroups/statusrights) resumes automatic failover while a split brain may stand, and writing one holds automation or ends a live run. Treat write access to it like the acknowledgement annotation.bloodraven.shipstream.io/planned-failover-stall-since,bloodraven.shipstream.io/planned-failover-manager-absent-since,bloodraven.shipstream.io/planned-failover-confirmedandbloodraven.shipstream.io/planned-failover-status-fence— Operator-owned bookkeeping of a run in flight (deadline starts, the confirmation time, the status-write fence). A writer can shorten or extend a run's deadlines, or make an ambiguous status write look settled. The operator accepts a deadline start only for the run it names.
All of these keys are listed in Well-known metadata.
Backing up and restoring a MysqlFailoverGroup
A restore re-creates the object without its status (the API server never
takes status on create), so a run in flight is not restored and its keys
would name nothing. When a backup tool (Velero, a GitOps export) restores the
object:
- Drop the request and acknowledgement annotations
(
planned-failover,acknowledge-planned-failover-hold): a restored request starts a planned failover, and a restored acknowledgement could clear a hold nobody reviewed. - Drop
planned-failover-stall-since,planned-failover-manager-absent-since,planned-failover-confirmedandplanned-failover-status-fence: they belong to the run whose status was not restored. - Keep a
planned-failover-terminalmarker whose outcome isManualRecoveryRequired: it is the only hold record left, and the restored group stays held until the split brain is checked and the hold is acknowledged withkubectl bloodraven ack-planned-failover-hold. Drop a marker with any other outcome. - Restore
last-failover,last-failover-target,promotion-gtid-executedandpromotion-gtid-executed-fortogether or not at all. Restoring them keeps the anti-flap cooldown and the watermark; dropping them makes the group look as if it never failed over, which shortens the cooldown and removes the watermark check until the next promotion.
An attacker with patch mysqlfailovergroups in the target namespace
can therefore cause a data wipe of one site (bounded by the GTID
confirmation interlock), an unscheduled failover, or a planned Dragonfly
cache/session outage during snapshot-restore upgrade. RBAC on the CRD
is the only gate — scope MysqlFailoverGroup edit rights as tightly
as you would scope kubectl delete pvc.
Write access to spec.sites[].taintNodeSelector reaches beyond the
namespace. The operator applies a NoExecute taint to every node in the
cluster that matches the selector, and the selector can name any node
labels. Anyone who can create or edit a MysqlFailoverGroup can therefore
evict untolerating pods from any node. Grant that right only to principals
you would trust to taint nodes.
Network surface
| Listener | Port | Authenticated? | Purpose |
|---|---|---|---|
| Operator metrics | :8080 | No | Prometheus scrape (/metrics). |
| Operator health probes | :8081 | No | Kubernetes liveness/readiness. |
| Operator auxiliary HTTP | :8082 | No | /status, /active-site, /pitr-cutoff, /ws/status (WebSocket). |
| Operator keyring escrow HTTPS | :8443 by default | Bearer token + TLS | /keyring/escrow; enabled with auxiliary.escrowTLS. |
| Operator deployment HTTPS | Same TLS :8443 listener | Projected ServiceAccount token + TokenReview + database allowlist | /deploy/v1; opt-in with auxiliary.deployAPI.enabled. |
| Sidecar HTTP | :8080 | No | /livez, /health, /status, /peer/ping, /peer/active-site, /archiver/status, /keyring/status. Consumed by the operator, the peer sidecar, and the kubelet (liveness on /peer/ping, readiness on /health). |
| MySQL | :3306 | Yes (password; TLS if spec.tls set) | Data plane. |
| Dragonfly client | :6379 by default | Password when spec.dragonfly.auth is set | Optional Redis-compatible cache/session data plane. |
| Dragonfly admin | :9999 by default | Password when spec.dragonfly.auth is set | Optional operator control surface for replication, promotion, and snapshot commands. |
Two HTTP API surfaces authenticate callers: keyring escrow and the
optional deployment API. The unauthenticated ones expose read-only status and GTID
coordinates and accept no state-changing requests. Sidecars do act on their
responses: they self-fence when /active-site or /peer/active-site names
another site as active, and the binlog archiver keeps binlogs back to the
/pitr-cutoff answer. The archiver asks with namespace, group,
archive (its BLOODRAVEN_PITR_ARCHIVE_ID) and coverage=gtid-v1; the
operator answers only for an archive ID derived for that group and the
destination of one of its archive bindings. Retention requires complete file GTID coverage in
every returned group backup bound, pauses new retirements during in-flight
backups, and retains the local 24-hour age floor. File proofs must match
verified content identities. The handler uses the informer cache, bounded
lookup admission, singleflight and safe deferral caching (AUDIT H2);
rate-limited replies carry no delete authority.
The coverage=gtid-v1 capability prevents older clients using a time-only
cutoff; it does not authenticate the caller or response. The response
exposes backup GTID bounds to allowed network callers. A sidecar ignores peer views while the operator answers and treats a peer
timestamp from its future as current, but a forged response can still fence a
site. That is why the NetworkPolicy assumption below matters. The design assumes the pod network is
protected by NetworkPolicy and that the operator + sidecars run in a
namespace that does not accept untrusted ingress. In particular:
- Do not expose
:8082outside the cluster. The/active-siteand/statusendpoints do not leak credentials but do leak topology, site names, zone labels, and GTID coordinates — useful reconnaissance for an attacker already inside the cluster. - The WebSocket push stream (
/ws/status) is likewise unauthenticated. Anyone that can reach port8082can read real-time failover events. Browser upgrades are limited to same-origin pages unlessauxiliary.service.wsAllowedOriginslists other origins; avoid setting it to"*", which lets any web page read the stream through a user's browser.auxiliary.service.wsMaxClients(default 100) caps concurrent stream clients, counting handshakes in progress, and rejects extra upgrades with 429.0removes the cap; avoid it. - If you require scrape auth, front the metrics endpoint with a
sidecar (e.g.
kube-rbac-proxy). The chart does not ship one. /keyring/statuson the sidecar reports digests and encryption coverage — never key material. It leaks the same class of reconnaissance as/status.
Keyring escrow: /keyring/escrow. This state-changing endpoint is isolated
on the TLS listener and authenticates independently of the deployment API.
Callers present a per-site bearer token that the operator mints into a
Secret mounted only into that site's pod, and only while that
site is deliberately unsealed. A sealed site carries no token at all.
The token Secret itself persists and is not rotated, so the operator also
rejects pushes for any site whose keyring is sealed (or escrowed and
about to seal); a token copied during an unseal window cannot be
replayed afterwards. Beyond the token: the body is size-capped, the claimed digest must match
a fresh hash of the payload, escrow Secrets are immutable (so an accepted
push can only add a version, never rewrite the one a running MySQL is
sealed against), and the reconcile loop independently re-reads the stored
Secret before it will seal anything. See
Data-at-rest encryption.
The sidecar validates the listener certificate against the CA mounted
from the failover group's spec.tls Secret. There is no plaintext
fallback and redirects are rejected so neither the bearer token nor the
keyring can be sent to another endpoint.
Deployment API authorization
/deploy/v1 is a second state-changing auth surface on the same TLS listener, not on plaintext auxiliary HTTP. It validates projected tokens with TokenReview using audience bloodraven-deploy by default. The authenticated namespace/ServiceAccount pair must appear in MysqlDatabase.spec.deploymentClients for the requested group. The database CR name defines the instance identity. Unauthorized requests return a non-disclosing 403; authentication failures return 401.
A deployment client can observe its authorized group, serialize migrations, and defer planned disruption while it holds a renewable lease. It cannot delay emergency failover or use the API to patch arbitrary resources. Restrict writes to deploymentClients: granting a caller access can let it repeatedly renew a hold and delay planned maintenance. Use a dedicated ServiceAccount rather than an application's general identity.
Lease tokens are random, hash-only in durable coordination Leases, and separate from ServiceAccount tokens. Repeating a same-operation POST rotates the lease token because the original cannot be recovered. Topology changes and explicit administrative revocation fence deployments; neither permits automatic continuation of uncertain DDL. Protect Lease writes and the revoke annotation as control-plane privileges.
The chart's optional deployment NetworkPolicy preserves escrow access for managed MySQL pods and leaves metrics, probes, and auxiliary HTTP reachable. It limits the shared TLS port, not URL paths, and does not replace authentication or a complete cluster network policy. See Deployment API for exact selectors and additional ingress rules.
TLS and data-in-flight
When spec.tls.secretName references a Secret with ca.crt,
tls.crt, and tls.key, the operator mounts it into every MySQL pod
and renders config that sets ssl-ca/ssl-cert/ssl-key andrequire-secure-transport=ON. Without TLS set, MySQL accepts
plaintext connections from anything on the pod network.
- Replication between sites uses the same MySQL port and the same
server certificate — cross-site replication is TLS-protected only
when
spec.tlsis set. Replication channels setSOURCE_SSL_CAto the mountedca.crt(MySQLVERIFY_CA): the replica checks that the source's certificate chains to that CA, but does not check the host name, because replicas dial the-internalper-site Service. Clone (CLONE INSTANCE ... REQUIRE SSL) is encrypted but does not verify the donor's certificate. Mirror a single issuer across sites, or ensure both sites' certs chain to a CA each other trusts. - The sidecar, operator, backup Jobs, and MySQL Shell dump workers
all trust the CA in
ca.crt. When you rotate the CA, rotate the mounted Secret; the operator detects the hash change and rolls pods. - Cert-manager integration is not automatic — Bloodraven expects the
Secret to exist. Use a cert-manager
Certificateresource to populate it.
Dragonfly traffic is not covered by spec.tls; configure NetworkPolicy
and, where required, Dragonfly-native TLS or a service mesh outside
Bloodraven's managed flags. Do not expose the Dragonfly admin port to
applications.
Secrets at rest and in memory
- Nothing is written to disk that wasn't already on disk. The
sidecar reads the operator Secret's
username/passwordfrom the read-only volume at/etc/mysql/creds/operator(MYSQL_CREDS_DIR), so that password is not in the sidecar's environment. It keeps the DSN in memory and does not write credentials to a file. - Logs never include passwords. DSNs are built for in-process use and are never logged; SQL statement errors are truncated before logging. Operator events and condition messages are safe to ship to a log aggregator.
- Secret content is copied into env and files as MySQL requires.
The MySQL container's environment holds
MYSQL_ROOT_PASSWORD; per-role passwords are mounted as files. The root password is only needed during first boot, but the operator keeps it in the pod spec for simplicity. Anything that cankubectl execinto the pod can read them. - Published hashes are keyed. The
bloodraven.shipstream.io/spec-hashannotation on each site's Deployment and pods, and thebloodraven.shipstream.io/credential-hashannotation on the group, change when a password or certificate changes so pods roll and grants re-apply. They are readable by anyone who can read those objects, so Secret contents enter them only as an HMAC under a random per-group key in themysql-<group>-digest-keySecret; without that key they cannot be used to check password guesses. Deleting the key Secret is safe: the operator creates a new one, and every site rolls once through the ordered update. - Backup passphrases are snapshotted. For an encrypted backup profile the
operator copies the trimmed passphrase into immutable Secrets it creates in
the group's namespace: one per accepted PITR archive key
(
mysql-<group>-pitr-key-<h10>, labelapp.kubernetes.io/name=mysql-pitr-key) and one per backup (mysql-dump-key). They are a second copy of the key behind the same Secret RBAC as your Secret, have no owner reference, and survive group deletion until released or deleted by label. The<h10>name suffix is an HMAC under the group's digest key over the destination and the bytes, so the name reveals nothing about the key. An archive snapshot'sbloodraven.shipstream.io/pitr-acceptedannotation carries the archive descriptor'skeyCheck, a passphrase verifier (offline brute-force material for a weak passphrase, like the descriptor itself); it lives only on the Secret, never in the group's status. Only the bytes pods consume enter the pod fingerprint (TLSca.crt/tls.crt/tls.key, roleusername/password,MYSQL_ROOT_PASSWORD); passphrase Secrets do not. See Key snapshots. - Dragonfly auth is optional but recommended for shared namespaces.
When
spec.dragonfly.authis set, the operator projects the password Secret into Dragonfly pods as an environment variable and uses it for its own Redis-protocol connections. - MySQL data at rest is plaintext unless you enable
spec.encryptionAtRest. With it enabled, tablespaces, redo, undo, binary and relay logs are encrypted and the keyring lives only on tmpfs — but the key then lives in a Kubernetes Secret, which means etcd becomes part of your key custody. Enabling the feature without API-server encryption at rest for Secrets moves the plaintext key from the MySQL disk to the control-plane disk rather than protecting it. See Data-at-rest encryption.
Compromise scenarios
The operator ServiceAccount token leaks
This is the worst-case credential compromise. With operator-level cluster access, an attacker can:
- Read every namespace's Secrets. Including MySQL credential
Secrets for every
MysqlFailoverGroupthe operator manages. Treat this as equivalent to leaking all per-tenant MySQL credentials and all TLS private keys referenced byspec.tls. - Create or patch arbitrary
Deployment,Pod,Job,CronJob,Service, andPVCobjects cluster-wide. Because the operator has full CRUD on these, the attacker has a general-purpose tenancy breakout through pod spec injection. - Write
DNSEndpointCRs that external-dns will push to the public DNS zone, enabling a DNS-hijack attack against any name external-dns manages (not just MySQL names). - Exfiltrate PVC contents by scheduling a pod that mounts the target PVC — the ServiceAccount can create pods and PVC mounts.
- Create or patch Dragonfly StatefulSets and Services when the feature is enabled, including changing cache/session routing labels.
Mitigations. Pin the chart's serviceAccountName, bind it to a
namespace-scoped Role where possible, run the operator in its own
namespace, and rely on the audit log to detect token reuse from
unexpected source IPs. Rotate the token (delete the ServiceAccount's
token Secret) on any suspected compromise — Kubernetes issues a new
projected token to the pod automatically.
This is the single largest blast radius in the product. A multi-operator, namespace-scoped deployment model is on the wishlist but not implemented today.
The MySQL operator / root password leaks
Inside the DB, operator and root are equivalent — both carry
ALL PRIVILEGES ... WITH GRANT OPTION. An attacker with either can:
- Read, modify, or drop any schema. Including running
SELECT ... INTO OUTFILEon any path MySQL can reach (bounded bysecure_file_priv, which Bloodraven sets to the data directory). - Create new MySQL users and grant them arbitrary privileges, surviving a password rotation of the known users.
- Run
STOP REPLICA/RESET REPLICA ALL/CHANGE REPLICATION SOURCE TO ...— directly breaking replication or repointing a replica at an attacker-controlled primary. - Trigger
SHUTDOWN(unlessSUPERis gated), rebooting the MySQL instance.
The blast radius is per-failover-group, not cluster-wide. The
operator will detect the resulting replication breakage and attempt
recovery or raise Degraded=True, but it cannot distinguish malicious
writes from legitimate ones — divergent GTIDs are the only signal that
reaches the status.sites[].divergentGtid field.
Mitigations. Store credential Secrets with a rotating provider
(ESO, sealed-secrets, Vault). Set alerts on
bloodraven_divergent_transactions > 0. Keep the operator's Secret
in the same namespace as the failover group so a namespace-compromise
does not cascade.
The MySQL app password leaks
Bounded by the app grants: read/write across every schema, but no
SUPER, no GRANT OPTION, no REPLICATION privileges. The
attacker cannot:
- Run
CREATE USER,GRANT, orSET GLOBAL, or write throughread_only(noSUPER/CONNECTION_ADMIN). - Break replication (lacks
REPLICATION CLIENT/REPLICATION SLAVE). - Run
KILLon other sessions (lacksPROCESSunless you deviate from the default). - Drop the MySQL server itself.
The attacker can run arbitrary SQL against the tenant data. This is
the same blast radius as an application SQL-injection — assume any
compromise of app yields full data read/write for the application's
schemas. It does not include writing the mysql system schema: the
operator runs MySQL with partial_revokes=ON and partially revokes every
write and DDL privilege there, so the attacker cannot edit the grant
tables to hand app the privileges above at the next privilege reload or
restart. The attacker can still read the mysql schema, including the
password hashes in mysql.user; treat a leaked app password as exposing
those hashes, and rotate the other roles' passwords as well.
The revoke replicates, and a replica running with partial_revokes=OFF
stops its SQL thread on it. The operator therefore turns
partial_revokes ON on every site, and applies the revoke only once it
has confirmed that on all of them. Until then, a CredentialReconcileFailed
event names the site it could not confirm and the app user can still
write the mysql schema. Once any partial revoke exists, MySQL refuses to
turn partial_revokes OFF again and forces it ON at startup.
Tenant MysqlDatabase reconciliation manages explicit schema grant rows and does not create partial restrictions on global privileges. The app user's mysql.* restriction described above remains intentional. Older tenant reconciliation could also leave unintended restrictions on tenant schemas for shared principals. The upgrade repairs identifiable escaped-name artifacts; real-schema restrictions outside the tenant's desired GRANT require review and recovery by the privilege owner, including after a tenant CR was deleted. Do not remove the intentional app restriction on mysql.* during recovery.
Mitigations. Rotate appSecretName on detection. The operator
propagates password changes via ALTER USER on the next reconcile.
Consumers that cache connection strings need to reconnect.
A sidecar pod is compromised
A sidecar pod shares the pod network namespace with its mysqld.
An attacker with code-execution in the sidecar can:
- Connect to
mysqldon127.0.0.1:3306using theoperatorcredentials the sidecar holds. This is the same authority as "operator password leaks", but scoped to the one MySQL instance. - Self-fence the MySQL (write
read-only=ON, kill its own pod) — the sidecar already does this legitimately, but an attacker can use it for denial-of-service. The peer sidecar will eventually promote, so the net effect is an unscheduled failover. - Write, overwrite, or delete archive objects with the PITR
storage credentials the sidecar holds. The sidecar legitimately
deletes archived binlogs during its retention sweep, so these
credentials carry delete rights. An attacker can corrupt the
site's manifest (making
status.pitr.archiveCoverageTomislead operators) and remove anything else the credentials reach. Scope the PITR bucket credentials to the PITR prefix, and enable bucket versioning or object lock where the provider supports it, so deleted objects can be recovered. The archive descriptor (archive.json) is write-once by protocol, not by storage permission. An attacker with these credentials can replace it; writers re-verify it before every manifest save and delete and stop on a change (archive descriptor changed), and on an encrypted archive a forged descriptor cannot carry a validkeyCheckwithout the passphrase. On a plaintext archive the descriptor is not authenticated. The descriptor is plaintext on every archive: it names the source group and its UID, the archive ID and the encryption mode. Backup stamps (<location>.bloodraven-backup.json) are encrypted with the dump passphrase when the profile encrypts. - Read the MySQL data directory — the sidecar's mount point on
/var/lib/mysqlis shared withmysqld.
The blast radius is one MySQL site of one failover group. The
attacker does not gain operator-level Kubernetes privileges — MySQL
pods run as the namespace's default ServiceAccount with
automountServiceAccountToken: false, so the sidecar holds no Kubernetes
token. A group with extraContainers or extraInitContainers keeps the
Kubernetes default and mounts the default ServiceAccount token, so grant
that account nothing those containers do not need. The attacker also does not reach
the peer site directly (the peer only trusts its own sidecar and the
operator). The worst observable consequence is a forced failover plus
data exfiltration from the one site.
Mitigations. Use a distroless sidecar image, keep its process
footprint small, and audit the sidecar image in the same supply-chain
pipeline as mysqld. Alert on bloodraven_failovers_total spikes
that coincide with sidecar restarts.
A backup bucket credential leaks
Bounded by the bucket IAM policy — not by Bloodraven. Bloodraven
passes the credentials (from spec.backup.profiles[].storage.secretRef)
to mysqlsh, mysqlbinlog, and the sidecar archiver as env vars.
With the credential, an attacker can:
- Delete or overwrite backup objects (full dumps and binlogs), destroying the PITR chain.
- Read every dump — they contain the full database content.
Mitigations. Scope bucket IAM to the specific profile's prefix. Enable object-lock / versioning / MFA-delete on the bucket. Rotate bucket credentials independently of MySQL credentials.
The DNS zone trust is compromised
Bloodraven writes DNSEndpoint CRs; external-dns translates them
into DNS records. If the DNS provider or external-dns itself is
compromised, an attacker can repoint MySQL traffic to an
attacker-controlled endpoint. Bloodraven cannot prevent this.
The only mitigation is defense in depth at the DNS layer — zone-
signing, IAM scoping of external-dns, and short TTLs that limit
the window during which a malicious repoint holds.
What Bloodraven does not defend against
Explicit non-goals, so the list above is complete:
- Container escape from
mysqld— Out of scope; treatmysqldas you would any other privileged workload in your cluster. - Kernel-level exfiltration of data in MySQL memory — out of scope; use confidential-computing primitives at the node level if required.
- Malicious application SQL — the operator cannot distinguish a legitimate write from a malicious one. Application-level access control is the app's responsibility.
- Kubernetes API server compromise — nothing below the API server can be trusted if the API server is compromised.
- Supply-chain attacks on the operator or sidecar images — pin and mirror per Production hardening.
- DoS of the control plane — an attacker with
patchon a CR can trigger reconciles at whatever rate the API server allows. The operator is rate-limited by controller-runtime's workqueue, but a motivated abuser can still produce noise. - Live compromise of a MySQL pod when
spec.encryptionAtRestis on — a runningmysqldholds decrypted keys in memory by necessity, and the keyring file is readable inside its own pod. Data-at-rest encryption defends stolen volumes, snapshots, and node disks; it does not defend a live process. It is also not Oracle "MySQL Enterprise TDE" and Oracle does not present file keyrings as a regulatory-compliance solution.
Hardening summary
If you only read one section, read this. The minimum for a production deployment:
- Give each workload its own
spec.credentialsrole Secret; never hand out the operator Secret. - Set
spec.tlsand require it in the MySQL clients. - Scope
patchonMysqlFailoverGrouptightly — it is the reclone and planned-failover vector. - Keep the operator in its own namespace and do not grant its ServiceAccount to anyone else.
- Apply NetworkPolicy for operator and peer access to sidecar
:8080, and sidecar, dashboard, and observability access to operator:8082. Preserve sidecar escrow access and explicitly admit deployment clients on TLS:8443when enabled. - Store credential Secrets with a rotating provider and rotate on a schedule.
- Lock down the backup bucket IAM to the prefix the profile uses, and enable versioning / object-lock.
- Subscribe to divergent-GTID alerts (
bloodraven_divergent_transactions > 0) — that is your "someone wrote to the wrong primary" tripwire.
For the full checklist see Production hardening.
Data-at-rest encryption
Bloodraven can run MySQL with InnoDB data-at-rest encryption turned on, keeping the encryption key off the MySQL data PVC and off worker-node disks. It uses the GPL component_keyring_file keyring component that ships in MySQL Community Edition, so it needs no Oracle Enterprise licence and no CSI-level encrypted storage.
Architecture
How the Bloodraven operator, sidecars, MySQL pods, optional Dragonfly sidekicks, Services, and DNS records interact during normal operation and failover.

