Configuration

Security model

This page documents Bloodraven's security posture: the trust boundaries between its components, the authorizations each component holds, and the blast radius when a specific secret or identity is compromised. It complements Production hardening, which lists the settings to turn on; this page explains why they matter and what remains exposed when they are all correct.

Bloodraven is an in-cluster control plane for a tenant MySQL service. It is not a zero-trust system on its own. It relies on the hosting Kubernetes cluster for authentication, namespace isolation, and network policy. The guarantees below all assume a competent cluster operator is running it.

Security quick wins

  • Use spec.credentials with five distinct role Secrets.
  • Enable spec.tls and require TLS in MySQL clients.
  • Keep the auxiliary Service reachable by sidecars for self-fencing and PITR retention; restrict ingress rather than disabling it without a replacement route.
  • Add NetworkPolicy for operator, sidecar, MySQL, Prometheus, app, and backup paths.
  • Pin and mirror operator, sidecar, MySQL, and backup images.
  • If Dragonfly is enabled, protect its password Secret, active Service, and admin port with the same care as the MySQL data plane.
  • Link Secret rotation to Credentials And TLS.

Threat-model assumptions

AssumptionConsequence
Kubernetes API authn/authz is trustedRBAC boundaries are meaningful.
CNI enforces NetworkPolicy correctlyUnauthenticated sidecar and auxiliary endpoints can be isolated.
Secret storage is protectedMySQL, Dragonfly, TLS, S3, and encryption secrets remain confidential.
Node administrators are trustedA node admin can read mounted Secrets and MySQL data on that node.
Application workloads are less trusted than the operatorApps should only receive app or read-only MySQL credentials.

Trust model at a glance

Four identities participate in the required MySQL data path. Managed Dragonfly adds an optional cache/session data path when enabled. Everything else in the cluster is considered untrusted.

IdentityWhat it isTrust level
OperatorSingle-replica Deployment running bloodraven, bound to a cluster-wide ServiceAccount. Owns the reconcile loop and DNS writes.Fully trusted — it is the control plane.
SidecarPer-MySQL-pod container running bloodraven-sidecar. Serves read-only status HTTP, manages MySQL startup, writes binlog manifests, and self-fences its MySQL.Trusted for the single pod it runs in.
MySQLmysqld itself, with the per-role users Bloodraven creates (operator, app, readonly, monitor, backup).Trusted for query execution; each MySQL user is trusted only for its role's grants.
DragonflyOptional per-site Dragonfly pods created when spec.dragonfly.enabled=true. The operator configures replication, role/traffic labels, and promotion.Trusted for cache/session data only. Do not store durable application state here.
ApplicationWorkloads that connect to the primary/replica Services using appSecretName / readOnlySecretName.Untrusted by the operator — apps are tenants.

All intra-cluster traffic between these identities flows over the pod network. Bloodraven does not authenticate HTTP calls between the operator and the sidecars, and it does not mTLS-protect the sidecar HTTP endpoint. Isolation is the hosting cluster's job (NetworkPolicy, namespace boundaries, CNI-level encryption if you require it).

Authorization — who can do what

Operator ServiceAccount

The operator runs with a cluster-scoped ClusterRole (config/rbac/role.yaml, mirrored in charts/bloodraven/templates/clusterrole.yaml). It holds the minimum set of verbs to reconcile MysqlFailoverGroup and MysqlBackup resources across any namespace. Notably:

  • Full CRUD on Secrets, ConfigMaps, Services, PersistentVolumeClaims, Jobs, CronJobs, Deployments, PodDisruptionBudgets — cluster-wide. The operator reads credential Secrets, renders MySQL config into ConfigMaps, and creates PVCs, Services, Pods, and Jobs per site.
  • Full CRUD on dnsendpoints.externaldns.k8s.io — the failover path writes DNSEndpoint CRs that external-dns then pushes to the DNS provider.
  • get/list/watch/patch on pods and nodes, plus get on pods/log — used for the tainter, the backup-sentinel log tail, and placement contract enforcement.
  • Full CRUD on leases.coordination.k8s.io — leader election and durable deployment coordination.
  • create on tokenreviews.authentication.k8s.io — validates projected ServiceAccount tokens for the optional deployment API.
  • create/get/update/patch on serviceaccounts, roles and rolebindings — each group with backup or verification schedules gets a mysql-<group>-schedule-trigger ServiceAccount, bound to a Role that can only create MysqlBackup and MysqlBackupVerification resources and read that group. The schedule CronJob pods run under it. Kubernetes only lets the operator grant verbs it already holds.

There is no namespaced partitioning. A single operator install reconciles groups in every namespace. If you need tenant isolation (one operator per tenant), deploy one operator per namespace scope and use a Role instead — the code supports this but the shipped chart does not.

MySQL user roles

When spec.credentials is used, the operator creates five MySQL users with the following effective grants (see CRD Reference → CredentialsSpec):

RoleEffective authority
operatorALL PRIVILEGES ON *.* WITH GRANT OPTION — full admin. Used by the operator, the sidecar and restore Jobs (which drop and load schemas, and must keep working under a full-instance restore's offline_mode fence).
appDML, DDL, views, routines, triggers, events, temporary tables, LOCK TABLES and SHOW DATABASES on *.*, with every write and DDL privilege partially revoked on the mysql system schema — no administrative privileges (SUPER, CREATE USER, PROCESS, replication, ...) and no GRANT OPTION. Read/write for tenant workloads.
readonlySELECT, SHOW VIEW, SHOW DATABASES, PROCESS — read replicas.
monitorPROCESS, REPLICATION CLIENT + SELECT on performance_schema.* — mysqld_exporter and similar scrapers.
backupSELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER, RELOAD, BACKUP_ADMIN, REPLICATION CLIENT — MySQL Shell dump jobs.

Only operatorSecretName is required. A role you leave unset has no user, so give each workload its own role Secret instead of the operator's: the operator account is full admin.

Destructive annotations on the CR

patch on mysqlfailovergroups is the narrow privilege that controls every operator-initiated destructive action. The annotations the operator acts on are:

  • bloodraven.shipstream.io/reclone-site=<site>:confirm=<group>:uid=<uid>[:gtidPrefix=<prefix>] — Triggers CLONE INSTANCE on the named site, overwriting its data directory from the current primary. The group name and uid keep a request from applying to another group or a re-created one of the same name, and on a diverged site the GTID-prefix confirmation (≥ 8 chars of the source UUID of status.sites[<site>].divergentGtid) prevents typos from wiping the wrong site. See Failover → Reclone flow.
  • bloodraven.shipstream.io/planned-failover=<site> — Drains writes, honours the anti-flap cooldown, waits for the target to catch up, and promotes the named site. When Dragonfly is enabled, this also drives the Dragonfly sync and REPLTAKEOVER phases.
  • bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID> — Acknowledges an automation hold after a planned failover ended ManualRecoveryRequired, which resumes automatic failover, split-brain resolution and DNS reconciliation for the group. Grant it with the same care as the planned-failover annotation, and only after the split brain is resolved.
  • bloodraven.shipstream.io/revoke-deployment-leases - Revokes the group's deployment leases with reason OperatorRevoked. This can unblock planned disruption and fences active deployment clients. See deployment coordination operations.
  • bloodraven.shipstream.io/release-pitr-archive-binding=<id>:confirm=<group>:uid=<uid> — Releases one PITR archive encryption binding and deletes its key snapshot Secrets, which may be the cluster's only copy of an archive's passphrase. Refused while the binding is in use; never commit it in GitOps. See Releasing a binding.
  • bloodraven.shipstream.io/last-failover and bloodraven.shipstream.io/last-failover-target — Operator-owned durable anti-flap history. A writer can extend or clear the effective cooldown and change the remembered fencing target. The operator rejects malformed, unpaired, unknown-site, and implausibly future-dated values, but these keys must still be protected from manual or third-party mutation.
  • bloodraven.shipstream.io/promotion-gtid-executed and bloodraven.shipstream.io/promotion-gtid-executed-for — Operator-owned out-of-band copy of the promotion watermark and the event it belongs to. The watermark gates which candidates may be promoted and how a returning old primary is judged, so a writer can weaken the promotion-data-regression check. The operator ignores a watermark whose binding does not name the event last-failover and last-failover-target record.
  • bloodraven.shipstream.io/planned-failover-terminal — Operator-owned terminal marker of a planned failover whose status writes stalled. With outcome ManualRecoveryRequired it is an automation-hold record on its own: deleting it (together with status.plannedFailoverHold and the AutomaticFailoverHeld condition, which need mysqlfailovergroups/status rights) resumes automatic failover while a split brain may stand, and writing one holds automation or ends a live run. Treat write access to it like the acknowledgement annotation.
  • bloodraven.shipstream.io/planned-failover-stall-since, bloodraven.shipstream.io/planned-failover-manager-absent-since, bloodraven.shipstream.io/planned-failover-confirmed and bloodraven.shipstream.io/planned-failover-status-fence — Operator-owned bookkeeping of a run in flight (deadline starts, the confirmation time, the status-write fence). A writer can shorten or extend a run's deadlines, or make an ambiguous status write look settled. The operator accepts a deadline start only for the run it names.

All of these keys are listed in Well-known metadata.

Backing up and restoring a MysqlFailoverGroup

A restore re-creates the object without its status (the API server never takes status on create), so a run in flight is not restored and its keys would name nothing. When a backup tool (Velero, a GitOps export) restores the object:

  • Drop the request and acknowledgement annotations (planned-failover, acknowledge-planned-failover-hold): a restored request starts a planned failover, and a restored acknowledgement could clear a hold nobody reviewed.
  • Drop planned-failover-stall-since, planned-failover-manager-absent-since, planned-failover-confirmed and planned-failover-status-fence: they belong to the run whose status was not restored.
  • Keep a planned-failover-terminal marker whose outcome is ManualRecoveryRequired: it is the only hold record left, and the restored group stays held until the split brain is checked and the hold is acknowledged with kubectl bloodraven ack-planned-failover-hold. Drop a marker with any other outcome.
  • Restore last-failover, last-failover-target, promotion-gtid-executed and promotion-gtid-executed-for together or not at all. Restoring them keeps the anti-flap cooldown and the watermark; dropping them makes the group look as if it never failed over, which shortens the cooldown and removes the watermark check until the next promotion.

An attacker with patch mysqlfailovergroups in the target namespace can therefore cause a data wipe of one site (bounded by the GTID confirmation interlock), an unscheduled failover, or a planned Dragonfly cache/session outage during snapshot-restore upgrade. RBAC on the CRD is the only gate — scope MysqlFailoverGroup edit rights as tightly as you would scope kubectl delete pvc.

Write access to spec.sites[].taintNodeSelector reaches beyond the namespace. The operator applies a NoExecute taint to every node in the cluster that matches the selector, and the selector can name any node labels. Anyone who can create or edit a MysqlFailoverGroup can therefore evict untolerating pods from any node. Grant that right only to principals you would trust to taint nodes.

Network surface

ListenerPortAuthenticated?Purpose
Operator metrics:8080NoPrometheus scrape (/metrics).
Operator health probes:8081NoKubernetes liveness/readiness.
Operator auxiliary HTTP:8082No/status, /active-site, /pitr-cutoff, /ws/status (WebSocket).
Operator keyring escrow HTTPS:8443 by defaultBearer token + TLS/keyring/escrow; enabled with auxiliary.escrowTLS.
Operator deployment HTTPSSame TLS :8443 listenerProjected ServiceAccount token + TokenReview + database allowlist/deploy/v1; opt-in with auxiliary.deployAPI.enabled.
Sidecar HTTP:8080No/livez, /health, /status, /peer/ping, /peer/active-site, /archiver/status, /keyring/status. Consumed by the operator, the peer sidecar, and the kubelet (liveness on /peer/ping, readiness on /health).
MySQL:3306Yes (password; TLS if spec.tls set)Data plane.
Dragonfly client:6379 by defaultPassword when spec.dragonfly.auth is setOptional Redis-compatible cache/session data plane.
Dragonfly admin:9999 by defaultPassword when spec.dragonfly.auth is setOptional operator control surface for replication, promotion, and snapshot commands.

Two HTTP API surfaces authenticate callers: keyring escrow and the optional deployment API. The unauthenticated ones expose read-only status and GTID coordinates and accept no state-changing requests. Sidecars do act on their responses: they self-fence when /active-site or /peer/active-site names another site as active, and the binlog archiver keeps binlogs back to the /pitr-cutoff answer. The archiver asks with namespace, group, archive (its BLOODRAVEN_PITR_ARCHIVE_ID) and coverage=gtid-v1; the operator answers only for an archive ID derived for that group and the destination of one of its archive bindings. Retention requires complete file GTID coverage in every returned group backup bound, pauses new retirements during in-flight backups, and retains the local 24-hour age floor. File proofs must match verified content identities. The handler uses the informer cache, bounded lookup admission, singleflight and safe deferral caching (AUDIT H2); rate-limited replies carry no delete authority. The coverage=gtid-v1 capability prevents older clients using a time-only cutoff; it does not authenticate the caller or response. The response exposes backup GTID bounds to allowed network callers. A sidecar ignores peer views while the operator answers and treats a peer timestamp from its future as current, but a forged response can still fence a site. That is why the NetworkPolicy assumption below matters. The design assumes the pod network is protected by NetworkPolicy and that the operator + sidecars run in a namespace that does not accept untrusted ingress. In particular:

  • Do not expose :8082 outside the cluster. The /active-site and /status endpoints do not leak credentials but do leak topology, site names, zone labels, and GTID coordinates — useful reconnaissance for an attacker already inside the cluster.
  • The WebSocket push stream (/ws/status) is likewise unauthenticated. Anyone that can reach port 8082 can read real-time failover events. Browser upgrades are limited to same-origin pages unless auxiliary.service.wsAllowedOrigins lists other origins; avoid setting it to "*", which lets any web page read the stream through a user's browser. auxiliary.service.wsMaxClients (default 100) caps concurrent stream clients, counting handshakes in progress, and rejects extra upgrades with 429. 0 removes the cap; avoid it.
  • If you require scrape auth, front the metrics endpoint with a sidecar (e.g. kube-rbac-proxy). The chart does not ship one.
  • /keyring/status on the sidecar reports digests and encryption coverage — never key material. It leaks the same class of reconnaissance as /status.

Keyring escrow: /keyring/escrow. This state-changing endpoint is isolated on the TLS listener and authenticates independently of the deployment API. Callers present a per-site bearer token that the operator mints into a Secret mounted only into that site's pod, and only while that site is deliberately unsealed. A sealed site carries no token at all. The token Secret itself persists and is not rotated, so the operator also rejects pushes for any site whose keyring is sealed (or escrowed and about to seal); a token copied during an unseal window cannot be replayed afterwards. Beyond the token: the body is size-capped, the claimed digest must match a fresh hash of the payload, escrow Secrets are immutable (so an accepted push can only add a version, never rewrite the one a running MySQL is sealed against), and the reconcile loop independently re-reads the stored Secret before it will seal anything. See Data-at-rest encryption.

The sidecar validates the listener certificate against the CA mounted from the failover group's spec.tls Secret. There is no plaintext fallback and redirects are rejected so neither the bearer token nor the keyring can be sent to another endpoint.

Deployment API authorization

/deploy/v1 is a second state-changing auth surface on the same TLS listener, not on plaintext auxiliary HTTP. It validates projected tokens with TokenReview using audience bloodraven-deploy by default. The authenticated namespace/ServiceAccount pair must appear in MysqlDatabase.spec.deploymentClients for the requested group. The database CR name defines the instance identity. Unauthorized requests return a non-disclosing 403; authentication failures return 401.

A deployment client can observe its authorized group, serialize migrations, and defer planned disruption while it holds a renewable lease. It cannot delay emergency failover or use the API to patch arbitrary resources. Restrict writes to deploymentClients: granting a caller access can let it repeatedly renew a hold and delay planned maintenance. Use a dedicated ServiceAccount rather than an application's general identity.

Lease tokens are random, hash-only in durable coordination Leases, and separate from ServiceAccount tokens. Repeating a same-operation POST rotates the lease token because the original cannot be recovered. Topology changes and explicit administrative revocation fence deployments; neither permits automatic continuation of uncertain DDL. Protect Lease writes and the revoke annotation as control-plane privileges.

The chart's optional deployment NetworkPolicy preserves escrow access for managed MySQL pods and leaves metrics, probes, and auxiliary HTTP reachable. It limits the shared TLS port, not URL paths, and does not replace authentication or a complete cluster network policy. See Deployment API for exact selectors and additional ingress rules.

TLS and data-in-flight

When spec.tls.secretName references a Secret with ca.crt, tls.crt, and tls.key, the operator mounts it into every MySQL pod and renders config that sets ssl-ca/ssl-cert/ssl-key andrequire-secure-transport=ON. Without TLS set, MySQL accepts plaintext connections from anything on the pod network.

  • Replication between sites uses the same MySQL port and the same server certificate — cross-site replication is TLS-protected only when spec.tls is set. Replication channels set SOURCE_SSL_CA to the mounted ca.crt (MySQL VERIFY_CA): the replica checks that the source's certificate chains to that CA, but does not check the host name, because replicas dial the -internal per-site Service. Clone (CLONE INSTANCE ... REQUIRE SSL) is encrypted but does not verify the donor's certificate. Mirror a single issuer across sites, or ensure both sites' certs chain to a CA each other trusts.
  • The sidecar, operator, backup Jobs, and MySQL Shell dump workers all trust the CA in ca.crt. When you rotate the CA, rotate the mounted Secret; the operator detects the hash change and rolls pods.
  • Cert-manager integration is not automatic — Bloodraven expects the Secret to exist. Use a cert-manager Certificate resource to populate it.

Dragonfly traffic is not covered by spec.tls; configure NetworkPolicy and, where required, Dragonfly-native TLS or a service mesh outside Bloodraven's managed flags. Do not expose the Dragonfly admin port to applications.

Secrets at rest and in memory

  • Nothing is written to disk that wasn't already on disk. The sidecar reads the operator Secret's username/password from the read-only volume at /etc/mysql/creds/operator (MYSQL_CREDS_DIR), so that password is not in the sidecar's environment. It keeps the DSN in memory and does not write credentials to a file.
  • Logs never include passwords. DSNs are built for in-process use and are never logged; SQL statement errors are truncated before logging. Operator events and condition messages are safe to ship to a log aggregator.
  • Secret content is copied into env and files as MySQL requires. The MySQL container's environment holds MYSQL_ROOT_PASSWORD; per-role passwords are mounted as files. The root password is only needed during first boot, but the operator keeps it in the pod spec for simplicity. Anything that can kubectl exec into the pod can read them.
  • Published hashes are keyed. The bloodraven.shipstream.io/spec-hash annotation on each site's Deployment and pods, and the bloodraven.shipstream.io/credential-hash annotation on the group, change when a password or certificate changes so pods roll and grants re-apply. They are readable by anyone who can read those objects, so Secret contents enter them only as an HMAC under a random per-group key in the mysql-<group>-digest-key Secret; without that key they cannot be used to check password guesses. Deleting the key Secret is safe: the operator creates a new one, and every site rolls once through the ordered update.
  • Backup passphrases are snapshotted. For an encrypted backup profile the operator copies the trimmed passphrase into immutable Secrets it creates in the group's namespace: one per accepted PITR archive key (mysql-<group>-pitr-key-<h10>, label app.kubernetes.io/name=mysql-pitr-key) and one per backup (mysql-dump-key). They are a second copy of the key behind the same Secret RBAC as your Secret, have no owner reference, and survive group deletion until released or deleted by label. The <h10> name suffix is an HMAC under the group's digest key over the destination and the bytes, so the name reveals nothing about the key. An archive snapshot's bloodraven.shipstream.io/pitr-accepted annotation carries the archive descriptor's keyCheck, a passphrase verifier (offline brute-force material for a weak passphrase, like the descriptor itself); it lives only on the Secret, never in the group's status. Only the bytes pods consume enter the pod fingerprint (TLS ca.crt/tls.crt/tls.key, role username/password, MYSQL_ROOT_PASSWORD); passphrase Secrets do not. See Key snapshots.
  • Dragonfly auth is optional but recommended for shared namespaces. When spec.dragonfly.auth is set, the operator projects the password Secret into Dragonfly pods as an environment variable and uses it for its own Redis-protocol connections.
  • MySQL data at rest is plaintext unless you enable spec.encryptionAtRest. With it enabled, tablespaces, redo, undo, binary and relay logs are encrypted and the keyring lives only on tmpfs — but the key then lives in a Kubernetes Secret, which means etcd becomes part of your key custody. Enabling the feature without API-server encryption at rest for Secrets moves the plaintext key from the MySQL disk to the control-plane disk rather than protecting it. See Data-at-rest encryption.

Compromise scenarios

The operator ServiceAccount token leaks

This is the worst-case credential compromise. With operator-level cluster access, an attacker can:

  1. Read every namespace's Secrets. Including MySQL credential Secrets for every MysqlFailoverGroup the operator manages. Treat this as equivalent to leaking all per-tenant MySQL credentials and all TLS private keys referenced by spec.tls.
  2. Create or patch arbitrary Deployment, Pod, Job, CronJob, Service, and PVC objects cluster-wide. Because the operator has full CRUD on these, the attacker has a general-purpose tenancy breakout through pod spec injection.
  3. Write DNSEndpoint CRs that external-dns will push to the public DNS zone, enabling a DNS-hijack attack against any name external-dns manages (not just MySQL names).
  4. Exfiltrate PVC contents by scheduling a pod that mounts the target PVC — the ServiceAccount can create pods and PVC mounts.
  5. Create or patch Dragonfly StatefulSets and Services when the feature is enabled, including changing cache/session routing labels.

Mitigations. Pin the chart's serviceAccountName, bind it to a namespace-scoped Role where possible, run the operator in its own namespace, and rely on the audit log to detect token reuse from unexpected source IPs. Rotate the token (delete the ServiceAccount's token Secret) on any suspected compromise — Kubernetes issues a new projected token to the pod automatically.

This is the single largest blast radius in the product. A multi-operator, namespace-scoped deployment model is on the wishlist but not implemented today.

The MySQL operator / root password leaks

Inside the DB, operator and root are equivalent — both carry ALL PRIVILEGES ... WITH GRANT OPTION. An attacker with either can:

  1. Read, modify, or drop any schema. Including running SELECT ... INTO OUTFILE on any path MySQL can reach (bounded by secure_file_priv, which Bloodraven sets to the data directory).
  2. Create new MySQL users and grant them arbitrary privileges, surviving a password rotation of the known users.
  3. Run STOP REPLICA / RESET REPLICA ALL / CHANGE REPLICATION SOURCE TO ... — directly breaking replication or repointing a replica at an attacker-controlled primary.
  4. Trigger SHUTDOWN (unless SUPER is gated), rebooting the MySQL instance.

The blast radius is per-failover-group, not cluster-wide. The operator will detect the resulting replication breakage and attempt recovery or raise Degraded=True, but it cannot distinguish malicious writes from legitimate ones — divergent GTIDs are the only signal that reaches the status.sites[].divergentGtid field.

Mitigations. Store credential Secrets with a rotating provider (ESO, sealed-secrets, Vault). Set alerts on bloodraven_divergent_transactions > 0. Keep the operator's Secret in the same namespace as the failover group so a namespace-compromise does not cascade.

The MySQL app password leaks

Bounded by the app grants: read/write across every schema, but no SUPER, no GRANT OPTION, no REPLICATION privileges. The attacker cannot:

  • Run CREATE USER, GRANT, or SET GLOBAL, or write through read_only (no SUPER / CONNECTION_ADMIN).
  • Break replication (lacks REPLICATION CLIENT / REPLICATION SLAVE).
  • Run KILL on other sessions (lacks PROCESS unless you deviate from the default).
  • Drop the MySQL server itself.

The attacker can run arbitrary SQL against the tenant data. This is the same blast radius as an application SQL-injection — assume any compromise of app yields full data read/write for the application's schemas. It does not include writing the mysql system schema: the operator runs MySQL with partial_revokes=ON and partially revokes every write and DDL privilege there, so the attacker cannot edit the grant tables to hand app the privileges above at the next privilege reload or restart. The attacker can still read the mysql schema, including the password hashes in mysql.user; treat a leaked app password as exposing those hashes, and rotate the other roles' passwords as well.

The revoke replicates, and a replica running with partial_revokes=OFF stops its SQL thread on it. The operator therefore turns partial_revokes ON on every site, and applies the revoke only once it has confirmed that on all of them. Until then, a CredentialReconcileFailed event names the site it could not confirm and the app user can still write the mysql schema. Once any partial revoke exists, MySQL refuses to turn partial_revokes OFF again and forces it ON at startup.

Tenant MysqlDatabase reconciliation manages explicit schema grant rows and does not create partial restrictions on global privileges. The app user's mysql.* restriction described above remains intentional. Older tenant reconciliation could also leave unintended restrictions on tenant schemas for shared principals. The upgrade repairs identifiable escaped-name artifacts; real-schema restrictions outside the tenant's desired GRANT require review and recovery by the privilege owner, including after a tenant CR was deleted. Do not remove the intentional app restriction on mysql.* during recovery.

Mitigations. Rotate appSecretName on detection. The operator propagates password changes via ALTER USER on the next reconcile. Consumers that cache connection strings need to reconnect.

A sidecar pod is compromised

A sidecar pod shares the pod network namespace with its mysqld. An attacker with code-execution in the sidecar can:

  1. Connect to mysqld on 127.0.0.1:3306 using the operator credentials the sidecar holds. This is the same authority as "operator password leaks", but scoped to the one MySQL instance.
  2. Self-fence the MySQL (write read-only=ON, kill its own pod) — the sidecar already does this legitimately, but an attacker can use it for denial-of-service. The peer sidecar will eventually promote, so the net effect is an unscheduled failover.
  3. Write, overwrite, or delete archive objects with the PITR storage credentials the sidecar holds. The sidecar legitimately deletes archived binlogs during its retention sweep, so these credentials carry delete rights. An attacker can corrupt the site's manifest (making status.pitr.archiveCoverageTo mislead operators) and remove anything else the credentials reach. Scope the PITR bucket credentials to the PITR prefix, and enable bucket versioning or object lock where the provider supports it, so deleted objects can be recovered. The archive descriptor (archive.json) is write-once by protocol, not by storage permission. An attacker with these credentials can replace it; writers re-verify it before every manifest save and delete and stop on a change (archive descriptor changed), and on an encrypted archive a forged descriptor cannot carry a valid keyCheck without the passphrase. On a plaintext archive the descriptor is not authenticated. The descriptor is plaintext on every archive: it names the source group and its UID, the archive ID and the encryption mode. Backup stamps (<location>.bloodraven-backup.json) are encrypted with the dump passphrase when the profile encrypts.
  4. Read the MySQL data directory — the sidecar's mount point on /var/lib/mysql is shared with mysqld.

The blast radius is one MySQL site of one failover group. The attacker does not gain operator-level Kubernetes privileges — MySQL pods run as the namespace's default ServiceAccount with automountServiceAccountToken: false, so the sidecar holds no Kubernetes token. A group with extraContainers or extraInitContainers keeps the Kubernetes default and mounts the default ServiceAccount token, so grant that account nothing those containers do not need. The attacker also does not reach the peer site directly (the peer only trusts its own sidecar and the operator). The worst observable consequence is a forced failover plus data exfiltration from the one site.

Mitigations. Use a distroless sidecar image, keep its process footprint small, and audit the sidecar image in the same supply-chain pipeline as mysqld. Alert on bloodraven_failovers_total spikes that coincide with sidecar restarts.

A backup bucket credential leaks

Bounded by the bucket IAM policy — not by Bloodraven. Bloodraven passes the credentials (from spec.backup.profiles[].storage.secretRef) to mysqlsh, mysqlbinlog, and the sidecar archiver as env vars. With the credential, an attacker can:

  • Delete or overwrite backup objects (full dumps and binlogs), destroying the PITR chain.
  • Read every dump — they contain the full database content.

Mitigations. Scope bucket IAM to the specific profile's prefix. Enable object-lock / versioning / MFA-delete on the bucket. Rotate bucket credentials independently of MySQL credentials.

The DNS zone trust is compromised

Bloodraven writes DNSEndpoint CRs; external-dns translates them into DNS records. If the DNS provider or external-dns itself is compromised, an attacker can repoint MySQL traffic to an attacker-controlled endpoint. Bloodraven cannot prevent this. The only mitigation is defense in depth at the DNS layer — zone- signing, IAM scoping of external-dns, and short TTLs that limit the window during which a malicious repoint holds.

What Bloodraven does not defend against

Explicit non-goals, so the list above is complete:

  • Container escape from mysqld — Out of scope; treat mysqld as you would any other privileged workload in your cluster.
  • Kernel-level exfiltration of data in MySQL memory — out of scope; use confidential-computing primitives at the node level if required.
  • Malicious application SQL — the operator cannot distinguish a legitimate write from a malicious one. Application-level access control is the app's responsibility.
  • Kubernetes API server compromise — nothing below the API server can be trusted if the API server is compromised.
  • Supply-chain attacks on the operator or sidecar images — pin and mirror per Production hardening.
  • DoS of the control plane — an attacker with patch on a CR can trigger reconciles at whatever rate the API server allows. The operator is rate-limited by controller-runtime's workqueue, but a motivated abuser can still produce noise.
  • Live compromise of a MySQL pod when spec.encryptionAtRest is on — a running mysqld holds decrypted keys in memory by necessity, and the keyring file is readable inside its own pod. Data-at-rest encryption defends stolen volumes, snapshots, and node disks; it does not defend a live process. It is also not Oracle "MySQL Enterprise TDE" and Oracle does not present file keyrings as a regulatory-compliance solution.

Hardening summary

If you only read one section, read this. The minimum for a production deployment:

  1. Give each workload its own spec.credentials role Secret; never hand out the operator Secret.
  2. Set spec.tls and require it in the MySQL clients.
  3. Scope patch on MysqlFailoverGroup tightly — it is the reclone and planned-failover vector.
  4. Keep the operator in its own namespace and do not grant its ServiceAccount to anyone else.
  5. Apply NetworkPolicy for operator and peer access to sidecar :8080, and sidecar, dashboard, and observability access to operator :8082. Preserve sidecar escrow access and explicitly admit deployment clients on TLS :8443 when enabled.
  6. Store credential Secrets with a rotating provider and rotate on a schedule.
  7. Lock down the backup bucket IAM to the prefix the profile uses, and enable versioning / object-lock.
  8. Subscribe to divergent-GTID alerts (bloodraven_divergent_transactions > 0) — that is your "someone wrote to the wrong primary" tripwire.

For the full checklist see Production hardening.