Tenant Databases

MysqlDatabase is a namespaced CRD that declares one database on a MysqlFailoverGroup, the user that owns it, any additional Secret-backed principals on it, and grants for principals that already exist.

It exists to answer one question: how does a caller provision a tenant database without being handed MySQL admin?

Why this exists

Before this CRD, the only way to create a per-tenant database on a Bloodraven-managed group was to connect to MySQL as an administrator. That meant the provisioner held the group's operator credential — GRANT ALL PRIVILEGES ON *.* WITH GRANT OPTION, plus MYSQL_ROOT_PASSWORD. Standing, long-lived, and effectively root on every tenant database in every group it provisioned into.

The obvious alternative — leasing a short-lived credential from a secrets engine such as OpenBao's database engine — does not work here, for reasons that are structural rather than incidental:

1. The secrets engine dials the database. It opens the connection itself and fails closed when it cannot reach the host. A MySQL instance reachable only on a private network cannot be configured as a target at all.
2. Bloodraven is already the credential authority. reconcileRole issues CREATE USER IF NOT EXISTS … IDENTIFIED BY and ALTER USER … IDENTIFIED BY from a referenced Secret's bytes. The Secret is desired state for the MySQL user, not a credential to a user that already exists. An external engine rotating the same principals would be a second writer with no arbitration.
3. The operator credential cannot be leased even in principle. It carries MYSQL_ROOT_PASSWORD — the value MySQL is initialized with — and the operator falls back to root with it when the operator user does not yet exist. A credential that must be known before the database exists cannot be minted by something that connects to the database.

So the component that holds MySQL admin is Bloodraven, because it already must. MysqlDatabase is the way to ask Bloodraven to create a tenant database without being given the keys to do it yourself. The caller's MySQL credential is replaced by Kubernetes RBAC on a namespaced CRD.

The security property: a caller can provision a tenant database while holding no MySQL credential and no Secret access.

Example

apiVersion: shipstream.io/v1alpha1
kind: MysqlDatabase
metadata:
  name: tenant-acme
  namespace: bloodraven
spec:
  groupRef:
    name: main                    # MysqlFailoverGroup in the same namespace
  databaseName: acme_wms          # ^[A-Za-z0-9_]{1,64}$
  characterSet: utf8mb4           # default
  collation: utf8mb4_unicode_ci   # default

  owner:
    # Secret with keys `username` and `password`, same contract as
    # spec.credentials.*Secret on MysqlFailoverGroup.
    secretName: acme-mysql-owner
    privileges: [ALL PRIVILEGES]  # ON acme_wms.* only, never WITH GRANT OPTION

  # Additional principals this database brings into existence, each one
  # Secret-backed exactly like the owner.
  users:
    - secretName: support-ro-mysql   # keys `username` + `password`
      privileges: [SELECT]           # ON acme_wms.* only; ALL PRIVILEGES rejected
      resourceLimits:
        maxUserConnections: 5
        maxQueriesPerHour: 3600

  # Principals that must ALREADY exist. Grant-only: never CREATE USER.
  grants:
    - username: maester
      privileges: [SELECT, DELETE]

  deletionPolicy: Retain          # default

The owner's password arrives the way every other Bloodraven credential does: you write a Secret, Bloodraven applies it. Bloodraven never generates, returns, or stores a password — if it generated one it would need somewhere to put it, which reintroduces the custody problem this CRD exists to remove.

Deployment clients

The optional spec.deploymentClients list authorizes projected ServiceAccounts to use the group's Deployment API. It does not grant MySQL privileges. Each entry contains namespace and serviceAccount; namespace is the list-map key, so one ServiceAccount can be declared per namespace. An omitted or empty list authorizes no deployment callers.

spec:
  deploymentClients:
    - namespace: application
      serviceAccount: application-deploy

The API checks the caller pair and spec.groupRef.name. Lease instance is this MysqlDatabase's metadata name, not the SQL database name. Permission to edit deploymentClients can grant group-level observation and migration/hold authority, so protect it as part of database provisioning RBAC. A deployment client does not need generic MFG patch permission or Secret access.

Status

status:
  phase: Ready                    # Pending | Creating | Ready | Failed | Deleting
  observedGeneration: 3
  databaseCreated: true
  ownerUser: acme_app             # echoed from the Secret; NOT the password
  ownerHosts: ["%"]               # every host the owner account may exist on
  # pendingOwnerUser / pendingOwnerHosts: set only mid username rotation
  appliedGrants: [acme_app, maester]
  appliedUsers:                   # the spec.users[] ledger
    - secretName: support-ro-mysql
      username: acme_support
      hosts: [35.1.2.3, 35.4.5.6, 35.7.8.9]
      # pendingUsername / pendingHosts: set only mid username rotation
  activeSite: dc1
  lastAppliedHash: a1b2c3d4e5f6
  message: database acme_wms ready on site dc1
  conditions:
    - type: Ready
      status: "True"
      reason: DatabaseReconciled

observedGeneration and the Ready condition are the contract. They are how a provisioner reports provisioning state back to its own callers without opening a MySQL connection — treat them as API surface, not diagnostics.

status never carries credential material. ownerUser is a username. lastAppliedHash fingerprints the Secret's revision (UID + resourceVersion), never a digest of its bytes — status is caller-readable, and a content digest would let a status reader offline-check password guesses.

Phases

A MysqlDatabase applied before its MysqlFailoverGroup goes Pending, not Failed — that ordering is normal, not a fault.

Privileges

privileges is an allowlist, not a passthrough string:

ALL PRIVILEGES, SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER,
INDEX, REFERENCES, LOCK TABLES, SHOW VIEW, TRIGGER, EVENT, EXECUTE,
CREATE TEMPORARY TABLES, CREATE VIEW

Anything outside the list is rejected by the API server (the field is an enum) and rejected again in Go before any SQL is rendered. ALL PRIVILEGES cannot be combined with other entries, and cannot appear in a users[] entry at all.

GRANT OPTION is not on the list and never will be. Bloodraven never emits WITH GRANT OPTION from this CRD — an owner that can grant is an owner that can escape its own database. ALL PRIVILEGES here means GRANT ALL PRIVILEGES ON \acme_wms`.*`, which confers nothing outside that schema.

Identifiers are validated before they reach SQL rendering, not merely escaped: databaseName, characterSet and collation must match ^[A-Za-z0-9_]{1,64}$, and usernames ^[A-Za-z0-9_][A-Za-z0-9_.$-]{0,31}$. Rejection is the contract; escaping is the belt on top of the braces. MySQL's own schemas — mysql, sys, information_schema, performance_schema, case-insensitively — are rejected outright as databaseName: a tenant CR must never hold privileges on the grant tables.

hosts — one principal, several source addresses

Both the owner and every users[] entry accept an optional hosts list — the host part of the MySQL account. Omitted means ["%"], anywhere. Each entry is an IPv4 or IPv6 address, an IPv4 CIDR (203.0.113.0/24), an IPv4/netmask pair (10.0.0.0/255.255.255.0) or a wildcarded IPv4 pattern — % matches any run of characters, _ exactly one (10.0.%, 10.0.0._). MySQL has no IPv6 CIDR form, so 2001:db8::/32 is rejected rather than rendered into an account host that matches no client; list IPv6 clients as literals. Hostnames are rejected — MySQL would resolve them on every authentication, which puts DNS on the auth path and makes "who is this" a time-varying answer.

owner:
  secretName: acme-mysql-owner
  hosts: [35.1.2.3, 35.4.5.6, 35.7.8.9]   # e.g. a platform's static egress IPs

MySQL has no multi-host account — 'u'@'a' and 'u'@'b' are two accounts that merely share a name — so a principal with three hosts is three accounts underneath. The CR hides that on purpose: it is one record, one Secret, one grant set, and the operator renders every operation as a single multi-account statement (CREATE USER IF NOT EXISTS … a, b, c, ALTER USER … a, b, c, GRANT … TO a, b, c), which MySQL 8 executes atomically, so a password rotation across three hosts is one statement rather than three windows of inconsistency. Adding a host creates that account with the same credential and grants; removing one revokes and drops exactly that account (OwnerHostsRemoved / UserHostsRemoved). If a live sibling CR still claims the name, the drop is skipped with OwnerUserDropSkipped / UserDropSkipped and only the revoke runs. The hosts an account name may exist on are recorded per recorded username: status.ownerHosts for ownerUser, status.pendingOwnerHosts for an in-flight owner rotation target, and hosts / pendingHosts for each status.appliedUsers[] entry's username / pendingUsername. Each is a write-ahead union until the apply reaches Ready, so a crash mid-apply never leaves an account on a host nothing remembers, and deletion, rotation and entry removal drop each name only on the hosts recorded for that name. Keeping the lists separate matters during a rotation that also changes hosts: rotating a@10.0.0.1 to b@10.0.0.2 drops a on 10.0.0.1 only, never an unrelated a@10.0.0.2.

Two things to know before relying on it:

• MySQL must see the client's real source address. Behind a LoadBalancer/NodePort Service with externalTrafficPolicy: Cluster, kube-proxy SNATs to a node IP and every host rule is either useless or a lockout. Community MySQL has no PROXY-protocol support, so set externalTrafficPolicy: Local on the group's Services (spec.serviceTemplate.externalTrafficPolicy, overridable per site under spec.sites[].serviceTemplate) or use an L4 balancer that preserves source IPs. Bloodraven's DNS steering already points one hostname at whichever site's lbIP is primary, so a client with static egress addresses connects to one stable name and matches its hosts entry on whichever site is active — the host rule is a second, per-account layer on top of whatever the load balancer's own allowlist enforces.
• resourceLimits are per account. maxUserConnections: 5 with three hosts is five connections per host, fifteen in total. MySQL enforces MAX_USER_CONNECTIONS and MAX_QUERIES_PER_HOUR on each 'user'@'host' separately; there is no server-side way to share a cap across them.

Host scoping applies to the accounts this CRD creates — the owner and users[]. grants[] principals are created elsewhere and are granted on '%', as every account in spec.credentials is.

grants[] is grant-only

A MysqlDatabase brings a MySQL principal into existence only when you placed that user's password in a Secret first — the owner, and each users[] entry.

Every grants[] entry, by contrast, names a user that must already exist. The reconciler verifies it (SELECT 1 FROM mysql.user WHERE user = ? AND host = '%') and fails the CR with reason: GrantUserMissing if it does not. It never creates the user.

Without that split, "create a database" would imply "create arbitrary MySQL users", and this CRD would be a privilege-escalation primitive rather than a narrowing of one. users[] does not reopen it: an entry can only name an account whose credential the caller already holds in a same-namespace Secret.

If you hit GrantUserMissing, the fix is to create the principal as a group-level concern — it is shared across tenants, so it does not belong to any one MysqlDatabase. The CR re-checks on its own and goes Ready once the user exists; no CR edit is needed.

users[] — additional Secret-backed principals

spec.users[] declares principals this database brings into existence alongside the owner. Each entry copies the owner's trust shape exactly:

• The account exists only because the caller supplied its credential in a same-namespace Secret with keys username and password. The Secret is desired state, so rotation is a Secret write and nothing else.
• Privileges are granted ON <databaseName>.* only, from the same allowlist, never WITH GRANT OPTION. Unlike the owner, ALL PRIVILEGES is rejected — a second all-privileges principal is a shadow owner, and the field's first consumer is a SELECT-only reader.
• The username may not be a group-level principal (reason: UserReserved), may not be the owner or a grants[] entry, and may not collide with another entry.

Entries are keyed by secretName, capped at 8, and each may carry optional resourceLimits:

users:
  - secretName: support-ro-mysql
    privileges: [SELECT]
    resourceLimits:
      maxUserConnections: 5     # MAX_USER_CONNECTIONS
      maxQueriesPerHour: 3600   # MAX_QUERIES_PER_HOUR

Limits are desired state in both directions: omitting one renders 0 — MySQL's "no account-level cap" — on the next apply, so removing a limit from the spec actually clears it rather than silently leaving the old cap enforced.

Note that maxUserConnections: 0 is not necessarily unlimited: MySQL falls back to the server's global max_user_connections for accounts with no account-level cap, so the account is only truly uncapped when that global is 0 too. maxQueriesPerHour: 0 has no such fallback and does mean unlimited.

Removing an entry revokes and drops its principal. That is the intended behaviour and it is deliberately stricter than the database itself: a users[] account is an access grant, not tenant data, so leaving it behind after it has been un-declared would be a lingering credential. The drop is authorized by status.appliedUsers, not by the Secret — the Secret is often already gone by then (a support account is typically de-provisioned from the secret store first), and the ledger is the only durable record pairing an entry with the account it created. A drop that is vetoed because another CR still claims the principal still revokes its privileges on this database (UserDropSkipped).

status.appliedUsers is also the write-ahead record: an entry is stamped before its first CREATE USER runs, so a reconcile that creates an account and then fails to persist status can attribute that account to itself on the retry instead of wedging on PreExistingUser. It is stamped only after every account it names was checked (see Adoption is refused), and it carries usernames and hosts only, never credential material.

No managed principal can be a group-level principal

Bloodraven applies ALTER USER … IDENTIFIED BY from the owner Secret's bytes. That is correct desired-state behaviour for a user this CRD owns — it is what makes rotation a Secret write. Pointed at a Secret whose username is root, the operator user, or any other account named by spec.credentials, the same statement would instead reset a privileged account's password to whatever the Secret's author chose.

So the reconciler refuses. If spec.owner.secretName resolves to a username belonging to the group — root, replicator, MySQL's built-in system accounts (mysql.sys, mysql.session, mysql.infoschema), or any spec.credentials principal — the CR fails with reason: OwnerUserReserved and no statement is built, let alone executed. Every spec.users[] Secret is checked identically, failing with reason: UserReserved. Both checks run before the "nothing changed" short-circuit, so a Ready CR whose world shifted underneath it (a group Secret rotating into one of its usernames) is re-arbitrated rather than skipped. The check fails closed: if a group credential Secret cannot be read for any reason other than not existing, the reconcile errors and retries rather than proceeding with a partial reserved set.

This matters most if you deviate from the recommended split. In the intended deployment the caller has no secrets verbs at all (the Secret is rendered by an external controller), so it cannot name anything. If your provisioner does write its own owner Secrets, this check is what stops "provision a tenant database" from becoming "reset the operator's password".

One database, one CR

Two MysqlDatabase CRs on the same group must not claim the same databaseName or the same owner principal: desired-state ALTER USER means shared owners take turns resetting each other's password, and deletionPolicy: Delete on one duplicate would drop the other's live data. The reconciler refuses: the older CR wins, the newer fails with reason: DatabaseNameConflict or OwnerConflict before any SQL is rendered.

spec.databaseName and spec.groupRef are both immutable, enforced by the API server. MySQL has no schema rename and the reconciler has no way to move a schema between groups; either edit would apply fresh state and orphan what already exists, and a later deletionPolicy: Delete would aim cleanup at the wrong objects. Renaming or re-grouping a tenant database is a migration, not a spec edit.

Rotation

Rotating the owner password is a Secret write and nothing else:

kubectl -n bloodraven patch secret acme-mysql-owner \
  --type merge -p '{"stringData":{"password":"new-password"}}'

The reconciler watches every Secret the CR references — the owner's and each users[] entry's — so it applies ALTER USER on the next reconcile. Re-applying an unchanged CR issues zero MySQL statements — the reconciler compares a fingerprint of the spec, every referenced Secret's revision, the active site and the group's identity against status.lastAppliedHash — which is what keeps a tenant-dense cluster from hammering the primary.

Rotating the username in the Secret is also just a Secret write, and it revokes what it replaces. Rotation is create-before-drop: the reconciler creates the new account, grants it, applies the database, and only then drops the previous one (OwnerUserRotated / UserRotated fires after the full handover). A failure mid-handover leaves both accounts alive and retried — never a window in which the tenant has no owner. A rotation performed because a credential leaked actually revokes the leaked credential — status.ownerUser and status.appliedUsers[].username keep the old name until the old account is really gone, so a failure mid-rotation retries rather than leaving a shadow account. If the Secret moves on again before that retry has dropped the previous account, the unresolved rotation target is dropped first — before the new target is written ahead — so an account this CR created never falls off the ledger between two rotations.

Renaming the Secret a users[] entry points at — an ESO target renamed, a chart refactor — while the username inside stays the same is neither a rotation nor a removal. The status records attribute accounts to the CR, not to the surface that happened to create them, so the new secretName adopts the existing account, its grants are re-applied, and the old record is re-keyed in place. Nothing is revoked or dropped and the credential never goes away; a UserTransferred event records the handover in place of the UserRemoved you would otherwise expect. The same rule covers a name moving between the owner and a users[] entry, and keeps a rotation from taking an account with it when another surface of the same CR — a new entry, the owner, or grants[] — declares the name the rotating principal just freed. Attribution is per account: the adopting surface takes over the hosts the old record held, drops the name on any of those hosts it does not declare, and still refuses a same-named account on a host no record of this CR names.

A users[] Secret that has not been rendered yet parks the whole CR in Pending (UserSecretMissing / UserSecretIncomplete) rather than applying a partial list — the same ordering tolerance the owner Secret gets, which is what lets a secret-store write, an ESO sync and a CR apply land in any order. The trade is that one lagging users[] Secret also delays unrelated spec changes on that CR until it arrives.

Privileges are desired state in both directions, for the owner and for every grants[] entry: the declared set is granted first, then only the surplus is revoked, so narrowing [ALL PRIVILEGES] to [SELECT] actually narrows it — while a failure mid-sequence leaves the principal over-granted for one requeue interval rather than with zero privileges on its own database. The revoke is scoped to this database, and the one-database-one-CR rule (above) is what makes it safe — no other CR can be managing the same grant. Removing an entry from grants[] revokes it on the next apply, and deletion revokes the union of the current list and the previously applied one, so no grant row outlives the CR.

Deletion

deletionPolicy defaults to Retain, and the default is the point.

Note the asymmetry with entry removal, which always revokes and drops: removing a users[] entry is an edit to a live CR — an explicit statement that the principal should not exist — while CR deletion under Retain is treated as possibly accidental, and "touch nothing" has to mean nothing, principals included. To retire the support account before retaining the database, remove the entry first and let it reconcile, then delete the CR.

Dropping a tenant database because a CR was garbage-collected by a GitOps prune, a namespace delete or a bad label selector is an unrecoverable data-loss incident. Offboarding should be an audited human action, so it takes an explicit deletionPolicy: Delete to express it. The zero value resolves to Retain too — a CR stored before the field existed, or one round-tripped by a client that dropped it, is never read as permission to drop data.

Delete only removes what the CR actually applied and exclusively owns:

• status.databaseCreated is a write-ahead record, stamped once the admin connection is open and the schema has passed the adoption check, before the first statement executes. A CR that failed before any SQL ran (invalid spec, reserved owner, ownership conflict, unreachable primary) releases with DatabaseDropSkipped and touches nothing — it must not drop a database something else created under the same name. A CR that failed mid-apply is covered in the other direction: its owner user is recorded and gets dropped rather than surviving as an orphaned privileged account.
• If another live CR on the group still declares the same databaseName, shares the owner principal, or still claims the account through its grants[], its owner or its own users[] ledger, the corresponding drop is skipped with a DatabaseDropSkipped/OwnerUserDropSkipped/UserDropSkipped warning, and an account that has since become a group-level principal is never dropped (OwnerUserReservedSkipped/UserReservedSkipped).
• The deleting CR's own grants[] is a claim too. A recorded owner or users[] principal that has since moved onto this CR's grants[] — the usual way to hand a tenant account over to a shared one — keeps its '%' account, dropped only on the other hosts grants[] never touches. It still loses its rights on this database, like every other vetoed drop. This is the same vetting the apply path applies when a rotated-away name lands on grants[].
• The grants[] revoke uses REVOKE ... IGNORE UNKNOWN USER, so a CR that failed on GrantUserMissing still deletes cleanly instead of wedging on the revoke of a user that never existed.

Under Delete, if the group has no active site — or its primary is fenced by an in-place restore or planned failover, or the connection fails — the drop is deferred, not skipped: the CR emits DatabaseDropDeferred and waits. If the group is gone entirely there is nothing to connect to, so the finalizer is released with a DatabaseCleanupSkipped warning rather than wedging the CR forever. While the finalizer runs, status.phase is Deleting.

Adoption is refused

Bloodraven only manages what it created. Before the reconciler writes anything ahead of the apply — the schema stamp, the owner record, the users[] ledger — it checks everything the apply is about to touch:

• the schema named by databaseName: if it already exists and this CR's status.databaseCreated does not say it created it, the CR fails with reason: DatabasePreExists;
• every owner account, one user@host at a time: an existing account is refused with reason: PreExistingOwnerUser unless one of this CR's records names that user on that host;
• every users[] account, likewise per user@host, refused with reason: PreExistingUser;
• every account named by a ledger entry whose Secret has left users[], to learn whether it exists at all — there is no refusal to make, because the record already attributes the name, but an account found absent was never created and its record is withdrawn rather than trusted into a later DROP USER.

The preflight is not the first thing the reconcile does, and the distinction matters when reading the statement log. Cleanup that an existing record already authorizes runs first: an abandoned rotation target the ledger still holds as pending, but no Secret names any more, is revoked and dropped — and its record cleared — before the preflight, so this reconcile's write-ahead cannot overwrite the only record that could still name it. What the preflight gates is the new record and the apply that follows it: no account or schema this CR has not already recorded is ever written ahead, or touched, before it has been verified.

"One of this CR's records" means ownerUser/ownerHosts, pendingOwnerUser/pendingOwnerHosts, or any appliedUsers[] entry's username/hosts or pendingUsername/pendingHosts, as they stood before the reconcile. Owning acme_app@10.0.0.1 therefore does not make acme_app@10.0.0.2 yours: adding a host on which someone else already has a same-named account is refused before any ALTER USER resets its password.

A users[] username that a live sibling CR on the group already claims — as its owner, in its grants[], or in its own status.appliedUsers ledger — is refused with reason: UserClaimedBySibling before the CR connects to MySQL at all, whether or not the account exists yet.

Because every check runs before the write-ahead stamp, a refusal leaves no record behind, and there is nothing to roll back. A later edit (removing the offending entry, say) cannot turn a refused account into an adopted one, and a later deletionPolicy: Delete never touches the refused schema or account. The owner Secret is never a source of drop candidates either — after a refused rotation it names somebody else's account.

Records can also be withdrawn after the stamp. If MySQL refuses every statement for a principal (a read-only primary, a password-policy rejection), the reconciler withdraws that principal's records down to what the CR already owned before the reconcile, and does the same for every principal whose statements never ran. It also drops any account the preflight has just found absent, even when an earlier reconcile recorded it: a record left behind by a failed status write is cleared by the next failed apply rather than trusted. The withdrawal is part of the same status write that sets the phase to Pending/Failed. A connection error mid-statement is ambiguous — the server may have executed it — so that principal's record stays.

A ledger entry whose Secret has left users[] is withdrawn on the same evidence but on a different schedule: it gets no write-ahead stamp of its own, so nothing later in the reconcile would re-check it, and the withdrawal rides the write-ahead patch instead — before the removal path reads the ledger. Records of accounts that do exist stay, because dropping those is exactly what removing an entry means.

One refused account parks the whole CR: the refusal happens before any of the apply's SQL, so the schema, the owner and every other users[] entry wait too, password rotations included, until the conflict is resolved. The message names the first refused user@host.

To hand an existing schema over to a MysqlDatabase, drop and recreate the CR only if you own the schema — otherwise pick a new databaseName.

Failover

Reconciliation runs against the primary only. Grants replicate, so after a failover the rows are already on the new primary — but a CR must not report Ready against a primary the operator has not spoken to since the flip, because that is reporting something it does not know.

The active site is part of the hash, so a failover invalidates the "nothing changed" short-circuit and forces a re-apply. The controller watches MysqlFailoverGroup and re-enqueues every MysqlDatabase in the namespace whose groupRef matches when status.activeSite changes, or when the group enters or leaves a fenced state. status.activeSite on the CR follows the group.

During an in-place restore or a planned failover the primary is fenced, and reconciliation backs off to Pending rather than erroring — a maintenance window working as designed should not turn every tenant CR red. The fence uses the same classifiers the topology manager freezes on, so "fenced" here always means what it means to the operator.

The same logic applies to unplanned failovers, where there is no fence to observe: the group watch re-enqueues tenants the moment status.activeSite moves, which can be before the promoted site has actually left super_read_only. Read-only and connection errors during an apply are classified as transient — the CR stays Pending with reason: PrimaryUnavailable and retries — rather than latching Failed on every ordinary failover. Failed is reserved for MySQL's verdicts about the CR itself.

RBAC

Two separate pieces, and the distinction is the security story.

The operator gets get;list;watch;update;patch on mysqldatabases and get;update;patch on mysqldatabases/status. Deliberately no create and no delete: it reconciles tenant databases that a caller declared, it never invents them. update is what lets it add and remove the finalizer.

The caller binds a namespaced Role, shipped as an example at config/rbac/mysqldatabase_caller_role.yaml and not installed by default:

rules:
  - apiGroups: [shipstream.io]
    resources: [mysqldatabases]
    verbs: [create, get, list, watch, update, patch, delete]
  - apiGroups: [shipstream.io]
    resources: [mysqldatabases/status]
    verbs: [get]

What is absent is the point:

• no secrets rule, so the caller cannot read the owner password it provisions against, nor the group's operator credential;
• no mysqlfailovergroups rule, so it cannot read the DSN, the credential Secret names, or the topology;
• no MySQL credential of any kind.

status is a subresource, so update on mysqldatabases does not let a caller forge Ready. Only Bloodraven writes status, which is what makes Ready mean "Bloodraven applied this".

If a future change makes a secrets rule necessary in that Role, it has reintroduced the standing root-equivalent credential this API exists to remove.

Writing the owner Secret is a separate concern and deliberately a separate principal — in ShipStream's deployment, External Secrets Operator renders it from OpenBao and the provisioner never touches it.

Quotas

Nothing in Bloodraven bounds how many databases a namespace may create. Use a Kubernetes ResourceQuota on the CRD count:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-databases
  namespace: bloodraven
spec:
  hard:
    count/mysqldatabases.shipstream.io: "200"

Upgrading to per-host write-ahead records

Operators before this change wrote write-ahead records ahead of their adoption checks and kept one host list for both usernames of a rotation. The new operator reads those records as it finds them: it cannot tell a speculative record from a real one, because MySQL keeps no record of which principal created an account.

• Audit before relying on Delete. A CR that is not Ready, or that hit PreExistingOwnerUser / PreExistingUser / DatabasePreExists while an older operator was running, can carry records naming accounts it never created. Compare status.ownerUser, status.pendingOwnerUser, status.ownerHosts and status.appliedUsers[] against the accounts the tenant is known to own. If a record looks wrong, set deletionPolicy: Retain before deleting the CR, or patch the record out of status.
• A rotation in flight across the upgrade has pendingOwnerUser or appliedUsers[].pendingUsername set with no pendingOwnerHosts / pendingHosts. The new operator falls back to the shared host list for that name, which matches the old behaviour, until the rotation reaches Ready.
• Do not downgrade the operator while any MysqlDatabase has pendingOwnerUser or appliedUsers[].pendingUsername set. An older operator ignores pendingOwnerHosts / pendingHosts, so it can leak the pending name's accounts. It can also leave a stale pendingOwnerHosts behind that a later upgrade would apply to a different rotation target. Let rotations reach Ready first.

Relationship to spec.credentials

MysqlDatabase does not replace spec.credentials on MysqlFailoverGroup. The five group-level roles (operator, app, readonly, monitor, backup) stay exactly as they are; this is per-tenant databases, a different granularity.

Both paths connect to the same primary through the same function — openAdminConnection in internal/controller/credentials.go — and manage disjoint principals, enforced in both directions: the tenant reconciler refuses a Secret that names a group-level principal (OwnerUserReserved, UserReserved), and group credential reconciliation fails closed before any SQL when a role username is claimed by a live MysqlDatabase — as its owner or through its status.appliedUsers ledger. That function is the only place in Bloodraven that assembles MySQL admin credentials, and it has exactly these two callers. A third would be a design decision, not a refactor.

Known gaps

• grants[] principals and spec.credentials roles are always '%'-hosted. hosts scopes the owner and users[], the accounts this CRD creates; the group-level roles in credentials.go stay on '%'.
• Simultaneous first creation of one username by two CRs is arbitrated by the ledger, not by a lock. Each CR checks its siblings' owner, grants[] and status.appliedUsers claims before stamping its own write-ahead record (UserClaimedBySibling), so the first persisted claim wins even when the account does not exist yet — but the check reads the informer cache, so two CRs whose Secrets resolve to the same not-yet-existing username and that reconcile within the same cache-propagation window can both record it and end up sharing one account with the last writer's password. That needs two namespaces to hold colliding credentials; derive users[] usernames from something tenant-unique (the platform derives from the instance id, as it does for the owner) and it cannot happen by accident.
• Out-of-band drift is not self-healed. A database or grant dropped directly in MySQL is not detected: the hash short-circuit means an unchanged Ready CR issues no statements, which is the deliberate trade for not hammering the primary. Force a re-apply by rotating the owner Secret or editing the spec.
• The adoption check and the CREATE are not atomic. The existence checks run, then one status write, then CREATE DATABASE IF NOT EXISTS / CREATE USER IF NOT EXISTS. A schema or account that somebody else creates with exactly that name, on exactly that host, inside that window is adopted: its password is reset and a later Delete drops it. Neither the CR spec nor its Secrets can trigger this; it needs a concurrent CREATE by another MySQL principal. A planned follow-up closes it with a plain CREATE for accounts found absent, plus MySQL account attributes (CREATE USER … ATTRIBUTE, MySQL 8.0.21+) to tag the accounts a CR created.
• A record can outlive a crash or a failed status write. This happens between the write-ahead stamp and the first statement, or when the status write carrying a withdrawal fails. Such a record names only accounts that were verified absent (or already owned) in that reconcile. The next reconcile re-checks them and withdraws any that are still absent, so exploiting the record needs a foreign CREATE of exactly that user@host before that next reconcile's preflight.
• Handing a CR-created account to this CR's own grants[] releases custody. Moving an owner or users[] username into grants[] keeps the account, which is the point, but retires its record. A later removal of the grants[] entry revokes it, and CR deletion revokes it, but neither drops it: grants[] principals are never dropped. If the move involved a pending rotation target and the reconcile fails before Ready, the pending record is overwritten before status.appliedGrants records the name. A grants[] removal before any successful apply then does not revoke it either. Move names out of the owner or users[] only through an apply that reaches Ready, and drop the account by hand if it should not outlive the grant.
• Host matching is case-sensitive in attribution. MySQL compares account hosts case-insensitively; the status records compare exact strings. Changing a host's case wedges the CR on PreExistingOwnerUser / PreExistingUser for an account it owns. It never adopts a foreign account. Revert the case, or remove and re-add the host.
• sql_mode=NO_BACKSLASH_ESCAPES is unsupported on the target group: password escaping assumes MySQL's default backslash semantics, so a password containing \ would be stored literally under that mode.