Tenant Databases

MysqlDatabase is a namespaced CRD that declares one database on a MysqlFailoverGroup, the user that owns it, and grants for principals that already exist.

It exists to answer one question: how does a caller provision a tenant database without being handed MySQL admin?

Why this exists

Before this CRD, the only way to create a per-tenant database on a Bloodraven-managed group was to connect to MySQL as an administrator. That meant the provisioner held the group's operator credential — GRANT ALL PRIVILEGES ON *.* WITH GRANT OPTION, plus MYSQL_ROOT_PASSWORD. Standing, long-lived, and effectively root on every tenant database in every group it provisioned into.

The obvious alternative — leasing a short-lived credential from a secrets engine such as OpenBao's database engine — does not work here, for reasons that are structural rather than incidental:

1. The secrets engine dials the database. It opens the connection itself and fails closed when it cannot reach the host. A MySQL instance reachable only on a private network cannot be configured as a target at all.
2. Bloodraven is already the credential authority. reconcileRole issues CREATE USER IF NOT EXISTS … IDENTIFIED BY and ALTER USER … IDENTIFIED BY from a referenced Secret's bytes. The Secret is desired state for the MySQL user, not a credential to a user that already exists. An external engine rotating the same principals would be a second writer with no arbitration.
3. The operator credential cannot be leased even in principle. It carries MYSQL_ROOT_PASSWORD — the value MySQL is initialized with — and the operator falls back to root with it when the operator user does not yet exist. A credential that must be known before the database exists cannot be minted by something that connects to the database.

So the component that holds MySQL admin is Bloodraven, because it already must. MysqlDatabase is the way to ask Bloodraven to create a tenant database without being given the keys to do it yourself. The caller's MySQL credential is replaced by Kubernetes RBAC on a namespaced CRD.

The security property: a caller can provision a tenant database while holding no MySQL credential and no Secret access.

Example

apiVersion: shipstream.io/v1alpha1
kind: MysqlDatabase
metadata:
  name: tenant-acme
  namespace: bloodraven
spec:
  groupRef:
    name: main                    # MysqlFailoverGroup in the same namespace
  databaseName: acme_wms          # ^[A-Za-z0-9_]{1,64}$
  characterSet: utf8mb4           # default
  collation: utf8mb4_unicode_ci   # default

  owner:
    # Secret with keys `username` and `password`, same contract as
    # spec.credentials.*Secret on MysqlFailoverGroup.
    secretName: acme-mysql-owner
    privileges: [ALL PRIVILEGES]  # ON acme_wms.* only, never WITH GRANT OPTION

  # Principals that must ALREADY exist. Grant-only: never CREATE USER.
  grants:
    - username: maester
      privileges: [SELECT, DELETE]

  deletionPolicy: Retain          # default

The owner's password arrives the way every other Bloodraven credential does: you write a Secret, Bloodraven applies it. Bloodraven never generates, returns, or stores a password — if it generated one it would need somewhere to put it, which reintroduces the custody problem this CRD exists to remove.

Status

status:
  phase: Ready                    # Pending | Creating | Ready | Failed | Deleting
  observedGeneration: 3
  databaseCreated: true
  ownerUser: acme_app             # echoed from the Secret; NOT the password
  appliedGrants: [acme_app, maester]
  activeSite: dc1
  lastAppliedHash: a1b2c3d4e5f6
  message: database acme_wms ready on site dc1
  conditions:
    - type: Ready
      status: "True"
      reason: DatabaseReconciled

observedGeneration and the Ready condition are the contract. They are how a provisioner reports provisioning state back to its own callers without opening a MySQL connection — treat them as API surface, not diagnostics.

status never carries credential material. ownerUser is a username. lastAppliedHash fingerprints the Secret's revision (UID + resourceVersion), never a digest of its bytes — status is caller-readable, and a content digest would let a status reader offline-check password guesses.

Phases

A MysqlDatabase applied before its MysqlFailoverGroup goes Pending, not Failed — that ordering is normal, not a fault.

Privileges

privileges is an allowlist, not a passthrough string:

ALL PRIVILEGES, SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER,
INDEX, REFERENCES, LOCK TABLES, SHOW VIEW, TRIGGER, EVENT, EXECUTE

Anything outside the list is rejected by the API server (the field is an enum) and rejected again in Go before any SQL is rendered. ALL PRIVILEGES cannot be combined with other entries.

GRANT OPTION is not on the list and never will be. Bloodraven never emits WITH GRANT OPTION from this CRD — an owner that can grant is an owner that can escape its own database. ALL PRIVILEGES here means GRANT ALL PRIVILEGES ON \acme_wms`.*`, which confers nothing outside that schema.

Identifiers are validated before they reach SQL rendering, not merely escaped: databaseName, characterSet and collation must match ^[A-Za-z0-9_]{1,64}$, and usernames ^[A-Za-z0-9_][A-Za-z0-9_.$-]{0,31}$. Rejection is the contract; escaping is the belt on top of the braces. MySQL's own schemas — mysql, sys, information_schema, performance_schema, case-insensitively — are rejected outright as databaseName: a tenant CR must never hold privileges on the grant tables.

grants[] is grant-only

A MysqlDatabase can bring exactly one MySQL principal into existence: its own owner, and only because you placed that user's password in a Secret first.

Every grants[] entry names a user that must already exist. The reconciler verifies it (SELECT 1 FROM mysql.user WHERE user = ? AND host = '%') and fails the CR with reason: GrantUserMissing if it does not. It never creates the user.

Without that split, "create a database" would imply "create arbitrary MySQL users", and this CRD would be a privilege-escalation primitive rather than a narrowing of one.

If you hit GrantUserMissing, the fix is to create the principal as a group-level concern — it is shared across tenants, so it does not belong to any one MysqlDatabase. The CR re-checks on its own and goes Ready once the user exists; no CR edit is needed.

The owner cannot be a group-level principal

Bloodraven applies ALTER USER … IDENTIFIED BY from the owner Secret's bytes. That is correct desired-state behaviour for a user this CRD owns — it is what makes rotation a Secret write. Pointed at a Secret whose username is root, the operator user, or any other account named by spec.credentials, the same statement would instead reset a privileged account's password to whatever the Secret's author chose.

So the reconciler refuses. If spec.owner.secretName resolves to a username belonging to the group — root, replicator, MySQL's built-in system accounts (mysql.sys, mysql.session, mysql.infoschema), or any spec.credentials principal — the CR fails with reason: OwnerUserReserved and no statement is built, let alone executed. The check fails closed: if a group credential Secret cannot be read for any reason other than not existing, the reconcile errors and retries rather than proceeding with a partial reserved set.

This matters most if you deviate from the recommended split. In the intended deployment the caller has no secrets verbs at all (the Secret is rendered by an external controller), so it cannot name anything. If your provisioner does write its own owner Secrets, this check is what stops "provision a tenant database" from becoming "reset the operator's password".

One database, one CR

Two MysqlDatabase CRs on the same group must not claim the same databaseName or the same owner principal: desired-state ALTER USER means shared owners take turns resetting each other's password, and deletionPolicy: Delete on one duplicate would drop the other's live data. The reconciler refuses: the older CR wins, the newer fails with reason: DatabaseNameConflict or OwnerConflict before any SQL is rendered.

spec.databaseName and spec.groupRef are both immutable, enforced by the API server. MySQL has no schema rename and the reconciler has no way to move a schema between groups; either edit would apply fresh state and orphan what already exists, and a later deletionPolicy: Delete would aim cleanup at the wrong objects. Renaming or re-grouping a tenant database is a migration, not a spec edit.

Rotation

Rotating the owner password is a Secret write and nothing else:

kubectl -n bloodraven patch secret acme-mysql-owner \
  --type merge -p '{"stringData":{"password":"new-password"}}'

The reconciler watches the referenced Secret, so it applies ALTER USER on the next reconcile. Re-applying an unchanged CR issues zero MySQL statements — the reconciler compares a fingerprint of the spec, the Secret's revision, the active site and the group's identity against status.lastAppliedHash — which is what keeps a tenant-dense cluster from hammering the primary.

Rotating the username in the Secret is also just a Secret write, and it revokes what it replaces. Rotation is create-before-drop: the reconciler creates the new owner, grants it, applies the database, and only then drops the previous owner account (OwnerUserRotated event fires after the full handover). A failure mid-handover leaves both accounts alive and retried — never a window in which the tenant has no owner. A rotation performed because a credential leaked actually revokes the leaked credential — status.ownerUser keeps the old name until the old account is really gone, so a failure mid-rotation retries rather than leaving a shadow account.

Privileges are desired state in both directions, for the owner and for every grants[] entry: the declared set is granted first, then only the surplus is revoked, so narrowing [ALL PRIVILEGES] to [SELECT] actually narrows it — while a failure mid-sequence leaves the principal over-granted for one requeue interval rather than with zero privileges on its own database. The revoke is scoped to this database, and the one-database-one-CR rule (above) is what makes it safe — no other CR can be managing the same grant. Removing an entry from grants[] revokes it on the next apply, and deletion revokes the union of the current list and the previously applied one, so no grant row outlives the CR.

Deletion

deletionPolicy defaults to Retain, and the default is the point.

Dropping a tenant database because a CR was garbage-collected by a GitOps prune, a namespace delete or a bad label selector is an unrecoverable data-loss incident. Offboarding should be an audited human action, so it takes an explicit deletionPolicy: Delete to express it. The zero value resolves to Retain too — a CR stored before the field existed, or one round-tripped by a client that dropped it, is never read as permission to drop data.

Delete only removes what the CR actually applied and exclusively owns:

• status.databaseCreated is a write-ahead record, stamped once the admin connection is open and before the first statement executes. A CR that failed before any SQL ran (invalid spec, reserved owner, ownership conflict, unreachable primary) releases with DatabaseDropSkipped and touches nothing — it must not drop a database something else created under the same name. A CR that failed mid-apply is covered in the other direction: its owner user is recorded and gets dropped rather than surviving as an orphaned privileged account.
• If another live CR on the group still declares the same databaseName, shares the owner principal, or still lists the owner in its grants[], the corresponding drop is skipped with a DatabaseDropSkipped/OwnerUserDropSkipped warning, and an owner that has since become a group-level principal is never dropped (OwnerUserReservedSkipped).
• The grants[] revoke uses REVOKE ... IGNORE UNKNOWN USER, so a CR that failed on GrantUserMissing still deletes cleanly instead of wedging on the revoke of a user that never existed.

Under Delete, if the group has no active site — or its primary is fenced by an in-place restore or planned failover, or the connection fails — the drop is deferred, not skipped: the CR emits DatabaseDropDeferred and waits. If the group is gone entirely there is nothing to connect to, so the finalizer is released with a DatabaseCleanupSkipped warning rather than wedging the CR forever. While the finalizer runs, status.phase is Deleting.

Adoption is refused

Bloodraven only manages what it created. If the schema named by databaseName already exists on the group and this CR's status.databaseCreated does not say it created it, the CR fails with reason: DatabasePreExists and runs no SQL. Likewise, if the owner username already exists in MySQL and is not recorded as this CR's owner, the CR fails with reason: PreExistingOwnerUser before any ALTER USER resets a foreign account's password. Neither refusal authorizes a later deletionPolicy: Delete to touch the foreign schema or account. To hand an existing schema over to a MysqlDatabase, drop and recreate the CR only if you own the schema — otherwise pick a new databaseName.

Failover

Reconciliation runs against the primary only. Grants replicate, so after a failover the rows are already on the new primary — but a CR must not report Ready against a primary the operator has not spoken to since the flip, because that is reporting something it does not know.

The active site is part of the hash, so a failover invalidates the "nothing changed" short-circuit and forces a re-apply. The controller watches MysqlFailoverGroup and re-enqueues every MysqlDatabase in the namespace whose groupRef matches when status.activeSite changes, or when the group enters or leaves a fenced state. status.activeSite on the CR follows the group.

During an in-place restore or a planned failover the primary is fenced, and reconciliation backs off to Pending rather than erroring — a maintenance window working as designed should not turn every tenant CR red. The fence uses the same classifiers the topology manager freezes on, so "fenced" here always means what it means to the operator.

The same logic applies to unplanned failovers, where there is no fence to observe: the group watch re-enqueues tenants the moment status.activeSite moves, which can be before the promoted site has actually left super_read_only. Read-only and connection errors during an apply are classified as transient — the CR stays Pending with reason: PrimaryUnavailable and retries — rather than latching Failed on every ordinary failover. Failed is reserved for MySQL's verdicts about the CR itself.

RBAC

Two separate pieces, and the distinction is the security story.

The operator gets get;list;watch;update;patch on mysqldatabases and get;update;patch on mysqldatabases/status. Deliberately no create and no delete: it reconciles tenant databases that a caller declared, it never invents them. update is what lets it add and remove the finalizer.

The caller binds a namespaced Role, shipped as an example at config/rbac/mysqldatabase_caller_role.yaml and not installed by default:

rules:
  - apiGroups: [shipstream.io]
    resources: [mysqldatabases]
    verbs: [create, get, list, watch, update, patch, delete]
  - apiGroups: [shipstream.io]
    resources: [mysqldatabases/status]
    verbs: [get]

What is absent is the point:

• no secrets rule, so the caller cannot read the owner password it provisions against, nor the group's operator credential;
• no mysqlfailovergroups rule, so it cannot read the DSN, the credential Secret names, or the topology;
• no MySQL credential of any kind.

status is a subresource, so update on mysqldatabases does not let a caller forge Ready. Only Bloodraven writes status, which is what makes Ready mean "Bloodraven applied this".

If a future change makes a secrets rule necessary in that Role, it has reintroduced the standing root-equivalent credential this API exists to remove.

Writing the owner Secret is a separate concern and deliberately a separate principal — in ShipStream's deployment, External Secrets Operator renders it from OpenBao and the provisioner never touches it.

Quotas

Nothing in Bloodraven bounds how many databases a namespace may create. Use a Kubernetes ResourceQuota on the CRD count:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-databases
  namespace: bloodraven
spec:
  hard:
    count/mysqldatabases.shipstream.io: "200"

Relationship to spec.credentials

MysqlDatabase does not replace spec.credentials on MysqlFailoverGroup. The five group-level roles (operator, app, readonly, monitor, backup) stay exactly as they are; this is per-tenant databases, a different granularity.

Both paths connect to the same primary through the same function — openAdminConnection in internal/controller/credentials.go — and manage disjoint principals, enforced in both directions: the tenant reconciler refuses a Secret that names a group-level principal (OwnerUserReserved), and group credential reconciliation fails closed before any SQL when a role username is claimed by a live MysqlDatabase. That function is the only place in Bloodraven that assembles MySQL admin credentials, and it has exactly these two callers. A third would be a design decision, not a refactor.

Known gaps

• Host scoping is '%', consistent with every account in credentials.go. Scoping tenant owners to a pod CIDR is worth doing, but as one change across both paths rather than here alone.
• Out-of-band drift is not self-healed. A database or grant dropped directly in MySQL is not detected: the hash short-circuit means an unchanged Ready CR issues no statements, which is the deliberate trade for not hammering the primary. Force a re-apply by rotating the owner Secret or editing the spec.
• sql_mode=NO_BACKSLASH_ESCAPES is unsupported on the target group: password escaping assumes MySQL's default backslash semantics, so a password containing \ would be stored literally under that mode.