Security model

Operator vs. platform responsibilities for pod hardening, network isolation, resource bounds, and secrets.

This document records the Firebolt Operator’s threat-model boundary: what the Firebolt Operator hardens by default versus what is the cluster-platform team’s responsibility.

What the Firebolt Operator enforces

The Firebolt Operator stamps a hardened security posture on the workloads it renders. How much you can change that posture depends on the CRD surface. Engine pods expose a wide pod-template merge layer through FireboltEngineClass and FireboltEngine. Gateway and metadata primary containers keep operator-owned hardening and only let you set image and resources on the primary container.

Per-field allowlists for each template surface live in api/v1alpha1/operatorauthority.go (PodTemplateRules per component) and in the CRD reference pages linked in each subsection below.

Engine

When neither a referenced FireboltEngineClass nor the engine’s own spec.template sets a container securityContext, the Firebolt Operator applies these defaults on the engine container:

Field	Default
UID / GID	3473 / 3473
runAsNonRoot	true
Capabilities	drop ALL
allowPrivilegeEscalation	false
Read-only root filesystem	true (unix domain socket via emptyDir at /run/firebolt; status file and engine data on the PVC at /var/lib/firebolt)
Seccomp	not set

You can replace that entire container securityContext wholesale through FireboltEngineClass.spec.template or FireboltEngine.spec.template.spec.containers[name=="engine"].securityContext. The engine template wins over the class. There is no merge or floor. A template value replaces the operator default completely, including the ability to weaken hardening.

Pod-level spec.template.spec.securityContext is also user-settable. The Firebolt Operator only stamps fsGroup (3473) and fsGroupChangePolicy: OnRootMismatch when you leave those fields unset. Sidecar containers and init containers pass through from your template without operator hardening.

Writable directories and arbitrary UIDs

The engine image owns two writable directories: the data dir (/var/lib/firebolt, the PVC mount) and the unix socket dir (/run/firebolt, an emptyDir). Both are group-owned by root (GID 0) with mode 2770 (g=u plus the setgid bit), not the broad 777 the image used previously. The read-only application payload at /opt/firebolt stays firebolt:firebolt 0755.

This follows the OpenShift “arbitrary UID” convention: a platform that assigns a random non-namespace UID still places the process in group 0, so the GID-0 group ownership keeps both directories readable and writable, while other gets nothing. The setgid bit makes files the engine creates at runtime inherit group 0. The engine runs with umask 0007 so it does not strip those group bits off new files. Under the default UID/GID 3473/3473 the user owns the directories outright, so the same layout works without OpenShift.

Mounted volumes override the image’s directory permissions. When a platform assigns an arbitrary UID, give the engine a writable data dir either through the stamped fsGroup (which chowns the PVC to the group) or by mounting volumes already group-0-writable, and avoid pinning a runAsUser that is not a member of group 0.

Validating webhooks on FireboltEngine and FireboltEngineClass (and the engine reconciler when webhooks are off) reject operator-owned paths on engine templates: command, args, ports, probes, reserved env keys, and firebolt.io/* labels. They do not reject a weakened securityContext. See the FireboltEngineClass CRD reference and FireboltEngine CRD reference.

Gateway (Envoy)

The Envoy primary container is operator-rendered end to end. The validating webhook rejects user input on its securityContext. You may only set image and resources on spec.gateway.template.spec.containers[name=="envoy"]. The Firebolt Operator stamps:

Field	Value
UID	101
runAsNonRoot	true
Capabilities	drop ALL
allowPrivilegeEscalation	false
Read-only root filesystem	true (scratch via emptyDir at /tmp)
Seccomp	not set on the container

Pod-level fields on spec.gateway.template (node selector, tolerations, securityContext, sidecars, init containers) pass through when allowed by the template rules. See the FireboltInstance CRD reference.

Metadata (Pensieve)

The metadata primary container follows the same pattern as the gateway. Operator-stamped hardening applies to containers[name=="metadata"]. You may set only image and resources there:

Field	Value
UID	1111 (pinned to the image’s dedicated-pensieve user)
runAsNonRoot	true
Capabilities	drop ALL
allowPrivilegeEscalation	false
Read-only root filesystem	true (scratch via emptyDir at /tmp)
Seccomp	not set on the container

At the pod level, the Firebolt Operator floors any user-supplied PodSecurityContext to runAsNonRoot: true with UID/GID 1111 and sets seccompProfile: RuntimeDefault when you do not supply one. A template cannot run the metadata pod under a different user without failing image ownership checks.

PostgreSQL (internal)

Internal PostgreSQL has no user template surface. The Firebolt Operator stamps hardening on every reconcile:

Level	Posture
Pod	UID/GID 70, runAsNonRoot: true, seccompProfile: RuntimeDefault
Container	UID 70, runAsNonRoot: true, drop ALL capabilities, allowPrivilegeEscalation: false, read-only root filesystem (PGDATA on the PVC, runtime sockets on emptyDir)

The Firebolt Operator’s own manager RBAC (rendered by the chart from the canonical config/rbac/role.yaml) is the minimal set of verbs needed to manage the three CRDs and their generated resources. The Firebolt Operator does not request * on any namespaced verb and does not request nodes, clusterrolebindings, or any other cluster-scope mutation.

The chart renders the manager rules in one of two shapes, picked at install time by watchNamespaces:

• Empty list (default): one cluster-scoped ClusterRole and one ClusterRoleBinding. The manager cache spans every namespace.
• Non-empty list, e.g. {tenant-a, tenant-b}: a Role plus RoleBinding pair in each listed namespace, no ClusterRole. The manager cache spans only those namespaces. Use this posture when multi-tenant compliance constraints bound the operator’s blast radius.

The default FireboltInstance.spec.metricScrapeMode=PodIP reaches engine metrics through pod IPs and does not need pods/proxy: get, so the chart’s manager RBAC does not include that verb. Setting metricScrapeMode=ApiserverProxy on any FireboltInstance requires rbac.apiserverProxyGrant=true on the operator chart, which renders a dedicated ClusterRole (or per-namespace Role when watchNamespaces is set) granting only that one verb. Without the toggle, the metric scrape surfaces as a 403 from the apiserver.

Resource maxima on the engine container’s resources block (set under FireboltEngine.spec.template.spec.containers[name=="engine"] or inherited from a referenced FireboltEngineClass) are enforced by the validating webhook (see “Resource bounds” below). The bounds protect a namespace from accidentally admitting an engine whose requests would starve sibling workloads at scheduling time.

Authentication and transport encryption

Both are opt-in and Instance-wide. Neither is configured per Engine: every Engine in an Instance runs the same authentication configuration, because each one both issues and validates JWTs and a token minted by one must validate on another. See architecture for the provisioning model.

Authentication

spec.auth layers two authorization servers:

• The local user store, always present when auth is enabled. It holds the Instance admin plus any users created through SQL, and backs password login through the Engine’s embedded authorization server.
• OIDC bearer validation through spec.auth.oidc, optional. Engines accept JWTs issued by the trusted providers you configure. Engines validate those tokens; they never run a login flow themselves, so no client secret for your identity provider is ever held by the Instance.

The Firebolt Operator never generates the admin password. It is supplied through a Secret you create, referenced by spec.auth.local.admin.password. A credential the operator invented and you never saw would not be usable, so the reconcile preflights the Secret’s existence and reports AuthReady=False/AdminSecretMissing rather than inventing one. The password is mounted into each Engine as a file and referenced by path, so no plaintext credential appears in a rendered config.

The shared JWT signing keypair is provisioned through cert-manager against an Issuer or ClusterIssuer you supply. Signing keys are cert-manager-only — unlike TLS, there is no bring-your-own secretRef path — and the private key is mounted by path, never inlined.

Engine and Gateway TLS

spec.tls.engine terminates TLS on each Engine’s query listener; spec.tls.gateway terminates client-facing TLS on the Gateway. The Gateway listener accepts either a cert-manager issuer or a secretRef to a certificate you manage yourself. The Engine listener is cert-manager-only: the operator issues a certificate per Engine generation so the SANs cover that generation’s pod hostnames, which a static Secret cannot do.

Two properties are worth knowing before you enable them:

• TLS replaces plaintext on the same port; it does not add a second listener. An Engine serving TLS is not reachable in plaintext. Enabling or disabling Engine TLS on a fleet already serving traffic therefore has a brief Gateway-to-Engine outage window — see the transient-outage note in architecture.
• Tightening the Gateway’s posture is fail-closed. Going from plaintext to TLS, from one-way to mutual TLS, or swapping the client CA withholds the client-facing listener and drains the old, looser pods before any secure pod serves. The client port refuses connections for that window rather than briefly accepting clients the new posture is meant to reject. Liveness and readiness probes ride the always-plaintext metrics port, so pods stay alive through it.

Adding clientCASecretRef upgrades the Gateway listener to mutual TLS, verifying client certificates against that CA. Loosening transitions carry no fail-open risk and are not staged.

Two things about mutual TLS are worth stating plainly, because the operator cannot enforce them for you:

• Verification is CA-scoped, not identity-scoped. Any certificate the configured CA signed is accepted. Point clientCASecretRef at a CA dedicated to this purpose, not at a broad organizational CA that already signs certificates for unrelated workloads.
• Prefer a separate issuer per purpose. The signing keypair, the Engine listener, the Gateway listener, and the client CA should not all chain to one issuer. The Gateway verifies Engines against the Engine issuer’s CA, so anything else able to obtain a certificate from that same issuer for a matching name is, to the Gateway, indistinguishable from an Engine.

crlSecretRef on either listener supplies a revocation list the Gateway checks alongside the trust anchor, so a compromised certificate can be rejected before it expires. Certificate lifetimes are bounded by default, which limits how long a leaked key stays usable but does not end it — a CRL is the only way to cut it short.

What the Firebolt Operator does not enforce

Pod-template authors can mount other Secrets in the namespace

A FireboltEngine, FireboltEngineClass, or FireboltInstance pod template may declare volumes and sidecar containers, and a volume may reference any Secret in that namespace. Treat edit rights on those templates as equivalent to read rights on the namespace’s Secrets, and scope RBAC accordingly: someone who can edit a template can mount a Secret and read it from a sidecar whose image and command they control, without any get secrets permission.

The operator’s own auth and TLS Secrets are excluded from this. The protected set is Instance-wide — every template is screened against every operator-managed Secret in the Instance, not only the ones its own pod mounts, so a Gateway template cannot reach a signing key and one Engine cannot reach another’s serving key. The checks match the Secret being referenced rather than the name the author gave it, so renaming evades nothing:

• A volume whose source reaches one — including through a projected source — is dropped from the rendered pod and reported as a Warning Event on the resource. It is reported rather than refused on purpose: the kubelet re-syncs a Secret volume for the life of the pod, so a volume that named a not-yet-existing Secret starts yielding real key material once the operator creates it. The remedy is to replace the pod, and refusing to render would freeze the Engine on the very generation holding the exposed volume.
• A container that reads one into its environment, through env[].valueFrom.secretKeyRef or envFrom[].secretRef, is refused outright. An env reference resolves once when the pod starts and is never re-synced, so declining to render is a complete remedy.

Both apply to the primary container, sidecars, and init containers, and to a referenced FireboltEngineClass template as well as the resource’s own.

Some of these Secrets are matched by name shape rather than by a name the Firebolt Operator has already recorded, because a template is validated before the Secret exists. Two name patterns are reserved, whether or not the Secret is provisioned yet:

• <instance>-auth-signing and <instance>-auth-signing-*, the JWT signing keypairs.
• <engine>-g<generation>-engine-tls, the per-generation Engine serving certificates.

These patterns match across every Instance and Engine in the namespace, not only the resource being validated. A template on one Instance cannot reach a sibling Instance’s signing key, and a template on one Engine cannot reach another Engine’s serving certificate. Protection is a property of the Secret, not of which resource the template belongs to.

Do not create your own Secrets under these names. A reference to one is treated as a reference to operator key material and is dropped or refused.

Secrets you name yourself are the exception, because the Firebolt Operator cannot recognize them by shape. The admin password Secret is screened by name against the Instance being validated, so a template on one Instance can still reference a sibling Instance’s admin password Secret if the two use different names for it. Treat template edit rights as read access to every Secret in the namespace, as above, and put unrelated Instances in separate namespaces when that matters.

Network isolation between pods

The Firebolt Operator emits no NetworkPolicy objects. All pod-to-pod and pod-to-external traffic is governed by whatever the cluster’s CNI and NetworkPolicy controller already enforce. In a default Kubernetes install with no NetworkPolicy controller installed, every pod can reach every other pod on every port.

This is a deliberate scoping decision: NetworkPolicy semantics depend on the CNI plugin (Calico, Cilium, Antrea, etc.), the cluster’s default-allow-vs-default-deny posture, and the Firebolt Operator-vs-platform ownership boundary for security primitives. Encoding any of those assumptions into Firebolt Operator-emitted NetworkPolicies would either be a no-op (no controller installed) or actively wrong for the deployment target.

Platform teams should apply NetworkPolicies covering at least the allowed flows below. Recommended selectors:

Pod kind	Selector
Engine pods	firebolt.io/engine exists (matches any generation of any engine in the namespace)
Gateway / Metadata / PostgreSQL	firebolt.io/component={gateway,metadata,postgres}
Instance scoping	firebolt.io/instance=<instance-name> (present on instance-level workloads only. Engines carry firebolt.io/engine instead, which is unique per engine)

Allowed flows

From	To	Port	Purpose
External clients	Gateway	8080	Query traffic (HTTP)
Gateway	Engine pods	3473	Query forwarding
Engine pods	Metadata	7000	Metadata gRPC
Metadata	PostgreSQL	5432	Metadata catalog reads/writes
Engine pods	External object store	443 / 80	Managed-storage reads/writes (S3, GCS, etc.)
Prometheus	Engine / Gateway / Operator	9090 / 9090 / 8443	Metrics scraping
kube-apiserver	Operator webhook	9443	Admission control

Engine-to-engine, engine-to-PostgreSQL, gateway-to-metadata, and gateway-to-PostgreSQL are not required by any Firebolt Operator-managed control flow and should be denied.

Example baseline NetworkPolicy

The snippet below denies all ingress and egress by default in the instance’s namespace, then re-allows the flows above. It assumes the gateway’s external clients live in a firebolt-clients namespace. The selector should be adjusted to match the actual client topology.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gateway-ingress
spec:
  podSelector:
    matchLabels:
      firebolt.io/component: gateway
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: firebolt-clients
      ports:
        - port: 8080
          protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: engine-from-gateway
spec:
  podSelector:
    matchExpressions:
      - key: firebolt.io/engine
        operator: Exists
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              firebolt.io/component: gateway
      ports:
        - port: 3473
          protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: metadata-from-engine
spec:
  podSelector:
    matchLabels:
      firebolt.io/component: metadata
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: firebolt.io/engine
                operator: Exists
      ports:
        - port: 7000
          protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-from-metadata
spec:
  podSelector:
    matchLabels:
      firebolt.io/component: postgres
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              firebolt.io/component: metadata
      ports:
        - port: 5432
          protocol: TCP

Add egress allow-rules in the same shape. The Firebolt Operator does not generate them.

Namespace-level resource ceilings

A ResourceQuota capping the aggregate of requests.cpu / requests.memory / pod count / PVC size across all engines in a namespace is a platform concern. The Firebolt Operator enforces per-engine upper bounds in admission (see below) but does not emit a ResourceQuota. The per-namespace budget is a deployment-target decision (test cluster vs. multi-tenant production).

Image provenance and supply-chain attestation

The Firebolt Operator pulls whatever image the user supplies via FireboltEngine.spec.template.spec.containers[engine].image, the referenced FireboltEngineClass.spec.template.spec.containers[engine].image, or the embedded defaults shipped with the Firebolt Operator binary (merged in that order, top wins). The Firebolt Operator does not validate signatures, attestations, or SBOMs. Use a cluster-level admission controller (Kyverno, Sigstore Policy Controller, etc.) if image-policy enforcement is required.

Resource bounds

The FireboltEngine validating webhook rejects engine-container resources entries above Firebolt Operator-configured maxima. The gate resolves the effective container the same way the reconciler will: the engine’s own spec.template.spec.containers[name=="engine"].resources wins wholesale when set, otherwise the referenced FireboltEngineClass’s container resources fill in. Both sources are checked, so a class with oversized requests cannot escape admission by being referenced from an engine that omits its own resources. The error message names the source class when the offending value came from class so the user knows which side to edit.

This is a defense-in-depth control against accidental over-provisioning. A typoed 100Gi instead of 10Gi is caught at admission rather than at scheduling time when it would silently exhaust namespace capacity and block other engines.

The maxima are configurable at Firebolt Operator install time. Defaults are sized for typical production deployments. Override via Helm values when running larger or smaller engines.

Secrets handling

The Firebolt Operator generates one Secret: the internal PostgreSQL credentials (<instance>-metadata-postgres-creds). The password is generated at first reconcile, persisted to a Kubernetes Secret with owner reference to the FireboltInstance, and never re-rotated by the Firebolt Operator. The metadata and PostgreSQL Deployments load it via envFrom.

User-supplied credentials (external PostgreSQL, external object store) are referenced by name on the FireboltInstance / FireboltEngine spec and resolved at reconcile time. The Firebolt Operator never reads or materializes a user-supplied secret’s value into its own status, events, or logs.

When auth or TLS is enabled, these additional Secrets exist:

Secret	Origin	Held by	Rotation
Admin password	You create it; referenced by spec.auth.local.admin.password	Every Engine, mounted read-only by path	Yours to rotate. Editing the Secret rolls the fleet onto the new value.
JWT signing keys	cert-manager, one per key generation	Every Engine, mounted read-only by path	Operator-coordinated when rotationInterval is set; otherwise a single permanent key.
Engine TLS certificate	cert-manager (no secretRef for this listener)	Engine pods, plus its CA in the Gateway’s trust bundle	cert-manager renewal on a bounded lifetime (90 days by default, certManager.duration). A reissue rolls a fresh Engine generation.
Gateway TLS certificate	cert-manager, or your own via secretRef	Gateway pods	cert-manager renewal on the same bounded default. A reissue rolls the Gateway Deployment.
Gateway client CA	You create it; referenced by clientCASecretRef	Gateway pods (public certificate only)	Yours. A swap is treated as a tightening transition.

The Firebolt Operator reads a signing key’s public-key fingerprint to detect key replacement, and a Secret’s resourceVersion to detect an in-place password change. Neither the private key nor the password value is ever written to status, events, or logs.

Signing key provenance

Signing key Secrets have deterministic names, and cert-manager reuses a private key that already exists in a Certificate’s target Secret. Without a check, anyone holding create secrets in the namespace could pre-create that name with key material of their choosing and have it become the Instance’s JWT signing key.

The Firebolt Operator refuses to sign with a Secret it cannot attribute to its own Certificate. It compares creation timestamps: cert-manager cannot write a target Secret before the Certificate exists, so a signing Secret older than its Certificate was put there by something else. The check uses creationTimestamp rather than cert-manager’s cert-manager.io/certificate-name annotation, because the annotation is ordinary metadata that whoever creates the Secret can set to any value, while the API server assigns creationTimestamp itself.

This stops a principal who can create Secrets. It does not stop one who can also create Certificate objects: that principal creates the Certificate first and plants the Secret after it, so the ordering holds honestly. Grant create certificates in a namespace running Firebolt Instances only to principals you would trust with the JWT signing key. Closing that case requires cert-manager to stop reusing pre-existing private keys, which conflicts with operator-coordinated key rotation.

You can hit this legitimately by deleting a signing Certificate and letting the Firebolt Operator recreate it, such as during a GitOps prune. The surviving Secret then predates its new Certificate. The Firebolt Operator allows this when the Secret holds the same public key it already recorded in status.auth.signingKeys[].observedPublicKeyFingerprint. If that fingerprint is also gone, the Firebolt Operator reports a SigningKeyProvenanceRejected event and waits. Deleting the Secret to clear the rejection destroys the signing key and invalidates every token signed with it, so confirm where the Secret came from first.

The TLS listeners do not need this check. Their certificates use rotationPolicy: Always, so cert-manager generates a fresh private key on every issuance and never adopts pre-existing material.

See also

• architecture: Full Firebolt Operator architecture.
• monitoring: Metrics and alerts the Firebolt Operator exposes.
• instance-crd-reference: Firebolt Operator-owned vs. user-owned fields on FireboltInstance.
• fireboltengineclass-crd-reference: Firebolt Operator-owned vs. user-owned fields on FireboltEngineClass.