Kubernetes Operator changelog

Kubernetes Operator v3.0.1

Restate Operator v3.0.1 Release Notes

See the v3.0.0 release notes for changes prior to this release.

Reduced RestateDeployment introspection query load

The RestateDeployment controller no longer issues a Restate introspection query (POST /query over sys_invocation_status) on every reconcile, and no longer reconciles on every HorizontalPodAutoscaler status update.

Owned HPAs are watched for spec changes only. The Kubernetes HPA controller re-writes an HPA’s status on a ~15s timer; with per-version autoscaling that is one such write per version (including every draining one), and each previously woke the owning RestateDeployment. The watch now filters to generation (spec) changes, so those heartbeats no longer trigger reconciles. The operator’s HPA cache still sees every update.
The deployment-usage query runs only when needed. The query answers whether this version is the one Restate routes new invocations to (registration) and whether older versions have drained (cleanup). In the common case — this version already latest, with no older version to drain — the operator reads “already latest” from the cheaper GET /services and skips the invocation-status query entirely. Registration, promotion, rollback, foreign-takeover, and drain paths run the full query exactly as before.

Why this matters

With per-version autoscaling enabled, a deployment carrying several draining versions produced a steady stream of reconciles — roughly one HPA heartbeat every 15 seconds per version — each issuing an invocation-status query. Across many deployments sharing one Restate admin endpoint this multiplied into sustained query load that could overwhelm Restate’s query engine, surfacing as 500 Internal Server Error (“No such scanner”) responses from /query.

Impact

No configuration change, and no change to registration, rollback, or drain behaviour.
Restate admin/query load from the operator drops sharply in the common case: introspection queries are issued during rollouts and drains, not continuously.
HorizontalPodAutoscaler status changes no longer trigger RestateDeployment reconciles; spec changes and (re)creation still do.

Related: #185.View on GitHub

Kubernetes Operator v3.0.0

Restate Operator v3.0.0 Release Notes

See the v2.8.1 release notes for changes prior to this release.This is a major release: upgrading an existing install requires a one-time CRD ownership hand-off (see Breaking Changes), and several deployment-lifecycle fixes change behavior on the first reconcile after upgrade (see Upgrade Notes).

Highlights

CRDs now upgrade with the chart. The three CRDs ship as a templated, standalone restate-operator-crds chart, so helm upgrade applies schema changes instead of silently skipping them. The first upgrade from ≤ 2.8.1 needs a one-time ownership hand-off.
The operator waits for its CRDs on startup instead of exiting/crashlooping, and surfaces the wait via /ready, a metric, and a Kubernetes event.
RestateDeployment deletion no longer hangs forever on its own latest version.
Rolling back now moves Restate’s routing, and Ready=True is stricter — inconsistent deployments are promoted on the next reconcile after upgrade.
GCP workload-identity IAMPolicyMember server-side-apply wedge is fixed.

Upgrade Notes
Breaking Changes
Improvements
Bug Fixes

Upgrade Notes

Read these before upgrading — each is expanded below.

CRDs: one-time ownership hand-off. The first upgrade to templated CRDs fails with invalid ownership metadata until you hand ownership to Helm. See Breaking Changes. Nothing is deleted if you skip it — the upgrade just aborts.
Chart and image move in lockstep. The readiness probe now targets /ready, which older operator images do not serve. If you pin the chart version to an image older than 3.0.0, pin the chart to a matching older version too — otherwise the /ready 404 leaves the pod NotReady forever.
Inconsistent RestateDeployments are promoted on the next reconcile. If a deployment’s desired revision is not the one Restate currently routes to, the operator will move routing to it — a routing change that happens without being asked for. Upgrade when that is acceptable. Endpoints registered outside their RestateDeployment now stall at Ready=False with reason ForeignDeployment instead of being silently tolerated.
Deletion now waits out the drain window. Deleting a RestateDeployment now takes at least spec.restate.drainDelaySeconds (default 300s) and still blocks on unfinished invocations (including scheduled ones, which have no upper bound).

Breaking Changes

CRD installation now upgrades with the chart — action required on first upgrade

Through 2.8.1 the CRDs shipped through Helm’s native crds/ directory, which is install-only: once the CRDs were in the cluster, helm upgrade never touched them again. That bit a customer — after an operator upgrade, new schema fields were silently pruned at admission until the CRDs were reapplied by hand.The CRDs now live in their own restate-operator-crds chart, rendered as ordinary templates, so upgrading the chart applies the new schema. They carry helm.sh/resource-policy: keep, so helm uninstall will not take the CRDs (or the custom resources under them) down with it. restate-operator-helm bundles this chart behind installCrds (default true), so a standalone operator install is unchanged. The CRD chart is also published on its own:

helm upgrade --install restate-operator-crds \
  oci://ghcr.io/restatedev/restate-operator-crds --version 3.0.0

One-time ownership hand-off. CRDs installed through the old crds/ directory (2.8.1 or earlier) carry no Helm ownership metadata, so the first upgrade to the templated CRDs fails with invalid ownership metadata until ownership is handed over. Pick one:

Helm ≥ 3.17 — add --take-ownership to the upgrade. Where you pass it depends on how you run the CRDs:
- Bundled (default, installCrds=true): helm upgrade restate-operator … --take-ownership
- Standalone CRD chart (installCrds=false): helm upgrade restate-operator-crds … --take-ownership
Older Helm — set the app.kubernetes.io/managed-by=Helm label plus the meta.helm.sh/release-name and meta.helm.sh/release-namespace annotations on the three CRDs, for the release that owns them.
ArgoCD / Flux — no action needed; they adopt the resources on sync and never hit Helm’s ownership gate.

Either way no custom resources are deleted — the failure aborts the upgrade, it does not cascade.If you don’t want Helm upgrading your CRDs. CRDs are cluster-scoped, so on a shared or multi-tenant cluster a helm upgrade now mutates a cluster-wide object that can affect other teams. To gate CRD changes behind manual review — or if you already manage CRDs via GitOps — set installCrds=false on the operator and own the CRD lifecycle yourself: apply crd/*.yaml directly, or install the standalone restate-operator-crds chart on your own cadence. This is the recommended path for ArgoCD/Flux.

Improvements

Startup: wait for CRDs instead of exiting (#166, #169)

If a required CRD (RestateCluster, RestateDeployment, RestateCloudEnvironment) is not yet installed when the operator starts, it no longer logs an error and exits. It now polls the apiserver’s discovery endpoint and waits for each CRD to appear, then begins reconciling. This matches how most Kubernetes operators behave and removes the ordering/restart dance in GitOps flows where the operator and its CRDs are applied together.Missing group/versions, 404/429/5xx responses, and an unreachable apiserver are all treated as “not ready yet, keep waiting”. Genuine configuration errors (401/403, bad kubeconfig) still log and exit. The PodIdentityAssociation check (when aws-pod-identity-association-cluster is set) still exits, since that is a configuration error rather than a race.The wait is observable three ways:

Readiness endpoint. /ready returns 200 only once every controller has its CRD and has started reconciling, otherwise 503 with the pending controllers listed:
{"ready":false,"pendingControllers":["RestateCluster","RestateDeployment"]}
/health keeps its always-200 behavior and is now the liveness probe.
Metric. restate_operator_crd_missing{crd="<plural>.<group>"} is 1 while waiting and 0 once available — alert with e.g. restate_operator_crd_missing > 0 for 5m.
Event. A single Warning/WaitingForCRD event is recorded against the operator’s own pod naming every CRD being waited on, closed out by a Normal/CRDsAvailable event. Nothing is emitted if the CRDs land within the first 10 seconds (the common case).

No new RBAC is required.

Helm Chart

The CRDs are delivered by the templated, standalone restate-operator-crds chart — see Breaking Changes.
The readiness probe now targets /ready; a liveness probe targeting /health was added. See the chart/image lockstep note.
The Deployment passes OPERATOR_POD_NAME and OPERATOR_POD_UID via the downward API so operator-level events attach to the right pod. If you deploy the operator without the chart, set both (or neither — without them the operator still logs and reports the metric, but emits no events).
The operator Service sets publishNotReadyAddresses: true, so a NotReady operator — exactly when restate_operator_crd_missing needs scraping — stays in the ServiceMonitor’s targets.

Examples & Docs

Example manifests, Pkl templates, and docs now reference Restate server 1.7 (#151). No forced default: the operator does not hardcode a server image, so existing clusters keep whatever version their manifests pin.

Bug Fixes

RestateDeployment

Deletion no longer hangs forever (#172). Cleanup previously conflated “a service still points at this deployment” with “it has unfinished invocations”, so during a deletion the latest version stayed “active” forever, the finalizer never completed, and kubectl delete restatedeployment never returned (namespace deletion inherited the hang). The operator now tracks the two facts separately: being a service’s current endpoint holds a version through a rollout but is ignored during a deletion; only unfinished invocations can hold a deletion, and they drain on their own.

Deletion now takes at least spec.restate.drainDelaySeconds (default 300s), and still blocks on unfinished invocations — scheduled invocations days out have no upper bound. A blocked deletion now backs off (30s → 5m) and the DeploymentInUse event names each version and its pinned/unpinned invocation counts.
No migration needed: a currently-stuck deletion proceeds on the next reconcile after upgrade (no manual finalizer edits). To see what is holding a deletion:
kubectl describe restatedeployment <name> -n <namespace>

Rolling back now moves Restate’s routing (#174). Because ReplicaSets/Services are named by a content hash, v1 → v2 → v1 re-adopts the original Restate deployment id, so restoring the Kubernetes revision was never enough on its own — Restate also has to be told the restored deployment is latest again. The operator now decides on whether Restate routes new invocations to the recorded deployment, re-registers with overwrite when it does not (preserving the deployment id and pinned invocations), verifies routing via GET /services before reporting Ready, and emits a Promoted event.

Ready=True is stricter: it now means Restate routes new invocations to the desired revision, not merely that the pods are up. See the routing-promotion upgrade note for the on-upgrade impact (unrequested promotions, ForeignDeployment stalls, and that a promoted deployment’s registration timestamp is reset in Restate).

To check for the inconsistent state this fixes:

kubectl get restatedeployment <name> -n <namespace> -o jsonpath='{.status.deploymentId}'
curl -s "$RESTATE_ADMIN/services/<ServiceName>" | jq -r .deployment_id

RestateCluster

Request-identity config emitted under the nested worker.invoker key (#158). With request signing enabled (spec.security.requestSigningPrivateKey), the operator now emits RESTATE_WORKER__INVOKER__REQUEST_IDENTITY_PRIVATE_KEY_PEM_FILE (→ worker.invoker.request-identity-private-key-pem-file) instead of the flat alias restate-server deprecated. Only the env var name changes; signing behavior is unchanged. No action required — this removes the startup deprecation warning and keeps working once restate-server drops the alias.GCP workload-identity IAMPolicyMember SSA wedge fixed (#136). The restate-workload-identity IAMPolicyMember CR is now applied with spec.resourceRef.apiVersion set, matching what Config Connector defaults at creation. Previously SSA stripped the defaulted value on every reconcile, Config Connector’s deny-immutable-field-updates webhook rejected it, and the reconcile loop blocked unrelated RestateCluster updates. Affects BYOC installs on GKE / Config Connector; existing wedged installs unblock once the upgraded operator reconciles.View on GitHub

Kubernetes Operator v2.8.1

Restate Operator v2.8.1 Release Notes

See the v2.7.0 release notes for changes prior to this release. (v2.8.0 was an unpublished, broken release and is superseded by v2.8.1.)

CRD installation

The chart’s CRDs (RestateCluster, RestateDeployment, RestateCloudEnvironment) now ship via an optional restate-operator-crds subchart that places them in Helm’s native crds/ directory. Installation is gated by the installCrds value (default true).

Fresh installs get all three CRDs by default.
Set installCrds: false to manage the CRDs out-of-band (BYOC / GitOps):
installCrds: false

Upgrading

Upgrading from <= 2.7 is uneventful. Existing CRDs are left untouched — there is no ownership adoption, no invalid ownership metadata error, and no --take-ownership step. helm uninstall never deletes the CRDs or your custom resources.

helm upgrade restate-operator restatedev/restate-operator --version 2.8.1

Note on CRD schema updates

Native crds/ are install-only under helm upgrade — this is Helm’s deliberately conservative CRD lifecycle, and it means chart upgrades will not apply CRD schema changes to an existing install. To update CRD schemas, apply the new CRDs explicitly:

kubectl apply -f crd/restateclusters.yaml -f crd/restatedeployments.yaml -f crd/restatecloudenvironments.yaml

or let a GitOps tool re-apply them on sync (e.g. ArgoCD). The CRDs are also published as GitHub release artifacts (crd/restateclusters.yaml, crd/restatedeployments.yaml, crd/restatecloudenvironments.yaml).View on GitHub

Kubernetes Operator v2.7.0

Restate Operator v2.7.0 Release Notes

Highlights

Pod lifecycle, sidecars, and extra volumes on RestateCluster — five new optional spec.compute fields (lifecycle, sidecars, terminationGracePeriodSeconds, extraVolumes, extraVolumeMounts) are passed through to the StatefulSet pod template, enabling graceful-shutdown hooks, companion containers, and mounting reusable content next to Restate. All fields are optional and defaults are unchanged.

New Features

Pod lifecycle, sidecars, termination grace period, and extra volumes

Five new optional fields on RestateCluster spec.compute are passed through to the underlying StatefulSet pod template:

lifecycle — container lifecycle hooks (postStart / preStop) for the Restate container. Note: Kubernetes runs postStart concurrently with the container entrypoint, so it is not a reliable “before Restate starts” hook; wrap the entrypoint via command/args for strict pre-start ordering.
sidecars — native sidecar containers. Each runs alongside the Restate container (started before it, terminated after it); the operator forces restartPolicy: Always so an entry behaves as a sidecar rather than a startup-blocking init container. Requires Kubernetes 1.29+ (native sidecars GA in 1.33).
terminationGracePeriodSeconds — overrides the pod termination grace period (previously hardcoded to 60 seconds). A value of 0 terminates immediately (SIGKILL, skipping any preStop hook).
extraVolumes — additional pod volumes alongside the operator-managed ones.
extraVolumeMounts — additional volume mounts for the Restate container; sidecars carry their own mounts and share these pod-level extraVolumes.

These let operators wire in graceful-shutdown hooks, run companion containers (log shippers, proxies, metrics exporters) next to Restate with correct start/stop ordering, and tune the shutdown window for their workload.Volume and mount collisions are validated up front: an extraVolumes name that reuses an operator-managed volume (storage, tmp, config, the trusted-CA volumes, or the request-signing volumes), or an extraVolumeMounts entry that reuses a path the operator already mounts on the Restate container, is rejected with a clear InvalidRestateConfig error rather than an opaque API-server failure.Impact on Users:

Existing deployments: no impact — all five fields are optional and omitting them preserves today’s behaviour (no lifecycle hooks, no sidecars, no extra volumes, 60s grace period).
New deployments: opt in via spec.compute.

Usage:

spec:
  compute:
    image: docker.restate.dev/restatedev/restate:latest
    terminationGracePeriodSeconds: 120
    extraVolumes:
      - name: hooks
        configMap:
          name: node-state-control
          defaultMode: 0755
    extraVolumeMounts:
      - name: hooks
        mountPath: /node-state-control
        readOnly: true
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "/node-state-control/pre-stop.sh"]
    sidecars:
      - name: log-shipper
        image: log-shipper:latest
        volumeMounts:
          - name: hooks
            mountPath: /node-state-control

Related: PR #153

Upgrading

This release adds new optional fields to the RestateCluster CRD. Apply the updated CRDs (server-side) and upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.7.0

​Restate Operator v3.0.1 Release Notes

​Reduced RestateDeployment introspection query load

​Why this matters

​Impact

​Restate Operator v3.0.0 Release Notes

​Highlights

​Table of Contents

​Upgrade Notes

​Breaking Changes

​CRD installation now upgrades with the chart — action required on first upgrade

​Improvements

​Startup: wait for CRDs instead of exiting (#166, #169)

​Helm Chart

​Examples & Docs

​Bug Fixes

​RestateDeployment

​RestateCluster

​Restate Operator v2.8.1 Release Notes

​CRD installation

​Upgrading

​Note on CRD schema updates

​Restate Operator v2.7.0 Release Notes

​Highlights

​New Features

​Pod lifecycle, sidecars, termination grace period, and extra volumes

​Upgrading

​Restate Operator v2.6.1 Release Notes

​Highlights

​Bug Fixes

​In-process tunnel mode rejected multi-label BYOC regions

​Upgrading

​Restate Operator v2.6.0 Release Notes

​Highlights

​New Features

​Per-version autoscaling for draining RestateDeployment versions

​tunnelMode: in-process for Restate Cloud deployments

​Tolerations and nodeSelector in the Helm chart

​Bug Fixes

​RestateDeployment

​Upgrading

​Restate Operator v2.5.1 Release Notes

​Highlights

​Bug Fixes

​Knative service URLs not tunneled for Restate Cloud

​Upgrading

​Restate Operator v2.5.0 Release Notes

​Highlights

​New Features

​Custom pod annotations and labels

​Bug Fixes

​Default canary image now ships a CA bundle

​Improvements

​Default tunnel client image bumped to 0.6.0

​Upgrading

​Restate Operator v2.4.0 Release Notes

​Highlights

​New Features

​Trusted CA certificates

​Configurable canary image

​Bug Fixes

​IPv6 dual-stack support

​Faster drain cleanup polling

​Upgrading

​Restate Operator v2.3.1 Release Notes

​Highlights

​Bug Fixes

​IAMPolicyMember cleanup causes 403 on non-GCP clusters

​Upgrading

​Restate Operator v2.3.0

​✨ New Features

​🐛 Bug Fixes

​⚙️ Configuration Changes

​⚠️ Upgrading Notes

​Restate Operator v2.2.0

​✨ New Features

​🐛 Bug Fixes

​⚙️ Configuration Changes

​📝 Documentation

​⚠️ Upgrading Notes

​New Contributors

Restate Operator v3.0.1 Release Notes

Reduced RestateDeployment introspection query load

Why this matters

Impact

Restate Operator v3.0.0 Release Notes

Highlights

Table of Contents

Upgrade Notes

Breaking Changes

CRD installation now upgrades with the chart — action required on first upgrade

Improvements

Startup: wait for CRDs instead of exiting (#166, #169)

Helm Chart

Examples & Docs

Bug Fixes

RestateDeployment

RestateCluster

Restate Operator v2.8.1 Release Notes

CRD installation

Upgrading

Note on CRD schema updates

Restate Operator v2.7.0 Release Notes

Highlights

New Features

Pod lifecycle, sidecars, termination grace period, and extra volumes

Upgrading

Restate Operator v2.6.1 Release Notes

Highlights

Bug Fixes

In-process tunnel mode rejected multi-label BYOC regions

Upgrading

Restate Operator v2.6.0 Release Notes

Highlights

New Features

Per-version autoscaling for draining RestateDeployment versions

`tunnelMode: in-process` for Restate Cloud deployments

Tolerations and nodeSelector in the Helm chart

Bug Fixes

RestateDeployment

Upgrading

Restate Operator v2.5.1 Release Notes

Highlights

Bug Fixes

Knative service URLs not tunneled for Restate Cloud

Upgrading

Restate Operator v2.5.0 Release Notes

Highlights

New Features

Custom pod annotations and labels

Bug Fixes

Default canary image now ships a CA bundle

Improvements

Default tunnel client image bumped to 0.6.0

Upgrading

Restate Operator v2.4.0 Release Notes

Highlights

New Features

Trusted CA certificates

Configurable canary image

Bug Fixes

IPv6 dual-stack support

Faster drain cleanup polling

Upgrading

Restate Operator v2.3.1 Release Notes

Highlights

Bug Fixes

IAMPolicyMember cleanup causes 403 on non-GCP clusters

Upgrading

Restate Operator v2.3.0

✨ New Features

🐛 Bug Fixes

⚙️ Configuration Changes

⚠️ Upgrading Notes

Restate Operator v2.2.0

✨ New Features

🐛 Bug Fixes

⚙️ Configuration Changes

📝 Documentation

⚠️ Upgrading Notes

New Contributors