Upgrade engrams
Move a deployment from one release to the next, and what happens to running sessions while you do.
A release of engrams is a git tag, v0.10.0 for example, and a set of container images with
the same version as their tag. The two charts in a checkout of that tag install those images
by default: an image tag you do not set falls back to the chart’s appVersion. To upgrade,
check out the newer tag and run helm upgrade on both releases.
engrams is a 0.x project, so any release can change a wire format, a database schema, or a Terraform module. The release notes say which. Read them before you start.
Before you upgrade
Section titled “Before you upgrade”- Back up both databases. The coordinator and the orchestrator apply their schema migrations during the upgrade, and the migrations only go forward. A backup is the one way back to the older release.
- Keep your values files. You need the two files you passed to
helm install, and the Terraform variables you applied with.
1. Check out the release
Section titled “1. Check out the release”git fetch --tagsgit checkout v0.10.0Use the tag of the release you are moving to. git tag --list 'v*' --sort=-v:refname lists
the releases, newest first.
2. Apply Terraform and render the values again
Section titled “2. Apply Terraform and render the values again”A release can add a secret, an identity, or a value that the charts expect. Apply the quickstart with the variables you used the first time, then write the two overlays again. On GCP:
cd deploy/terraform/gcp/quickstartterraform initterraform apply # with your -var flags or your .tfvars fileterraform output -raw engram_values > /tmp/engram.tfvalues.yamlterraform output -raw host_fleet_values > /tmp/host-fleet.tfvalues.yamlcd ../../../..On AWS the directory is deploy/terraform/aws/quickstart. Read the plan before you approve
it: a release should not replace the cluster, the database, or the bucket, and a plan that
says it will is a reason to stop.
Then compare your two values files with the examples in the new checkout,
deploy/helm/engram/values-<cloud>.yaml.example and
deploy/helm/engram-host-fleet/values-<cloud>.yaml.example, and copy over any key that is
new. If your files set an image tag, remove it, or the upgrade keeps the old images.
3. Upgrade the control plane
Section titled “3. Upgrade the control plane”helm upgrade engram deploy/helm/engram \ -n engrams -f /tmp/engram-values.yaml -f /tmp/engram.tfvalues.yamlHelm first runs a job that migrates the orchestrator’s database, and stops the upgrade if the job fails. The coordinator migrates its own database when its new pods start. Then the three Deployments roll. Sessions run on the hosts, so a control-plane roll does not touch them; the dashboard reconnects on its own.
kubectl get pods -n engrams should show the coordinator, web, and orchestrator pods
Running on the new image.
4. Upgrade the host fleet
Section titled “4. Upgrade the host fleet”Helm installs a custom resource definition once and never updates it, so apply the new one by hand before the chart:
kubectl apply -f deploy/helm/engram-host-fleet/crds/
helm upgrade hf deploy/helm/engram-host-fleet \ -n engrams-hosts -f /tmp/fleet-values.yaml -f /tmp/host-fleet.tfvalues.yamlThis command returns at once and no host pod restarts yet. The operator sees the new images
and replaces the host pods one node at a time: it cordons the node, deletes the pod, waits
until the new pod is ready, and moves to the next. The VMs on the node keep running while
the pod is replaced, and the new pod reattaches to them. Watch the roll with kubectl get pods -n engrams-hosts -w, or on the dashboard’s
Settings › Fleet page.
Do steps 3 and 4 together. The coordinator and the host agents share a wire version and refuse to talk across a mismatch, so after a release that changes it, new sessions queue until the first host is on the new version.
When a release changes how hosts run VMs
Section titled “When a release changes how hosts run VMs”Most releases roll the host pods in place, and the new pod reattaches to the running VMs.
Some releases change how a host serves a VM’s disk or memory, so the new host agent cannot
adopt the VMs the old one started. The charts handle that release for you, and both
helm upgrade commands stay the same:
- Before the upgrade changes anything, a job compares the new host agent with every host. If a host runs VMs the new build cannot adopt, the job cordons it and evicts each session on it to a checkpoint. A session that never had a checkpoint gets one first; a paused session is resumed so it can be evicted. Both charts run this job, so either can go first.
- If the job cannot evict a session within
upgradeDrain.timeoutSeconds(an hour by default), it fails and the upgrade stops before any change. The release notes and the job’s log name the sessions. - After the host fleet chart upgrades, a second job replaces the drained host pods so they start the new build; with the operator enabled, the operator does it. When the new build’s files would leave the disk nearly full, node-prep clears the chunk cache first; it refills from the chunk store.
- Each host comes back on the new build, and the coordinator lifts the job’s cordon. A session resumes from its checkpoint the next time someone uses it.
Every running session sees one eviction and one resume, not a live move. An upgrade like this also lifts any cordon you placed by hand on the hosts it drains.
Run a build that is not a release
Section titled “Run a build that is not a release”Set the image tags to the full commit SHA of a commit on main; every build is published
under that tag. The control-plane chart takes image.tag, web.image.tag, and
orchestrator.image.tag, and the fleet chart takes image.tag, operator.image.tag, and
nodeAssets.image.tag. Use the charts from the same commit.
Do not deploy the main tag. It moves, and the charts pull with IfNotPresent, so a
node that already has main keeps the build it has while a new node pulls a newer one.