# Sandbox backends

> Firecracker for production, Apple Virtualization for Mac development, subprocesses for iterating.

Source: https://engramsfactory.com/docs/concepts/sandbox-backends/

A backend is the driver for whatever runs a session: a microVM monitor or, in one case,
nothing at all. The rest of engrams does not know which one is in use. Three ship.

| Backend | Isolation | Snapshots | Runs on | Use it for |
|---|---|---|---|---|
| Firecracker | microVM on KVM | memory snapshot with lazy paging, disk on a kernel thin volume | Linux with `/dev/kvm` | production |
| Apple Virtualization | microVM on Apple's Hypervisor framework | clone of the root filesystem | macOS 12 or later on Apple Silicon | development on a Mac with real isolation |
| Process | none; a host subprocess | tarball of the working directory | macOS or Linux | iterating on the orchestration layer without a VM |

Production isolation is always Firecracker. The other two exist so the same code paths can
be exercised on a laptop; the Apple backend runs the same in-guest daemon and harness bundles
and the same image pipeline, so a session that works on a Mac works on the fleet.

`just dev` picks the backend for the machine it runs on: Firecracker when `/dev/kvm` is
readable and writable, Apple Virtualization on an Apple Silicon Mac, subprocesses otherwise.
Set `ENGRAM_SANDBOX_BACKEND` to force one.

## What a Firecracker host needs

A production host is a Linux machine with KVM. On managed Kubernetes that means a node pool
with nested virtualization, which every provider offers on Intel only: the C3 family on GKE,
m8i or bare metal on EKS. The fleet's Helm chart runs a node-prep step that does the setup
below on each node; this list is what that step does, for anyone running hosts outside
Kubernetes.

**KVM.** `/dev/kvm` must be readable and writable by the host agent's user.

```sh
[ -r /dev/kvm ] && [ -w /dev/kvm ] && echo OK || echo FAIL
```

**Lazy memory paging.** The host agent needs `/dev/userfaultfd` with mode `0666` (a udev
rule) and the sysctl `vm.unprivileged_userfaultfd = 1`.

**Session disks.** The kernel needs device-mapper thin provisioning (the `dm_thin_pool`
module, shipped with the Ubuntu cloud image kernel) and the node needs `thin-provisioning-tools`. The node-prep step
builds a pool from two files on the host's working filesystem and checks it on every boot;
`ENGRAM_THIN_POOL` gives the host agent the pool's name. A Firecracker host agent refuses to
start without one.

**The guest kernel and the bundles.** Each host stages a guest kernel and the read-only
bundles for the in-guest daemon, the harnesses, and the skills. The VM boots with
`init=/sbin/engram-init`, a shim that copies the daemon out of its bundle and hands over to
it, so nothing of engrams is in your image.

**The egress proxy.** Every host runs a filtering proxy on TCP 443 and DNS on port 53, and
every VM's outbound traffic is redirected through it. It is mandatory: a host that cannot
load the proxy's certificate authority or bind its ports refuses to start. Every host in a
fleet loads the same CA pair, so a session that migrates keeps trusting its new host.

## The CPU platform is a one-way door

A base snapshot captured on a newer CPU never restores on an older one. Keep a fleet on one
platform, or bake images per platform. The Terraform quickstarts pin GKE to Sapphire Rapids
(C3) and EKS to Granite Rapids (m8i) by default; if you run both clouds and want one bake to
serve them, the AWS side offers `m7i.metal-24xl` for parity, at a much higher price.
