Skip to content

PAI release runbook

This is the operational guide for the release captain cutting a PAI ISO release. It is paired with CHANGELOG.md (what shipped) and MIGRATION.md (what users need to do to upgrade).

1. Cadence

PAI has no fixed release cadence. Cut a release when a meaningful set of user-visible changes has accumulated — new features, model updates, or non-trivial fixes. Avoid releases that only churn internal structure.

Security fixes ship out-of-band as PATCH releases as soon as a fix is verified. Do not batch a security fix behind unrelated feature work.

2. Versioning rules (SemVer, PAI flavor)

PAI follows SemVer 2.0.0. For this project:

  • MAJOR (X.0.0) — bump when a change breaks existing users:
  • Debian base version changes (e.g. Bookworm → Trixie).
  • LUKS / persistence volume format changes that require a one-shot migration tool.
  • Kernel or bootloader changes that invalidate existing USB installs.
  • Removal of a preinstalled app or model users depend on.
  • MINOR (0.X.0) — bump for backwards-compatible additions:
  • New preinstalled app, new model, new supported architecture.
  • New feature in the Flash app or website.
  • PATCH (0.0.X) — bump for fixes that do not change behavior:
  • Security fixes, package updates, bugfixes.

When in doubt, prefer the higher bump.

3. Pre-flight checklist

Before starting a build, confirm:

  • CHANGELOG.md [Unreleased] moved under a new [X.Y.Z] — YYYY-MM-DD heading, with compare-link footer updated.
  • MIGRATION.md has a section for this version transition (or explicitly says "no migration required").
  • Website changelog entry written in website/src/content/changelog/vX-Y-Z.mdx.
  • Docs in docs/ and website/src/content/docs/ reflect the release.
  • CI green on both x86_64 and ARM64.
  • Smoke-tested a fresh ISO on real hardware for each architecture (boot, connect to network, run Ollama, unlock persistence).
  • README version references, if any, are updated.

4. Build steps

Build from a clean worktree on the tagged commit.

AMD64 / x86_64

./build.sh

Output: pai-vX.Y.Z-amd64.iso in the build output directory.

ARM64 (including Apple Silicon)

./arm64/build.sh

Output: pai-vX.Y.Z-arm64.iso.

Both builds run inside the Docker image defined by Dockerfile.build.

5. Signing and checksums

For each ISO produced:

sha256sum pai-vX.Y.Z-amd64.iso > pai-vX.Y.Z-amd64.iso.sha256
sha256sum pai-vX.Y.Z-arm64.iso > pai-vX.Y.Z-arm64.iso.sha256

minisign -Sm pai-vX.Y.Z-amd64.iso
minisign -Sm pai-vX.Y.Z-arm64.iso

The minisign secret key lives only on the release captain's hardware token. It is never committed, never copied to CI, and never typed on a networked machine that is not the release workstation.

The minisign public key is committed to the repository root as minisign.pub and is also published on the GitHub Releases page so users can verify downloads independently of the repo.

Users verify with:

minisign -Vm pai-vX.Y.Z-amd64.iso -p minisign.pub
sha256sum -c pai-vX.Y.Z-amd64.iso.sha256

6. Publishing

  1. Create a signed, annotated git tag from the release commit: 
    git tag -s vX.Y.Z -m "PAI vX.Y.Z"
git push origin vX.Y.Z
  2. Create a GitHub Release for the tag. Title: PAI vX.Y.Z. Body: copy the matching CHANGELOG section verbatim, with a link back to CHANGELOG.md and MIGRATION.md.
  3. Upload, for each architecture:
  4. pai-vX.Y.Z-<arch>.iso
  5. pai-vX.Y.Z-<arch>.iso.sha256
  6. pai-vX.Y.Z-<arch>.iso.minisig
  7. Keep the release as a draft until all six artifacts are attached. Publish only when complete.

7. Post-release

  • Announce on the project's social channels (Mastodon, Twitter/X).
  • Rebuild the website so the new release shows up on /apps/changelog and /apps/flash: 
    cd website && npm run build
  • Open a follow-up PR that re-adds an empty [Unreleased] section at the top of CHANGELOG.md and updates the compare-link footer to v<new>...HEAD.

8. Hotfix flow

When a shipped release has a critical bug or security issue:

  1. Branch from the tag: git checkout -b hotfix/vX.Y.Z+1 vX.Y.Z.
  2. Cherry-pick (or write) the minimal fix. No unrelated changes.
  3. Bump PATCH in CHANGELOG and any version files.
  4. Follow sections 3 → 6 as normal.
  5. Fast-forward main with the hotfix so the fix is not lost on the next MINOR release.

9. Yanking a release

If a release must be withdrawn (dangerous bug, broken crypto, wrong artifact):

  1. On the GitHub Release page, check "Set as a pre-release" and edit the body to start with a prominent ⚠️ YANKED banner explaining why, and which version users should use instead.
  2. Add a top-level note in CHANGELOG.md directly under the version heading: 
    ## [X.Y.Z] — YYYY-MM-DD — YANKED

**This release has been yanked.** Reason: <one line>.
Users on this version should upgrade to <replacement> immediately.
See [MIGRATION.md](./MIGRATION.md).
  3. Do not delete the tag or the artifacts — users need them to verify what they already installed. Yanking is about signaling, not unpublishing.
  4. Cut a replacement release (PATCH bump) as soon as a fix exists.