hal0 documentation

hal0 turns a Linux box into a polished, OpenAI-compatible inference appliance. It is built for the AMD Ryzen AI Max+ 395 (“Strix Halo”) — iGPU + XDNA NPU + 128 GB of unified memory — and falls back to portable hardware parsers on every other host.

Every inference workload runs as its own podman container, supervised by a hal0-slot@<name>.service systemd unit. A single control plane, hal0-api on port 8080, owns the slot state machines, dispatches OpenAI-compatible /v1/* requests to the right slot, and serves the dashboard. There is no shared inference daemon to babysit.

Current release

This documentation tracks the hal0 v0.9.x beta line — the container-runtime era, where each slot (utility, embed, rerank, stt, tts, img, vision, and agent — plus the NPU-conditional slots stt-npu and embed-npu when FastFlowLM is installed) runs in a dedicated podman container. agent is the capable default/fallback anchor and utility is the cheap helper used for background jobs like memory extraction; the older chat/primary slot names are retired. See the changelog for the exact current version.

What you get

• OpenAI-compatible /v1/* API — chat, completions, embeddings, rerank (/v1/rerankings, plus the /v1/rerank alias for llama.cpp/Cohere-style clients), audio transcription, speech, and image generation. Point any OpenAI SDK at http://localhost:8080/v1 and go.
• Hardware-aware slots — each capability gets its own container, image, flags, port, and lifecycle, sharing only the GPU through an arbiter.
• Profiles — bench-tuned llama-server flag bundles per backend (rocm, rocm-dnse, rocm-moe, vulkan, cuda, dedicated embed/ rerank lanes, and more) that a slot picks up for its image, flags, and MTP behavior.
• Stacks — named, exportable bundles of slot + profile + model assignments (built-in saber, forge, and pi) that you can load, clone, and re-apply as a single declarative unit.
• First-run terminal setup — the hal0 setup TUI picks a hardware-anchored tier and provisions a coherent set of slots, models, and extensions.
• Prewired chat UI — OpenWebUI on port 3001, zero config.
• Signed self-update — cosign-verified tarballs with one-flag rollback.

The hal0 dashboard — slot status, hardware metrics, and quick actions.

Start here

Install hal0 One-line install: fetch, verify, unpack, and bring up the API.

First-run setup Run the hal0 setup TUI to provision your first slots, models, and extensions.

Load your first model Pull a model from Hugging Face and assign it to a slot.

Send your first chat Chat from OpenWebUI or curl the OpenAI-compatible endpoint.

No built-in auth or TLS

hal0-api binds 0.0.0.0:8080 and treats every request as authenticated by network reachability — the right posture for a trusted LAN appliance. If hal0 is reachable from anywhere you do not control, front it with a reverse proxy that owns auth and TLS.