design: harbormaster v0.1 architecture + state-machine sketch #1

New issue

Open

opened 2026-06-19 21:04:46 +02:00 by bosun · 0 comments

bosun commented

2026-06-19 21:04:46 +02:00

Owner

What harbormaster is

A Go daemon that arbitrates VRAM on a single GPU host between two coexisting workloads — an LLM served by Ollama, and an image generator served by ComfyUI — without forcing the operator to choose one or the other at config time. Both workloads stay continuously available (HTTP daemons up); only their VRAM residency is gated by harbormaster's state machine.

The motivating problem: a single 24 GiB consumer GPU can't host both a useful LLM (14–22 GiB) and a useful image generation model (Flux at ~17 GiB) simultaneously. Existing tooling (Ollama, ComfyUI, LiteLLM, gpustack, Triton) targets adjacent but different problems. There is no published solution for hybrid LLM-and-image-gen VRAM arbitration with hysteresis-based switching on a single host.

The target deployment is a single GPU host on a small home server, where one operator (or a small group of agent processes) sends requests to both workloads sporadically, and round-robin or static partitioning would produce a worse experience than intelligent VRAM swapping.

Design anchors

These are the load-bearing observations from Caymans Admin's investigation 2026-06-19. Each one rules out a class of designs and points at the right shape.

1. ComfyUI lazy-loads its weights

ComfyUI's systemd service stays at ~386 MiB VRAM when its HTTP daemon is up but no inference job is running. The 17 GiB number only materializes when a KSampler node executes a workflow. Implication: harbormaster does not need to manage ComfyUI's process lifecycle (start/stop the systemd service). It just needs to gate when inference requests are executed. The ComfyUI daemon stays up always; harbormaster either lets a queued workflow run (paying the load-cost on entry) or holds it until the LLM is evicted.

This is the single biggest design simplification. An earlier sketch assumed systemctl --user start/stop comfyui as the heavy switch action; that's wrong. The right shape is a request-queue-with-policy in front of ComfyUI's API, not a process supervisor.

2. Ollama's `keep_alive` knob is the LLM-side switch

Ollama supports a per-request keep_alive parameter that overrides the system-wide OLLAMA_KEEP_ALIVE default. Harbormaster uses this to control when the LLM stays VRAM-resident:

LLM is priority mode: send chat requests with keep_alive: -1 (or a long duration) — model stays hot in VRAM between requests.
Demote to evict: send a no-op chat request (or any inference) with keep_alive: 0 — model unloads immediately after the response. VRAM is freed for ComfyUI's queue.
Grace-period exit / prewarm: when grace expires and harbormaster is swapping back to LLM-priority, send a prompt: "" warmup request with keep_alive: -1 to repopulate VRAM before the user's next visible turn lands.

3. State machine with policy-as-data

The demotion rule ("swap LLM out for image gen when …") will accumulate dimensions over time: queue depth threshold, max-wait-time threshold, time-of-day overrides, per-caller priority hints, batch-job vs interactive-job distinctions. Bake the state machine kernel into Go; load the policy from a config file (TOML or YAML). Don't hardcode thresholds as Go constants.

Proposed state set for v0.1:

LLM_LOADED       (Ollama hot, ComfyUI cold; default steady state)
LLM_LOADED_SWAPPING_TO_IMAGE   (transition; LLM being evicted, image queue head about to start)
IMAGE_LOADED     (ComfyUI active, LLM cold; transient)
IMAGE_LOADED_GRACE   (no image requests pending, grace deadline counting down)
IMAGE_LOADED_GRACE_SWAPPING_TO_LLM   (transition; prewarming LLM, ComfyUI weights about to evict)

Transitions are policy-driven (queue thresholds, wait timeouts, grace deadlines), not hardcoded. State transitions are atomic from the API caller's perspective — a chat request landing during LLM_LOADED_SWAPPING_TO_IMAGE queues with an estimated wait time rather than racing.

4. Grace-period extends on each new image request

Iterative image work — tweaking prompts, trying alternatives — generates request bursts, not isolated calls. If the grace period was fixed-duration-from-last-swap, the user would face a full cold reload on every prompt iteration. Instead: the grace deadline resets on each new image request received while in IMAGE_LOADED_GRACE. A user doing 10 minutes of prompt iteration pays the cold-reload cost once at session start, not 10 times.

The operator's framing on this is verbatim: "Generated images frequently need to be tweaked around for a bit, or other ideas need to be tried out, and it would be cumbersome to wait a full hour each time." The grace-resets-on-activity shape is what makes the UX bearable.

5. Don't cancel mid-stream chat for image demotion

If a streamed chat completion is mid-response when an image request crosses the demotion threshold, let the chat finish. Cancellation introduces visible mid-response truncation which is the worst possible UX, and the chat completion is bounded (~20s at gpt-oss:20b's 170 tok/s for a long response). Image requests wait their fair share.

v0.1 HTTP API shape (sketch)

OpenAI-compatible chat endpoint plus a ComfyUI-shaped image endpoint, both terminating at harbormaster which proxies to the actual workload:

POST /v1/chat/completions   # OpenAI-compatible, proxies to Ollama
POST /v1/images/generations # ComfyUI-shaped queue submission
GET  /v1/images/jobs/{id}   # poll job status / fetch result
GET  /healthz               # daemon liveness
GET  /v1/state              # current state machine view (which workload is loaded, queue depths, grace deadline)

The /v1/state endpoint is for harbormaster's own observability — a Grafana panel can poll it. Not strictly required for v0.1 but cheap to include.

OpenAI-compatibility on the chat side means aichat (or any other OpenAI-compatible client) can point at harbormaster's URL transparently. The Cabin Boy chamber's aichat config switches from http://caymans:11434/v1/... (direct Ollama) to http://caymans:<hmport>/v1/... (harbormaster) once harbormaster is live, with no client-side changes beyond the URL.

Config schema sketch (TOML)

[server]
listen = "0.0.0.0:8090"

[ollama]
url = "http://127.0.0.1:11434"
default_model = "gpt-oss:20b"
keep_alive_priority = "-1"    # while LLM is priority workload
keep_alive_demote = "0"       # on demote, evict immediately

[comfyui]
url = "http://127.0.0.1:8188"

[policy.demote_to_image]
queue_depth_threshold = 1            # demote LLM as soon as any image queued
max_wait_seconds = 0                 # immediate; image requests don't wait

[policy.grace]
seconds_after_last_image_request = 600   # 10-minute grace before swapping back to LLM

Thresholds are conservative defaults intended for a single-user homelab. The operator-stakes-tuning loop is just "edit toml, restart harbormaster."

Open questions for v0.1

Not blockers — design questions to settle while building, or to defer to v0.2:

Per-caller priority hints. Should the API support a priority field on incoming requests so a low-priority batch job can be reordered behind interactive traffic? Probably no for v0.1; add when we have an actual second caller class.
Model swapping within Ollama. Harbormaster v0.1 assumes a single default LLM (gpt-oss:20b). If the Cabin Boy chamber's fallback to qwen2.5-coder:32b for code-narrow tasks needs to coexist, does harbormaster need to manage Ollama model state too? Probably defer — Ollama handles its own model swapping fine; harbormaster's concern is whether an LLM is loaded, not which.
Multi-tenancy for image jobs. Should images-in-flight be tagged by submitter for cancellation purposes? Probably no for v0.1.
Authentication. Currently the design assumes LAN-only deployment. Open-sourcing for wider use will eventually need at least an API key knob.
What happens when ComfyUI's underlying workflow fails? Harbormaster should mark the job failed and continue draining the queue; explicit error-state handling needs to be in the state machine.

Naming history

The project name harbormaster was settled 2026-06-19 from Caymans Admin's proposed shortlist (gpud, gpuswap, gpubroker, holster, harbormaster). Reasons it won: (a) naval-coherent with the rest of the alcatraz crew (Bosun, Pilot, Carpenter, Surveyor, Lookout, Shipwright, Quartermaster, Herald, Cabin Boy — and now Harbormaster); (b) semantically right — the harbormaster controls who uses the harbor (the GPU); (c) quartermaster is taken; (d) reads cleanly at the CLI (harbormaster --config foo.toml).

Substrate references

Caymans Admin benchmark and design anchors — 2026-06-19 ~17:30 + ~20:30 CEST. The lazy-load discovery (§1 above) is theirs.
tmux-tell #580 — internal fan-out throttle, per-pool amendment. Adjacent substrate-mechanism work; harbormaster's ollama pool from the chamber-side is the no-throttle case that #580's design needs to model.
Cabin Boy chamber plan — the paired project whose v1 needs are what harbormaster's first deployment will serve.
CELLBLOCK retrospective at frankenbit/cellblock#11 — origin of the trusted-cache-hides-fresh-state family discipline. Harbormaster's state-as-data approach is partly motivated by avoiding hardcoded-policy as a similar shape.

What this issue is for

This is the design substrate-of-record for harbormaster's v0.1. As decisions are made, append-amend below (or in linked PRs / issues). When v0.1.0 ships, this issue closes with a synthesis link to the actual code+docs that implement what's above.

— Bosun (filing 2026-06-19)

## What harbormaster is A Go daemon that arbitrates VRAM on a single GPU host between two coexisting workloads — an LLM served by Ollama, and an image generator served by ComfyUI — without forcing the operator to choose one or the other at config time. Both workloads stay continuously *available* (HTTP daemons up); only their *VRAM residency* is gated by harbormaster's state machine. The motivating problem: a single 24 GiB consumer GPU can't host both a useful LLM (14–22 GiB) and a useful image generation model (Flux at ~17 GiB) simultaneously. Existing tooling (Ollama, ComfyUI, LiteLLM, gpustack, Triton) targets adjacent but different problems. There is no published solution for hybrid LLM-and-image-gen VRAM arbitration with hysteresis-based switching on a single host. The target deployment is a single GPU host on a small home server, where one operator (or a small group of agent processes) sends requests to both workloads sporadically, and round-robin or static partitioning would produce a worse experience than intelligent VRAM swapping. ## Design anchors These are the load-bearing observations from Caymans Admin's investigation 2026-06-19. Each one rules out a class of designs and points at the right shape. ### 1. ComfyUI lazy-loads its weights ComfyUI's systemd service stays at ~386 MiB VRAM when its HTTP daemon is up but no inference job is running. The 17 GiB number only materializes when a KSampler node executes a workflow. Implication: **harbormaster does not need to manage ComfyUI's process lifecycle (start/stop the systemd service).** It just needs to gate when inference requests are *executed*. The ComfyUI daemon stays up always; harbormaster either lets a queued workflow run (paying the load-cost on entry) or holds it until the LLM is evicted. This is the single biggest design simplification. An earlier sketch assumed `systemctl --user start/stop comfyui` as the heavy switch action; that's wrong. The right shape is a request-queue-with-policy in front of ComfyUI's API, not a process supervisor. ### 2. Ollama's `keep_alive` knob is the LLM-side switch Ollama supports a per-request `keep_alive` parameter that overrides the system-wide `OLLAMA_KEEP_ALIVE` default. Harbormaster uses this to control when the LLM stays VRAM-resident: - **LLM is priority mode**: send chat requests with `keep_alive: -1` (or a long duration) — model stays hot in VRAM between requests. - **Demote to evict**: send a no-op chat request (or any inference) with `keep_alive: 0` — model unloads immediately after the response. VRAM is freed for ComfyUI's queue. - **Grace-period exit / prewarm**: when grace expires and harbormaster is swapping back to LLM-priority, send a `prompt: ""` warmup request with `keep_alive: -1` to repopulate VRAM *before* the user's next visible turn lands. ### 3. State machine with policy-as-data The demotion rule ("swap LLM out for image gen when …") will accumulate dimensions over time: queue depth threshold, max-wait-time threshold, time-of-day overrides, per-caller priority hints, batch-job vs interactive-job distinctions. **Bake the state machine *kernel* into Go; load the *policy* from a config file (TOML or YAML).** Don't hardcode thresholds as Go constants. Proposed state set for v0.1: ``` LLM_LOADED (Ollama hot, ComfyUI cold; default steady state) LLM_LOADED_SWAPPING_TO_IMAGE (transition; LLM being evicted, image queue head about to start) IMAGE_LOADED (ComfyUI active, LLM cold; transient) IMAGE_LOADED_GRACE (no image requests pending, grace deadline counting down) IMAGE_LOADED_GRACE_SWAPPING_TO_LLM (transition; prewarming LLM, ComfyUI weights about to evict) ``` Transitions are policy-driven (queue thresholds, wait timeouts, grace deadlines), not hardcoded. State transitions are atomic from the API caller's perspective — a chat request landing during `LLM_LOADED_SWAPPING_TO_IMAGE` queues with an estimated wait time rather than racing. ### 4. Grace-period extends on each new image request Iterative image work — tweaking prompts, trying alternatives — generates request bursts, not isolated calls. If the grace period was fixed-duration-from-last-swap, the user would face a full cold reload on every prompt iteration. Instead: the grace deadline *resets* on each new image request received while in `IMAGE_LOADED_GRACE`. A user doing 10 minutes of prompt iteration pays the cold-reload cost once at session start, not 10 times. The operator's framing on this is verbatim: *"Generated images frequently need to be tweaked around for a bit, or other ideas need to be tried out, and it would be cumbersome to wait a full hour each time."* The grace-resets-on-activity shape is what makes the UX bearable. ### 5. Don't cancel mid-stream chat for image demotion If a streamed chat completion is mid-response when an image request crosses the demotion threshold, **let the chat finish.** Cancellation introduces visible mid-response truncation which is the worst possible UX, and the chat completion is bounded (~20s at gpt-oss:20b's 170 tok/s for a long response). Image requests wait their fair share. ## v0.1 HTTP API shape (sketch) OpenAI-compatible chat endpoint plus a ComfyUI-shaped image endpoint, both terminating at harbormaster which proxies to the actual workload: ``` POST /v1/chat/completions # OpenAI-compatible, proxies to Ollama POST /v1/images/generations # ComfyUI-shaped queue submission GET /v1/images/jobs/{id} # poll job status / fetch result GET /healthz # daemon liveness GET /v1/state # current state machine view (which workload is loaded, queue depths, grace deadline) ``` The `/v1/state` endpoint is for harbormaster's *own* observability — a Grafana panel can poll it. Not strictly required for v0.1 but cheap to include. OpenAI-compatibility on the chat side means aichat (or any other OpenAI-compatible client) can point at harbormaster's URL transparently. The Cabin Boy chamber's aichat config switches from `http://caymans:11434/v1/...` (direct Ollama) to `http://caymans:<hmport>/v1/...` (harbormaster) once harbormaster is live, with no client-side changes beyond the URL. ## Config schema sketch (TOML) ```toml [server] listen = "0.0.0.0:8090" [ollama] url = "http://127.0.0.1:11434" default_model = "gpt-oss:20b" keep_alive_priority = "-1" # while LLM is priority workload keep_alive_demote = "0" # on demote, evict immediately [comfyui] url = "http://127.0.0.1:8188" [policy.demote_to_image] queue_depth_threshold = 1 # demote LLM as soon as any image queued max_wait_seconds = 0 # immediate; image requests don't wait [policy.grace] seconds_after_last_image_request = 600 # 10-minute grace before swapping back to LLM ``` Thresholds are conservative defaults intended for a single-user homelab. The operator-stakes-tuning loop is just "edit toml, restart harbormaster." ## Open questions for v0.1 Not blockers — design questions to settle while building, or to defer to v0.2: 1. **Per-caller priority hints.** Should the API support a `priority` field on incoming requests so a low-priority batch job can be reordered behind interactive traffic? Probably no for v0.1; add when we have an actual second caller class. 2. **Model swapping within Ollama.** Harbormaster v0.1 assumes a single default LLM (gpt-oss:20b). If the Cabin Boy chamber's fallback to qwen2.5-coder:32b for code-narrow tasks needs to coexist, does harbormaster need to manage Ollama model state too? Probably defer — Ollama handles its own model swapping fine; harbormaster's concern is *whether* an LLM is loaded, not *which*. 3. **Multi-tenancy for image jobs.** Should images-in-flight be tagged by submitter for cancellation purposes? Probably no for v0.1. 4. **Authentication.** Currently the design assumes LAN-only deployment. Open-sourcing for wider use will eventually need at least an API key knob. 5. **What happens when ComfyUI's underlying workflow fails?** Harbormaster should mark the job failed and continue draining the queue; explicit error-state handling needs to be in the state machine. ## Naming history The project name `harbormaster` was settled 2026-06-19 from Caymans Admin's proposed shortlist (`gpud`, `gpuswap`, `gpubroker`, `holster`, `harbormaster`). Reasons it won: (a) naval-coherent with the rest of the alcatraz crew (Bosun, Pilot, Carpenter, Surveyor, Lookout, Shipwright, Quartermaster, Herald, Cabin Boy — and now Harbormaster); (b) semantically right — the harbormaster controls who uses the harbor (the GPU); (c) `quartermaster` is taken; (d) reads cleanly at the CLI (`harbormaster --config foo.toml`). ## Substrate references - **Caymans Admin benchmark and design anchors** — 2026-06-19 ~17:30 + ~20:30 CEST. The lazy-load discovery (§1 above) is theirs. - **[tmux-tell #580](https://git.frankenbit.de/frankenbit/tmux-tell/issues/580)** — internal fan-out throttle, per-pool amendment. Adjacent substrate-mechanism work; harbormaster's `ollama` pool from the chamber-side is the no-throttle case that #580's design needs to model. - **[Cabin Boy chamber plan](memory:project_cabin_boy_harbormaster)** — the paired project whose v1 needs are what harbormaster's first deployment will serve. - **CELLBLOCK retrospective** at [frankenbit/cellblock#11](https://git.frankenbit.de/frankenbit/cellblock/issues/11) — origin of the *trusted-cache-hides-fresh-state* family discipline. Harbormaster's state-as-data approach is partly motivated by avoiding hardcoded-policy as a similar shape. ## What this issue is for This is the design substrate-of-record for harbormaster's v0.1. As decisions are made, append-amend below (or in linked PRs / issues). When v0.1.0 ships, this issue closes with a synthesis link to the actual code+docs that implement what's above. — Bosun (filing 2026-06-19)