glucose-dashboard/docs/design-profile-tuning.md

Moved. Implementation now lives in the separate glucose-profile-tuning repo (per this doc's own §2 isolation decision); the canonical, actively maintained copy of this file is at glucose-profile-tuning/docs/design-profile-tuning.md. This copy is kept for history and is no longer updated.

Design specification: oref0-autotune profile tuning

Status: final — user reviewed Requirements: requirements-profile-tuning.md (FR1–FR9, NFR1–NFR5) Related: design-reporting.md (same conventions, separate app)

This is a design spec, not code. Nothing here has been implemented yet — implementation starts after this doc is reviewed/approved, one phase at a time, each shipped through its own CI/CD pipeline (§8).

1. FR1 recap — infra decision (already executed)

FR1 (boot + inspect LXC 102/107) is done; full findings are recorded directly in requirements-profile-tuning.md §4 FR1. Summary: LXC 102 (autotune, 192.168.2.13) already has a working, pinned Docker image — ghcr.io/houthacker/nightscout-autotune:latest — that bundles the entire oref0 npm toolkit, not just the autotune algorithm:

Tool (already installed in the image)	Role in this design
nightscout-autotune (image's default CMD)	Fetches NS entries/treatments/profile, runs oref0-autotune, writes results to $OPENAPS_WORKDIR
oref0-upload-profile profile.json NSURL secret [--preview] [--switch]	Converts an oref0-format profile back to Nightscout's profile-store format and uploads it, optionally issuing a Profile Switch treatment
convert-ns-profile / mm-format-ns-profile	The NS-format ⇄ oref0-format conversion primitives, used internally by the two tools above

This changes the shape of the whole feature for the better: we do not need to write any Nightscout-profile-format conversion code ourselves. Both directions (NS → oref0 for analysis, oref0 → NS for staging) are already solved, tested, and previously proven against the user's real Nightscout instance (bash history on LXC 102 shows a full run against https://zb-ns.duckdns.org completing 9 months ago). This is exactly the "had to manually parse parts of the profile" pain point called out in FR2 — resolved by wrapping existing tools instead of reimplementing them.

LXC 107 is not used; it's a messier, native (non-Docker) install of the same underlying toolkit with no functional advantage over 102. It stays stopped.

2. Architecture overview

Two new pieces, one existing piece untouched:

┌─────────────────────────────┐        ┌──────────────────────────────────┐
│ LXC 120 (glucose-app)        │        │ LXC 102 (autotune)                │
│                              │        │                                    │
│ ┌──────────────────────────┐ │  HTTPS │ ┌────────────────────────────────┐│
│ │ glucose-dashboard         │ │  Bearer│ │ autotune-service (new)         ││
│ │ (existing, untouched)     │ │  token │ │ Node/Express, image FROM       ││
│ └──────────────────────────┘ │◄───────┼─┤ ghcr.io/houthacker/            ││
│                              │        │ │ nightscout-autotune             ││
│ ┌──────────────────────────┐ │        │ │  - owns the NS write token      ││
│ │ autotune-app (new)        │─┼────────┼─▶  - owns the SQLite audit DB    ││
│ │ Streamlit, own auth gate  │ │        │ │  - runs analyses (spawn)       ││
│ └──────────────────────────┘ │        │ │  - runs oref0-upload-profile    ││
│                              │        │ │  - nightly scheduler (04:00)    ││
└──────────────────────────────┘        │ └────────────────────────────────┘│
                                          │            │                     │
                                          └────────────┼─────────────────────┘
                                                        │ HTTPS (existing NS
                                                        │ read/write token)
                                                        ▼
                                              Nightscout (zb-ns.duckdns.org)

Key property driving this split: the frontend never talks to Nightscout directly, and never holds a Nightscout credential. It only holds a shared bearer token to call autotune-service. All Nightscout I/O — reading entries/treatments/profile for analysis, and writing the staged profile / Profile Switch — happens on LXC 102, inside autotune-service, using the one write-scoped NS token (NFR2). This keeps the dosing-adjacent credential in exactly one place, matches architecture decision #3 in the requirements doc (isolating write-capable code from the read-only dashboard), and means the Streamlit app can be redeployed or rebuilt at any time without touching secrets or state — it's a pure, stateless UI.

Repo layout: one new repo, glucose-profile-tuning, with two subdirectories and their own Dockerfiles:

• service/ — the Node microservice (deployed to LXC 102)
• app/ — the Streamlit frontend (deployed to LXC 120, alongside glucose-dashboard as a second docker-compose service — that LXC has ~800MB RAM headroom today, plenty for a second, mostly-idle Streamlit UI)

One repo keeps this a single reviewable unit (matching how the user framed FR1–FR9 as one feature); the two Dockerfiles/deploy targets keep the runtime isolation NFR1 asks for.

3. autotune-service (LXC 102)

3.1 Why Node, not Python

The base image is Alpine + Node 24 with the entire oref0 toolkit already installed as Node/shell scripts under /usr/local/bin. Adding Python + FastAPI on top would mean two language runtimes doing the same job one already does. Building the HTTP layer in Node (Express) lets it spawn() nightscout-autotune and oref0-upload-profile as sibling processes in the same container/filesystem — no Docker socket, no Docker-in-Docker. (Mounting the host's docker.sock to let a container launch sibling containers was considered and rejected — unnecessary privilege escalation for a homelab service that already has direct in-image access to the tools it needs.)

3.2 Job model

A single SQLite table, analysis_runs:

column	type	notes
id	TEXT (uuid)	PK
trigger	TEXT	manual | nightly
requested_at / started_at / finished_at	TEXT (ISO 8601)
status	TEXT	pending | running | done | failed
days, profile_name, autosens_min, autosens_max, insulin_type, min_5m_carb_impact, uam_as_basal	job params as submitted	profile_name nullable — omitted means "NS's current default", per the tool's own documented behavior
error_message	TEXT, nullable
current_profile_json / proposed_profile_json	TEXT (JSON), nullable until done	oref0 pump-profile shape (basalprofile[], isf, carbratio, dia, autosens_min/max) — this is the input profile.json and output newprofile.json the tool already produces
recommendations_log	TEXT, nullable	verbatim autotune_recommendations.log — human-readable Pump/Autotune/Days-missing table, used for FR4's estimation detail
staged_at	TEXT, nullable	set once FR6 staging succeeds
staged_profile_json	TEXT (JSON), nullable	the final merged profile actually sent to NS
switched	INTEGER (bool), nullable	whether --switch was used
staging_decisions_json	TEXT (JSON), nullable	per-slot accept/reject/custom record, for FR7 audit trail

Only one job may be pending/running at a time (LXC 102 is 1 core / 1GB — not built for concurrency); a second trigger while one is in flight gets 409 Conflict.

Job execution: POST /v1/analyses inserts a pending row, immediately returns {id, status: "pending"}, then asynchronously:

1. mark running, set started_at
2. spawn("nightscout-autotune", [], {env: {...NS_HOST, AUTOTUNE_DAYS, UAM_AS_BASAL, NS_TOKEN (sha1'd), NS_PROFILE, AUTOSENS_MIN/MAX, INSULIN_TYPE, MIN_5MIN_CARBIMPACT, OPENAPS_WORKDIR: "/data/jobs/<id>"}})
3. on exit code 0: read profile.json, newprofile.*.json, and autotune_recommendations.log out of the job's workdir, populate the row, mark done
4. on non-zero exit: capture stderr/stdout tail as error_message, mark failed

3.3 HTTP API

All endpoints (except /healthz) require Authorization: Bearer <AUTOTUNE_SERVICE_API_KEY>.

• POST /v1/analyses — body {days, profile_name?, autosens_min?, autosens_max?, insulin_type?, min_5m_carb_impact?, uam_as_basal, trigger: "manual"} → 202 {id, status} or 409 if one's already running.
• GET /v1/analyses/{id} — full row as JSON (shape above).
• GET /v1/analyses?trigger=&limit= — history list, newest first (backs FR7 and FR9's nightly-runs page).
• POST /v1/analyses/{id}/stage — body {profile: <merged oref0-shape JSON>, switch: bool, decisions: [...]} → runs oref0-upload-profile <tmpfile> $NS_HOST <secret> [--switch], updates the row, returns 200 or 502 with the CLI's stderr on failure. To verify during implementation: whether oref0-upload-profile's positional api-secret argument accepts a token interchangeably (the analyze side supports both NS_API_SECRET and NS_TOKEN; the upload side's --help only documents api-secret positionally) — confirm against the installed script's source before wiring this up, rather than assuming.
• GET /v1/profiles — thin passthrough listing NS profile-store names, so the frontend can offer a dropdown instead of a free-text field for FR3's "select a current profile."
• GET /healthz — liveness, no auth required (used by the frontend's connectivity check and Docker's own healthcheck).

3.4 Nightly scheduler (FR9)

node-cron inside the same process, firing daily at 04:00 (container timezone set to Europe/Amsterdam). Calls the same code path as POST /v1/analyses with trigger: "nightly", profile_name omitted (always tunes from whatever is currently active — correct behavior for an unattended run), and days from NIGHTLY_AUTOTUNE_DAYS (env var, default 7 — a reasonable middle ground between enough data for stable estimates and not diluting recent pattern changes; the doc's own wording — "whatever gives usually the most accurate results" — doesn't have one universally-correct answer, so this is exposed as a tunable env var rather than a hardcoded guess, and the user can adjust it after watching a few nights of real output). No result is ever auto-staged; a nightly run populates a done/ failed row exactly like a manual one, staging is always a separate FR6 action.

3.5 Logging (NFR "production grade")

Structured JSON lines to stdout via pino, one line per job lifecycle event (job.started, job.finished, job.failed with id/duration_ms), WARN for FR8 large-jump flags, ERROR for NS/upload failures — never logging the raw token/secret. Captured by Docker's json-file driver with max-size/max-file set in docker-compose.yml (no external log aggregator exists in the homelab today per inspection of the monitoring LXC, so this is deliberately self-contained rather than assuming a Loki/Elastic-style sink that isn't there).

4. autotune-app (LXC 120, Streamlit)

4.1 Framework decision (NFR5)

Streamlit, not Shiny for Python. Reasons: every functional requirement here (form inputs, polling with a spinner, a comparison table, an editable grid, history lists) is already something glucose-dashboard does or Streamlit does natively — st.data_editor with column_config.NumberColumn(step=0.01) directly implements FR5's "editable numeric fields, 0.01 steps" requirement without extra code. Shiny's reactive model would add a second UI paradigm for the user to maintain with no functional gain here. The password-gate (auth.py), theming, and AppTest-based testing conventions from glucose-dashboard carry over directly (copied into this new repo, not imported cross-repo, per NFR1's isolation).

4.2 Pages

1. app.py — landing page, require_auth(), reads AUTOTUNE_SERVICE_URL + AUTOTUNE_SERVICE_API_KEY from env, shows a connectivity check against GET /healthz.
2. pages/1_Nieuwe_Analyse.py (FR3) — form: days, starting profile (dropdown from GET /v1/profiles), an expander for advanced overrides (autosens min/max, insulin type, min 5-min carb impact). Submits POST /v1/analyses, then polls GET /v1/analyses/{id} (via st.spinner + a rerun loop) until done/failed; a clear NS-connection error surfaces the service's error_message verbatim (FR3 acceptance: fail loud, don't hang).
3. pages/2_Overzicht_Resultaat.py (FR4 + FR5 + FR6 + FR8) — given a job id: summary cards (ISF/CR pump vs. autotune, autosens_min/max bounds shown explicitly per FR8), an st.data_editor grid — columns time-slot (hourly), huidig profiel, voorstel, bron (select: current/suggested/ custom), definitieve waarde (editable number, step 0.01, disabled unless bron = custom) — plus a raw autotune_recommendations.log expander for the underlying estimation detail. Rows whose suggested value differs from current by more than a configurable threshold (default: 20% relative or 0.1 U/hr absolute, whichever is smaller magnitude) are visually flagged (FR8's "unusually large jump"). A "Stage to Nightscout" section requires an explicit confirmation checkbox before POST /v1/analyses/{id}/stage fires (FR8: no silent staging), with a separate "ook direct activeren (Profile Switch)" checkbox controlling switch.
4. pages/3_Nachtelijke_Runs.py (FR9) — GET /v1/analyses?trigger=nightly, latest first; failed rows show error_message inline; clicking a row opens it in the same view as page 2.
5. pages/4_Geschiedenis.py (FR7) — full audit list (manual + nightly), filterable by status/trigger, showing staged/switched state — this is the audit trail; no separate mechanism needed since every run (and every staging decision) is already a row in autotune-service's database.

4.3 Merge logic (the one piece of real business logic on this side)

merge_profile(current, proposed, decisions) -> final_profile — a pure function, unit-tested exactly like reporting.py's style in glucose-dashboard: for each slot, decisions[slot] is "current"/"suggested"/"custom" (+ a value when "custom"); rejecting a slot must leave current's value byte-for-byte in final_profile (FR5 acceptance criterion). This function, plus the large-jump-flagging function, are the only two pieces of logic in this app worth unit tests beyond AppTest page smoke tests.

5. Nightscout token (NFR2)

A new Subject/Access Token created in Nightscout's Admin Tools, distinct from glucose-dashboard's existing read-only token. Role: careportal — the conventional Nightscout role that grants treatment and profile writes on top of read access (vs. readable, read-only, or admin, full control this doesn't need). To confirm during implementation: exact role-string behavior can vary slightly by Nightscout version — verify a careportal-role token can actually write /api/v1/profile on this specific instance before relying on it, rather than assuming the classic-NS role semantics hold exactly. Only autotune-service's environment ever holds this token; it is never sent to or stored in the browser/frontend.

6. Data requirements — resolved vs. the original doc

The original requirements doc's §6 assumed nightscout_client.py-style GET/POST /api/v1/profile.json functions would need to be written. With the LXC 102 findings, that's no longer true for the write direction (oref0-upload-profile already does it) or the analysis-time read direction (nightscout-autotune already does it). The frontend's only remaining NS-ish need is the profile-name dropdown, served via autotune-service's GET /v1/profiles passthrough — so no Nightscout client code is written in Python at all for this feature. All NS-format handling stays inside the existing, already-proven Node/oref0 tooling.

7. Testing strategy (NFR3)

• service/: unit tests mock child_process.spawn and the filesystem — never invoke the real nightscout-autotune/oref0-upload-profile binaries or a real Nightscout in CI. Cover: job state transitions, the pending/running 409 guard, SQLite persistence (temp-file DB per test), route handlers, and the large-jump/merge-adjacent logic that lives here (none currently — kept in the frontend, see §4.3).
• app/: merge_profile/large-jump pure-function unit tests; AppTest page tests mocking requests against autotune-service (same FakeResponse/monkeypatch pattern as glucose-dashboard's suite); auth.py tests copied from the existing suite.
• UAT (post-implementation, against the real LXC 102 service and the user's real Nightscout): trigger a manual analysis → review the comparison/flagged-jump view → reject a slot and confirm it stays at the current value → accept + custom-edit others → stage without switch → manually confirm in Nightscout the new profile appears correctly and the previously active one is untouched → separately test staging with switch → confirm a nightly run appears on the nightly page the next morning, including a deliberately-forced-failure case showing a clear reason.

8. CI/CD (NFR1, NFR4)

New Gitea repo glucose-profile-tuning, pipeline modeled on glucose-dashboard/.gitea/workflows/pipeline.yml with two differences:

• Two build targets: service/Dockerfile (deploy → LXC 102) and app/Dockerfile (deploy → LXC 120, second compose service alongside glucose-dashboard), each independently Trivy-scanned and pushed to gitea.zb-kw.duckdns.org.
• pull_request trigger added to the test/lint job (NFR4) — unlike glucose-dashboard's current push-only pipeline, this one can end in a write to a real medical-data store, so tests must gate merge, not just run after the fact.

New secrets/variables to provision (mirroring the existing dashboard's setup): SSH deploy credentials for both target hosts, the new NS careportal token, and a freshly generated AUTOTUNE_SERVICE_API_KEY shared between the two new containers.

9. Implementation phasing

Same incremental, ship-behind-CI approach used for the reporting feature — each phase is a separate PR, merged only once green:

1. Scaffold — both Dockerfiles building, empty pipeline green end-to-end (test → Trivy → deploy to both LXCs), before any real feature — this phase alone would have caught the hostname/secret/token issues the reporting feature's deployment hit, ahead of time.
2. Microservice core (FR2) — SQLite schema, job runner wrapping nightscout-autotune, POST/GET /v1/analyses only, unit-tested with mocked spawn. No frontend yet — verified via curl against the real LXC 102 (matches FR2's own acceptance criterion).
3. Frontend read path (FR3 + FR4) — pages 1 + 2 up to the comparison view, no staging yet.
4. Staging (FR5 + FR6 + FR8) — st.data_editor merge grid, /stage endpoint, confirmation gating, large-jump flags.
5. Nightly + audit (FR9 + FR7) — scheduler, nightly-runs page, history page.

10. Open items carried into implementation (not blocking doc approval)

• Verify oref0-upload-profile's secret-vs-token argument behavior (§3.3).
• Verify the careportal role actually permits profile writes on this NS version before building against it (§5).
• Confirm the default NIGHTLY_AUTOTUNE_DAYS=7 after watching real output for a week or two; it's a tunable, not a fixed clinical constant (§3.4).