glucose-dashboard/docs/design-profile-tuning.md

> **Moved.** Implementation now lives in the separate `glucose-profile-tuning`
> repo (per this doc's own §2 isolation decision); the canonical, actively
> maintained copy of this file is at `glucose-profile-tuning/docs/design-profile-tuning.md`.
> This copy is kept for history and is no longer updated.

# Design specification: oref0-autotune profile tuning

Status: **final — user reviewed**
Requirements: [requirements-profile-tuning.md](requirements-profile-tuning.md) (FR1–FR9, NFR1–NFR5)
Related: [design-reporting.md](design-reporting.md) (same conventions, separate app)

This is a design spec, not code. Nothing here has been implemented yet —
implementation starts after this doc is reviewed/approved, one phase at a
time, each shipped through its own CI/CD pipeline (§8).

## 1. FR1 recap — infra decision (already executed)

FR1 (boot + inspect LXC 102/107) is done; full findings are recorded directly
in [requirements-profile-tuning.md §4 FR1](requirements-profile-tuning.md).
Summary: **LXC 102 (`autotune`, 192.168.2.13)** already has a working, pinned
Docker image — `ghcr.io/houthacker/nightscout-autotune:latest` — that bundles
the *entire* oref0 npm toolkit, not just the autotune algorithm:

| Tool (already installed in the image) | Role in this design |
|---|---|
| `nightscout-autotune` (image's default CMD) | Fetches NS entries/treatments/profile, runs oref0-autotune, writes results to `$OPENAPS_WORKDIR` |
| `oref0-upload-profile profile.json NSURL secret [--preview] [--switch]` | Converts an oref0-format profile back to Nightscout's profile-store format **and** uploads it, optionally issuing a Profile Switch treatment |
| `convert-ns-profile` / `mm-format-ns-profile` | The NS-format ⇄ oref0-format conversion primitives, used internally by the two tools above |

This changes the shape of the whole feature for the better: **we do not need
to write any Nightscout-profile-format conversion code ourselves.** Both
directions (NS → oref0 for analysis, oref0 → NS for staging) are already
solved, tested, and previously proven against the user's real Nightscout
instance (bash history on LXC 102 shows a full run against
`https://zb-ns.duckdns.org` completing 9 months ago). This is exactly the
"had to manually parse parts of the profile" pain point called out in FR2 —
resolved by wrapping existing tools instead of reimplementing them.

LXC 107 is not used; it's a messier, native (non-Docker) install of the same
underlying toolkit with no functional advantage over 102. It stays stopped.

## 2. Architecture overview

Two new pieces, one existing piece untouched:

```
┌─────────────────────────────┐        ┌──────────────────────────────────┐
│ LXC 120 (glucose-app)        │        │ LXC 102 (autotune)                │
│                              │        │                                    │
│ ┌──────────────────────────┐ │  HTTPS │ ┌────────────────────────────────┐│
│ │ glucose-dashboard         │ │  Bearer│ │ autotune-service (new)         ││
│ │ (existing, untouched)     │ │  token │ │ Node/Express, image FROM       ││
│ └──────────────────────────┘ │◄───────┼─┤ ghcr.io/houthacker/            ││
│                              │        │ │ nightscout-autotune             ││
│ ┌──────────────────────────┐ │        │ │  - owns the NS write token      ││
│ │ autotune-app (new)        │─┼────────┼─▶  - owns the SQLite audit DB    ││
│ │ Streamlit, own auth gate  │ │        │ │  - runs analyses (spawn)       ││
│ └──────────────────────────┘ │        │ │  - runs oref0-upload-profile    ││
│                              │        │ │  - nightly scheduler (04:00)    ││
└──────────────────────────────┘        │ └────────────────────────────────┘│
                                          │            │                     │
                                          └────────────┼─────────────────────┘
                                                        │ HTTPS (existing NS
                                                        │ read/write token)
                                                        ▼
                                              Nightscout (zb-ns.duckdns.org)
```

Key property driving this split: **the frontend never talks to Nightscout
directly, and never holds a Nightscout credential.** It only holds a shared
bearer token to call `autotune-service`. All Nightscout I/O — reading
entries/treatments/profile for analysis, and writing the staged profile /
Profile Switch — happens on LXC 102, inside `autotune-service`, using the one
write-scoped NS token (NFR2). This keeps the dosing-adjacent credential in
exactly one place, matches architecture decision #3 in the requirements doc
(isolating write-capable code from the read-only dashboard), and means the
Streamlit app can be redeployed or rebuilt at any time without touching
secrets or state — it's a pure, stateless UI.

Repo layout: one new repo, **`glucose-profile-tuning`**, with two
subdirectories and their own Dockerfiles:
- `service/` — the Node microservice (deployed to LXC 102)
- `app/` — the Streamlit frontend (deployed to LXC 120, alongside
  `glucose-dashboard` as a second `docker-compose` service — that LXC has
  ~800MB RAM headroom today, plenty for a second, mostly-idle Streamlit UI)

One repo keeps this a single reviewable unit (matching how the user framed
FR1–FR9 as one feature); the two Dockerfiles/deploy targets keep the runtime
isolation NFR1 asks for.

## 3. `autotune-service` (LXC 102)

### 3.1 Why Node, not Python

The base image is Alpine + Node 24 with the entire oref0 toolkit already
installed as Node/shell scripts under `/usr/local/bin`. Adding Python +
FastAPI on top would mean two language runtimes doing the same job one
already does. Building the HTTP layer in Node (Express) lets it `spawn()`
`nightscout-autotune` and `oref0-upload-profile` as sibling processes in the
same container/filesystem — no Docker socket, no Docker-in-Docker. (Mounting
the host's `docker.sock` to let a container launch sibling containers was
considered and rejected — unnecessary privilege escalation for a homelab
service that already has direct in-image access to the tools it needs.)

### 3.2 Job model

A single SQLite table, `analysis_runs`:

| column | type | notes |
|---|---|---|
| `id` | TEXT (uuid) | PK |
| `trigger` | TEXT | `manual` \| `nightly` |
| `requested_at` / `started_at` / `finished_at` | TEXT (ISO 8601) | |
| `status` | TEXT | `pending` \| `running` \| `done` \| `failed` |
| `days`, `profile_name`, `autosens_min`, `autosens_max`, `insulin_type`, `min_5m_carb_impact`, `uam_as_basal` | job params as submitted | `profile_name` nullable — omitted means "NS's current default", per the tool's own documented behavior |
| `error_message` | TEXT, nullable | |
| `current_profile_json` / `proposed_profile_json` | TEXT (JSON), nullable until done | oref0 pump-profile shape (`basalprofile[]`, `isf`, `carbratio`, `dia`, `autosens_min/max`) — this is the *input* `profile.json` and *output* `newprofile.json` the tool already produces |
| `recommendations_log` | TEXT, nullable | verbatim `autotune_recommendations.log` — human-readable Pump/Autotune/Days-missing table, used for FR4's estimation detail |
| `staged_at` | TEXT, nullable | set once FR6 staging succeeds |
| `staged_profile_json` | TEXT (JSON), nullable | the final merged profile actually sent to NS |
| `switched` | INTEGER (bool), nullable | whether `--switch` was used |
| `staging_decisions_json` | TEXT (JSON), nullable | per-slot accept/reject/custom record, for FR7 audit trail |

Only one job may be `pending`/`running` at a time (LXC 102 is 1 core / 1GB —
not built for concurrency); a second trigger while one is in flight gets
`409 Conflict`.

Job execution: `POST /v1/analyses` inserts a `pending` row, immediately
returns `{id, status: "pending"}`, then asynchronously:
1. mark `running`, set `started_at`
2. `spawn("nightscout-autotune", [], {env: {...NS_HOST, AUTOTUNE_DAYS, UAM_AS_BASAL, NS_TOKEN (sha1'd), NS_PROFILE, AUTOSENS_MIN/MAX, INSULIN_TYPE, MIN_5MIN_CARBIMPACT, OPENAPS_WORKDIR: "/data/jobs/<id>"}})`
3. on exit code 0: read `profile.json`, `newprofile.*.json`, and
   `autotune_recommendations.log` out of the job's workdir, populate the row,
   mark `done`
4. on non-zero exit: capture stderr/stdout tail as `error_message`, mark
   `failed`

### 3.3 HTTP API

All endpoints (except `/healthz`) require `Authorization: Bearer
<AUTOTUNE_SERVICE_API_KEY>`.

- `POST /v1/analyses` — body `{days, profile_name?, autosens_min?, autosens_max?, insulin_type?, min_5m_carb_impact?, uam_as_basal, trigger: "manual"}` → `202 {id, status}` or `409` if one's already running.
- `GET /v1/analyses/{id}` — full row as JSON (shape above).
- `GET /v1/analyses?trigger=&limit=` — history list, newest first (backs FR7 and FR9's nightly-runs page).
- `POST /v1/analyses/{id}/stage` — body `{profile: <merged oref0-shape JSON>, switch: bool, decisions: [...]}` → runs `oref0-upload-profile <tmpfile> $NS_HOST <secret> [--switch]`, updates the row, returns `200` or `502` with the CLI's stderr on failure. **To verify during implementation:** whether `oref0-upload-profile`'s positional `api-secret` argument accepts a token interchangeably (the analyze side supports both `NS_API_SECRET` and `NS_TOKEN`; the upload side's `--help` only documents `api-secret` positionally) — confirm against the installed script's source before wiring this up, rather than assuming.
- `GET /v1/profiles` — thin passthrough listing NS profile-store names, so the frontend can offer a dropdown instead of a free-text field for FR3's "select a current profile."
- `GET /healthz` — liveness, no auth required (used by the frontend's connectivity check and Docker's own healthcheck).

### 3.4 Nightly scheduler (FR9)

`node-cron` inside the same process, firing daily at 04:00 (container
timezone set to `Europe/Amsterdam`). Calls the same code path as
`POST /v1/analyses` with `trigger: "nightly"`, `profile_name` omitted (always
tunes from whatever is currently active — correct behavior for an unattended
run), and `days` from `NIGHTLY_AUTOTUNE_DAYS` (env var, default **7** — a
reasonable middle ground between enough data for stable estimates and not
diluting recent pattern changes; the doc's own wording — "whatever gives
usually the most accurate results" — doesn't have one universally-correct
answer, so this is exposed as a tunable env var rather than a hardcoded
guess, and the user can adjust it after watching a few nights of real
output). No result is ever auto-staged; a nightly run populates a `done`/
`failed` row exactly like a manual one, staging is always a separate FR6
action.

### 3.5 Logging (NFR "production grade")

Structured JSON lines to stdout via `pino`, one line per job lifecycle event
(`job.started`, `job.finished`, `job.failed` with `id`/`duration_ms`), `WARN`
for FR8 large-jump flags, `ERROR` for NS/upload failures — **never** logging
the raw token/secret. Captured by Docker's `json-file` driver with
`max-size`/`max-file` set in `docker-compose.yml` (no external log
aggregator exists in the homelab today per inspection of the `monitoring`
LXC, so this is deliberately self-contained rather than assuming a
Loki/Elastic-style sink that isn't there).

## 4. `autotune-app` (LXC 120, Streamlit)

### 4.1 Framework decision (NFR5)

**Streamlit**, not Shiny for Python. Reasons: every functional requirement
here (form inputs, polling with a spinner, a comparison table, an editable
grid, history lists) is already something `glucose-dashboard` does or
Streamlit does natively — `st.data_editor` with
`column_config.NumberColumn(step=0.01)` directly implements FR5's "editable
numeric fields, 0.01 steps" requirement without extra code. Shiny's reactive
model would add a second UI paradigm for the user to maintain with no
functional gain here. The password-gate (`auth.py`), theming, and
`AppTest`-based testing conventions from `glucose-dashboard` carry over
directly (copied into this new repo, not imported cross-repo, per NFR1's
isolation).

### 4.2 Pages

1. **`app.py`** — landing page, `require_auth()`, reads
   `AUTOTUNE_SERVICE_URL` + `AUTOTUNE_SERVICE_API_KEY` from env, shows a
   connectivity check against `GET /healthz`.
2. **`pages/1_Nieuwe_Analyse.py`** (FR3) — form: days, starting profile
   (dropdown from `GET /v1/profiles`), an expander for advanced overrides
   (autosens min/max, insulin type, min 5-min carb impact). Submits
   `POST /v1/analyses`, then polls `GET /v1/analyses/{id}` (via
   `st.spinner` + a rerun loop) until `done`/`failed`; a clear NS-connection
   error surfaces the service's `error_message` verbatim (FR3 acceptance:
   fail loud, don't hang).
3. **`pages/2_Overzicht_Resultaat.py`** (FR4 + FR5 + FR6 + FR8) — given a job
   id: summary cards (ISF/CR pump vs. autotune, `autosens_min`/`max` bounds
   shown explicitly per FR8), an `st.data_editor` grid — columns *time-slot*
   (hourly), *huidig profiel*, *voorstel*, *bron* (select: current/suggested/
   custom), *definitieve waarde* (editable number, step 0.01, disabled unless
   *bron* = custom) — plus a raw `autotune_recommendations.log` expander for
   the underlying estimation detail. Rows whose suggested value differs from
   current by more than a configurable threshold (default: 20% relative or
   0.1 U/hr absolute, whichever is smaller magnitude) are visually flagged
   (FR8's "unusually large jump"). A "Stage to Nightscout" section requires
   an explicit confirmation checkbox before `POST /v1/analyses/{id}/stage`
   fires (FR8: no silent staging), with a separate "ook direct activeren
   (Profile Switch)" checkbox controlling `switch`.
4. **`pages/3_Nachtelijke_Runs.py`** (FR9) — `GET /v1/analyses?trigger=nightly`,
   latest first; failed rows show `error_message` inline; clicking a row
   opens it in the same view as page 2.
5. **`pages/4_Geschiedenis.py`** (FR7) — full audit list (manual + nightly),
   filterable by status/trigger, showing staged/switched state — this *is*
   the audit trail; no separate mechanism needed since every run (and every
   staging decision) is already a row in `autotune-service`'s database.

### 4.3 Merge logic (the one piece of real business logic on this side)

`merge_profile(current, proposed, decisions) -> final_profile` — a pure
function, unit-tested exactly like `reporting.py`'s style in
`glucose-dashboard`: for each slot, `decisions[slot]` is
`"current"`/`"suggested"`/`"custom"` (+ a value when `"custom"`); rejecting a
slot must leave `current`'s value byte-for-byte in `final_profile`
(FR5 acceptance criterion). This function, plus the large-jump-flagging
function, are the only two pieces of logic in this app worth unit tests
beyond `AppTest` page smoke tests.

## 5. Nightscout token (NFR2)

A new Subject/Access Token created in Nightscout's Admin Tools, distinct from
`glucose-dashboard`'s existing read-only token. Role: **`careportal`** — the
conventional Nightscout role that grants treatment and profile writes on top
of read access (vs. `readable`, read-only, or `admin`, full control this
doesn't need). **To confirm during implementation**: exact role-string
behavior can vary slightly by Nightscout version — verify a `careportal`-role
token can actually write `/api/v1/profile` on this specific instance before
relying on it, rather than assuming the classic-NS role semantics hold
exactly. Only `autotune-service`'s environment ever holds this token; it is
never sent to or stored in the browser/frontend.

## 6. Data requirements — resolved vs. the original doc

The original requirements doc's §6 assumed `nightscout_client.py`-style
`GET`/`POST /api/v1/profile.json` functions would need to be written. With
the LXC 102 findings, that's no longer true for the write direction
(`oref0-upload-profile` already does it) or the analysis-time read direction
(`nightscout-autotune` already does it). The frontend's only remaining NS-ish
need is the profile-name dropdown, served via `autotune-service`'s
`GET /v1/profiles` passthrough — so **no Nightscout client code is written in
Python at all** for this feature. All NS-format handling stays inside the
existing, already-proven Node/oref0 tooling.

## 7. Testing strategy (NFR3)

- **`service/`**: unit tests mock `child_process.spawn` and the filesystem —
  never invoke the real `nightscout-autotune`/`oref0-upload-profile` binaries
  or a real Nightscout in CI. Cover: job state transitions, the
  pending/running 409 guard, SQLite persistence (temp-file DB per test),
  route handlers, and the large-jump/merge-adjacent logic that lives here
  (none currently — kept in the frontend, see §4.3).
- **`app/`**: `merge_profile`/large-jump pure-function unit tests; `AppTest`
  page tests mocking `requests` against `autotune-service` (same
  `FakeResponse`/`monkeypatch` pattern as `glucose-dashboard`'s suite);
  `auth.py` tests copied from the existing suite.
- **UAT** (post-implementation, against the real LXC 102 service and the
  user's real Nightscout): trigger a manual analysis → review the
  comparison/flagged-jump view → reject a slot and confirm it stays at the
  current value → accept + custom-edit others → stage without switch →
  manually confirm in Nightscout the new profile appears correctly and the
  previously active one is untouched → separately test staging *with*
  switch → confirm a nightly run appears on the nightly page the next
  morning, including a deliberately-forced-failure case showing a clear
  reason.

## 8. CI/CD (NFR1, NFR4)

New Gitea repo `glucose-profile-tuning`, pipeline modeled on
`glucose-dashboard/.gitea/workflows/pipeline.yml` with two differences:
- **Two build targets**: `service/Dockerfile` (deploy → LXC 102) and
  `app/Dockerfile` (deploy → LXC 120, second compose service alongside
  `glucose-dashboard`), each independently Trivy-scanned and pushed to
  `gitea.zb-kw.duckdns.org`.
- **`pull_request` trigger added to the test/lint job** (NFR4) — unlike
  `glucose-dashboard`'s current `push`-only pipeline, this one can end in a
  write to a real medical-data store, so tests must gate merge, not just run
  after the fact.

New secrets/variables to provision (mirroring the existing dashboard's
setup): SSH deploy credentials for both target hosts, the new NS
`careportal` token, and a freshly generated `AUTOTUNE_SERVICE_API_KEY`
shared between the two new containers.

## 9. Implementation phasing

Same incremental, ship-behind-CI approach used for the reporting feature —
each phase is a separate PR, merged only once green:

1. **Scaffold** — both Dockerfiles building, empty pipeline green end-to-end
   (test → Trivy → deploy to both LXCs), before any real feature — this
   phase alone would have caught the hostname/secret/token issues the
   reporting feature's deployment hit, ahead of time.
2. **Microservice core** (FR2) — SQLite schema, job runner wrapping
   `nightscout-autotune`, `POST`/`GET /v1/analyses` only, unit-tested with
   mocked `spawn`. No frontend yet — verified via `curl` against the real
   LXC 102 (matches FR2's own acceptance criterion).
3. **Frontend read path** (FR3 + FR4) — pages 1 + 2 up to the comparison
   view, no staging yet.
4. **Staging** (FR5 + FR6 + FR8) — `st.data_editor` merge grid, `/stage`
   endpoint, confirmation gating, large-jump flags.
5. **Nightly + audit** (FR9 + FR7) — scheduler, nightly-runs page, history
   page.

## 10. Open items carried into implementation (not blocking doc approval)

- Verify `oref0-upload-profile`'s secret-vs-token argument behavior (§3.3).
- Verify the `careportal` role actually permits profile writes on this NS
  version before building against it (§5).
- Confirm the default `NIGHTLY_AUTOTUNE_DAYS=7` after watching real output
  for a week or two; it's a tunable, not a fixed clinical constant (§3.4).