docs: record FR1 findings and add profile-tuning design spec
requirements-profile-tuning.md: document the LXC 102/107 investigation outcome (LXC 102's houthacker/nightscout-autotune image already solves both the analysis and NS-upload directions) and resolve the two open questions this depended on. design-profile-tuning.md: full design spec for the separate glucose-profile-tuning app/service, reviewed and approved.
This commit is contained in:
@@ -0,0 +1,328 @@
|
||||
> **Moved.** Implementation now lives in the separate `glucose-profile-tuning`
|
||||
> repo (per this doc's own §2 isolation decision); the canonical, actively
|
||||
> maintained copy of this file is at `glucose-profile-tuning/docs/design-profile-tuning.md`.
|
||||
> This copy is kept for history and is no longer updated.
|
||||
|
||||
# Design specification: oref0-autotune profile tuning
|
||||
|
||||
Status: **final — user reviewed**
|
||||
Requirements: [requirements-profile-tuning.md](requirements-profile-tuning.md) (FR1–FR9, NFR1–NFR5)
|
||||
Related: [design-reporting.md](design-reporting.md) (same conventions, separate app)
|
||||
|
||||
This is a design spec, not code. Nothing here has been implemented yet —
|
||||
implementation starts after this doc is reviewed/approved, one phase at a
|
||||
time, each shipped through its own CI/CD pipeline (§8).
|
||||
|
||||
## 1. FR1 recap — infra decision (already executed)
|
||||
|
||||
FR1 (boot + inspect LXC 102/107) is done; full findings are recorded directly
|
||||
in [requirements-profile-tuning.md §4 FR1](requirements-profile-tuning.md).
|
||||
Summary: **LXC 102 (`autotune`, 192.168.2.13)** already has a working, pinned
|
||||
Docker image — `ghcr.io/houthacker/nightscout-autotune:latest` — that bundles
|
||||
the *entire* oref0 npm toolkit, not just the autotune algorithm:
|
||||
|
||||
| Tool (already installed in the image) | Role in this design |
|
||||
|---|---|
|
||||
| `nightscout-autotune` (image's default CMD) | Fetches NS entries/treatments/profile, runs oref0-autotune, writes results to `$OPENAPS_WORKDIR` |
|
||||
| `oref0-upload-profile profile.json NSURL secret [--preview] [--switch]` | Converts an oref0-format profile back to Nightscout's profile-store format **and** uploads it, optionally issuing a Profile Switch treatment |
|
||||
| `convert-ns-profile` / `mm-format-ns-profile` | The NS-format ⇄ oref0-format conversion primitives, used internally by the two tools above |
|
||||
|
||||
This changes the shape of the whole feature for the better: **we do not need
|
||||
to write any Nightscout-profile-format conversion code ourselves.** Both
|
||||
directions (NS → oref0 for analysis, oref0 → NS for staging) are already
|
||||
solved, tested, and previously proven against the user's real Nightscout
|
||||
instance (bash history on LXC 102 shows a full run against
|
||||
`https://zb-ns.duckdns.org` completing 9 months ago). This is exactly the
|
||||
"had to manually parse parts of the profile" pain point called out in FR2 —
|
||||
resolved by wrapping existing tools instead of reimplementing them.
|
||||
|
||||
LXC 107 is not used; it's a messier, native (non-Docker) install of the same
|
||||
underlying toolkit with no functional advantage over 102. It stays stopped.
|
||||
|
||||
## 2. Architecture overview
|
||||
|
||||
Two new pieces, one existing piece untouched:
|
||||
|
||||
```
|
||||
┌─────────────────────────────┐ ┌──────────────────────────────────┐
|
||||
│ LXC 120 (glucose-app) │ │ LXC 102 (autotune) │
|
||||
│ │ │ │
|
||||
│ ┌──────────────────────────┐ │ HTTPS │ ┌────────────────────────────────┐│
|
||||
│ │ glucose-dashboard │ │ Bearer│ │ autotune-service (new) ││
|
||||
│ │ (existing, untouched) │ │ token │ │ Node/Express, image FROM ││
|
||||
│ └──────────────────────────┘ │◄───────┼─┤ ghcr.io/houthacker/ ││
|
||||
│ │ │ │ nightscout-autotune ││
|
||||
│ ┌──────────────────────────┐ │ │ │ - owns the NS write token ││
|
||||
│ │ autotune-app (new) │─┼────────┼─▶ - owns the SQLite audit DB ││
|
||||
│ │ Streamlit, own auth gate │ │ │ │ - runs analyses (spawn) ││
|
||||
│ └──────────────────────────┘ │ │ │ - runs oref0-upload-profile ││
|
||||
│ │ │ │ - nightly scheduler (04:00) ││
|
||||
└──────────────────────────────┘ │ └────────────────────────────────┘│
|
||||
│ │ │
|
||||
└────────────┼─────────────────────┘
|
||||
│ HTTPS (existing NS
|
||||
│ read/write token)
|
||||
▼
|
||||
Nightscout (zb-ns.duckdns.org)
|
||||
```
|
||||
|
||||
Key property driving this split: **the frontend never talks to Nightscout
|
||||
directly, and never holds a Nightscout credential.** It only holds a shared
|
||||
bearer token to call `autotune-service`. All Nightscout I/O — reading
|
||||
entries/treatments/profile for analysis, and writing the staged profile /
|
||||
Profile Switch — happens on LXC 102, inside `autotune-service`, using the one
|
||||
write-scoped NS token (NFR2). This keeps the dosing-adjacent credential in
|
||||
exactly one place, matches architecture decision #3 in the requirements doc
|
||||
(isolating write-capable code from the read-only dashboard), and means the
|
||||
Streamlit app can be redeployed or rebuilt at any time without touching
|
||||
secrets or state — it's a pure, stateless UI.
|
||||
|
||||
Repo layout: one new repo, **`glucose-profile-tuning`**, with two
|
||||
subdirectories and their own Dockerfiles:
|
||||
- `service/` — the Node microservice (deployed to LXC 102)
|
||||
- `app/` — the Streamlit frontend (deployed to LXC 120, alongside
|
||||
`glucose-dashboard` as a second `docker-compose` service — that LXC has
|
||||
~800MB RAM headroom today, plenty for a second, mostly-idle Streamlit UI)
|
||||
|
||||
One repo keeps this a single reviewable unit (matching how the user framed
|
||||
FR1–FR9 as one feature); the two Dockerfiles/deploy targets keep the runtime
|
||||
isolation NFR1 asks for.
|
||||
|
||||
## 3. `autotune-service` (LXC 102)
|
||||
|
||||
### 3.1 Why Node, not Python
|
||||
|
||||
The base image is Alpine + Node 24 with the entire oref0 toolkit already
|
||||
installed as Node/shell scripts under `/usr/local/bin`. Adding Python +
|
||||
FastAPI on top would mean two language runtimes doing the same job one
|
||||
already does. Building the HTTP layer in Node (Express) lets it `spawn()`
|
||||
`nightscout-autotune` and `oref0-upload-profile` as sibling processes in the
|
||||
same container/filesystem — no Docker socket, no Docker-in-Docker. (Mounting
|
||||
the host's `docker.sock` to let a container launch sibling containers was
|
||||
considered and rejected — unnecessary privilege escalation for a homelab
|
||||
service that already has direct in-image access to the tools it needs.)
|
||||
|
||||
### 3.2 Job model
|
||||
|
||||
A single SQLite table, `analysis_runs`:
|
||||
|
||||
| column | type | notes |
|
||||
|---|---|---|
|
||||
| `id` | TEXT (uuid) | PK |
|
||||
| `trigger` | TEXT | `manual` \| `nightly` |
|
||||
| `requested_at` / `started_at` / `finished_at` | TEXT (ISO 8601) | |
|
||||
| `status` | TEXT | `pending` \| `running` \| `done` \| `failed` |
|
||||
| `days`, `profile_name`, `autosens_min`, `autosens_max`, `insulin_type`, `min_5m_carb_impact`, `uam_as_basal` | job params as submitted | `profile_name` nullable — omitted means "NS's current default", per the tool's own documented behavior |
|
||||
| `error_message` | TEXT, nullable | |
|
||||
| `current_profile_json` / `proposed_profile_json` | TEXT (JSON), nullable until done | oref0 pump-profile shape (`basalprofile[]`, `isf`, `carbratio`, `dia`, `autosens_min/max`) — this is the *input* `profile.json` and *output* `newprofile.json` the tool already produces |
|
||||
| `recommendations_log` | TEXT, nullable | verbatim `autotune_recommendations.log` — human-readable Pump/Autotune/Days-missing table, used for FR4's estimation detail |
|
||||
| `staged_at` | TEXT, nullable | set once FR6 staging succeeds |
|
||||
| `staged_profile_json` | TEXT (JSON), nullable | the final merged profile actually sent to NS |
|
||||
| `switched` | INTEGER (bool), nullable | whether `--switch` was used |
|
||||
| `staging_decisions_json` | TEXT (JSON), nullable | per-slot accept/reject/custom record, for FR7 audit trail |
|
||||
|
||||
Only one job may be `pending`/`running` at a time (LXC 102 is 1 core / 1GB —
|
||||
not built for concurrency); a second trigger while one is in flight gets
|
||||
`409 Conflict`.
|
||||
|
||||
Job execution: `POST /v1/analyses` inserts a `pending` row, immediately
|
||||
returns `{id, status: "pending"}`, then asynchronously:
|
||||
1. mark `running`, set `started_at`
|
||||
2. `spawn("nightscout-autotune", [], {env: {...NS_HOST, AUTOTUNE_DAYS, UAM_AS_BASAL, NS_TOKEN (sha1'd), NS_PROFILE, AUTOSENS_MIN/MAX, INSULIN_TYPE, MIN_5MIN_CARBIMPACT, OPENAPS_WORKDIR: "/data/jobs/<id>"}})`
|
||||
3. on exit code 0: read `profile.json`, `newprofile.*.json`, and
|
||||
`autotune_recommendations.log` out of the job's workdir, populate the row,
|
||||
mark `done`
|
||||
4. on non-zero exit: capture stderr/stdout tail as `error_message`, mark
|
||||
`failed`
|
||||
|
||||
### 3.3 HTTP API
|
||||
|
||||
All endpoints (except `/healthz`) require `Authorization: Bearer
|
||||
<AUTOTUNE_SERVICE_API_KEY>`.
|
||||
|
||||
- `POST /v1/analyses` — body `{days, profile_name?, autosens_min?, autosens_max?, insulin_type?, min_5m_carb_impact?, uam_as_basal, trigger: "manual"}` → `202 {id, status}` or `409` if one's already running.
|
||||
- `GET /v1/analyses/{id}` — full row as JSON (shape above).
|
||||
- `GET /v1/analyses?trigger=&limit=` — history list, newest first (backs FR7 and FR9's nightly-runs page).
|
||||
- `POST /v1/analyses/{id}/stage` — body `{profile: <merged oref0-shape JSON>, switch: bool, decisions: [...]}` → runs `oref0-upload-profile <tmpfile> $NS_HOST <secret> [--switch]`, updates the row, returns `200` or `502` with the CLI's stderr on failure. **To verify during implementation:** whether `oref0-upload-profile`'s positional `api-secret` argument accepts a token interchangeably (the analyze side supports both `NS_API_SECRET` and `NS_TOKEN`; the upload side's `--help` only documents `api-secret` positionally) — confirm against the installed script's source before wiring this up, rather than assuming.
|
||||
- `GET /v1/profiles` — thin passthrough listing NS profile-store names, so the frontend can offer a dropdown instead of a free-text field for FR3's "select a current profile."
|
||||
- `GET /healthz` — liveness, no auth required (used by the frontend's connectivity check and Docker's own healthcheck).
|
||||
|
||||
### 3.4 Nightly scheduler (FR9)
|
||||
|
||||
`node-cron` inside the same process, firing daily at 04:00 (container
|
||||
timezone set to `Europe/Amsterdam`). Calls the same code path as
|
||||
`POST /v1/analyses` with `trigger: "nightly"`, `profile_name` omitted (always
|
||||
tunes from whatever is currently active — correct behavior for an unattended
|
||||
run), and `days` from `NIGHTLY_AUTOTUNE_DAYS` (env var, default **7** — a
|
||||
reasonable middle ground between enough data for stable estimates and not
|
||||
diluting recent pattern changes; the doc's own wording — "whatever gives
|
||||
usually the most accurate results" — doesn't have one universally-correct
|
||||
answer, so this is exposed as a tunable env var rather than a hardcoded
|
||||
guess, and the user can adjust it after watching a few nights of real
|
||||
output). No result is ever auto-staged; a nightly run populates a `done`/
|
||||
`failed` row exactly like a manual one, staging is always a separate FR6
|
||||
action.
|
||||
|
||||
### 3.5 Logging (NFR "production grade")
|
||||
|
||||
Structured JSON lines to stdout via `pino`, one line per job lifecycle event
|
||||
(`job.started`, `job.finished`, `job.failed` with `id`/`duration_ms`), `WARN`
|
||||
for FR8 large-jump flags, `ERROR` for NS/upload failures — **never** logging
|
||||
the raw token/secret. Captured by Docker's `json-file` driver with
|
||||
`max-size`/`max-file` set in `docker-compose.yml` (no external log
|
||||
aggregator exists in the homelab today per inspection of the `monitoring`
|
||||
LXC, so this is deliberately self-contained rather than assuming a
|
||||
Loki/Elastic-style sink that isn't there).
|
||||
|
||||
## 4. `autotune-app` (LXC 120, Streamlit)
|
||||
|
||||
### 4.1 Framework decision (NFR5)
|
||||
|
||||
**Streamlit**, not Shiny for Python. Reasons: every functional requirement
|
||||
here (form inputs, polling with a spinner, a comparison table, an editable
|
||||
grid, history lists) is already something `glucose-dashboard` does or
|
||||
Streamlit does natively — `st.data_editor` with
|
||||
`column_config.NumberColumn(step=0.01)` directly implements FR5's "editable
|
||||
numeric fields, 0.01 steps" requirement without extra code. Shiny's reactive
|
||||
model would add a second UI paradigm for the user to maintain with no
|
||||
functional gain here. The password-gate (`auth.py`), theming, and
|
||||
`AppTest`-based testing conventions from `glucose-dashboard` carry over
|
||||
directly (copied into this new repo, not imported cross-repo, per NFR1's
|
||||
isolation).
|
||||
|
||||
### 4.2 Pages
|
||||
|
||||
1. **`app.py`** — landing page, `require_auth()`, reads
|
||||
`AUTOTUNE_SERVICE_URL` + `AUTOTUNE_SERVICE_API_KEY` from env, shows a
|
||||
connectivity check against `GET /healthz`.
|
||||
2. **`pages/1_Nieuwe_Analyse.py`** (FR3) — form: days, starting profile
|
||||
(dropdown from `GET /v1/profiles`), an expander for advanced overrides
|
||||
(autosens min/max, insulin type, min 5-min carb impact). Submits
|
||||
`POST /v1/analyses`, then polls `GET /v1/analyses/{id}` (via
|
||||
`st.spinner` + a rerun loop) until `done`/`failed`; a clear NS-connection
|
||||
error surfaces the service's `error_message` verbatim (FR3 acceptance:
|
||||
fail loud, don't hang).
|
||||
3. **`pages/2_Overzicht_Resultaat.py`** (FR4 + FR5 + FR6 + FR8) — given a job
|
||||
id: summary cards (ISF/CR pump vs. autotune, `autosens_min`/`max` bounds
|
||||
shown explicitly per FR8), an `st.data_editor` grid — columns *time-slot*
|
||||
(hourly), *huidig profiel*, *voorstel*, *bron* (select: current/suggested/
|
||||
custom), *definitieve waarde* (editable number, step 0.01, disabled unless
|
||||
*bron* = custom) — plus a raw `autotune_recommendations.log` expander for
|
||||
the underlying estimation detail. Rows whose suggested value differs from
|
||||
current by more than a configurable threshold (default: 20% relative or
|
||||
0.1 U/hr absolute, whichever is smaller magnitude) are visually flagged
|
||||
(FR8's "unusually large jump"). A "Stage to Nightscout" section requires
|
||||
an explicit confirmation checkbox before `POST /v1/analyses/{id}/stage`
|
||||
fires (FR8: no silent staging), with a separate "ook direct activeren
|
||||
(Profile Switch)" checkbox controlling `switch`.
|
||||
4. **`pages/3_Nachtelijke_Runs.py`** (FR9) — `GET /v1/analyses?trigger=nightly`,
|
||||
latest first; failed rows show `error_message` inline; clicking a row
|
||||
opens it in the same view as page 2.
|
||||
5. **`pages/4_Geschiedenis.py`** (FR7) — full audit list (manual + nightly),
|
||||
filterable by status/trigger, showing staged/switched state — this *is*
|
||||
the audit trail; no separate mechanism needed since every run (and every
|
||||
staging decision) is already a row in `autotune-service`'s database.
|
||||
|
||||
### 4.3 Merge logic (the one piece of real business logic on this side)
|
||||
|
||||
`merge_profile(current, proposed, decisions) -> final_profile` — a pure
|
||||
function, unit-tested exactly like `reporting.py`'s style in
|
||||
`glucose-dashboard`: for each slot, `decisions[slot]` is
|
||||
`"current"`/`"suggested"`/`"custom"` (+ a value when `"custom"`); rejecting a
|
||||
slot must leave `current`'s value byte-for-byte in `final_profile`
|
||||
(FR5 acceptance criterion). This function, plus the large-jump-flagging
|
||||
function, are the only two pieces of logic in this app worth unit tests
|
||||
beyond `AppTest` page smoke tests.
|
||||
|
||||
## 5. Nightscout token (NFR2)
|
||||
|
||||
A new Subject/Access Token created in Nightscout's Admin Tools, distinct from
|
||||
`glucose-dashboard`'s existing read-only token. Role: **`careportal`** — the
|
||||
conventional Nightscout role that grants treatment and profile writes on top
|
||||
of read access (vs. `readable`, read-only, or `admin`, full control this
|
||||
doesn't need). **To confirm during implementation**: exact role-string
|
||||
behavior can vary slightly by Nightscout version — verify a `careportal`-role
|
||||
token can actually write `/api/v1/profile` on this specific instance before
|
||||
relying on it, rather than assuming the classic-NS role semantics hold
|
||||
exactly. Only `autotune-service`'s environment ever holds this token; it is
|
||||
never sent to or stored in the browser/frontend.
|
||||
|
||||
## 6. Data requirements — resolved vs. the original doc
|
||||
|
||||
The original requirements doc's §6 assumed `nightscout_client.py`-style
|
||||
`GET`/`POST /api/v1/profile.json` functions would need to be written. With
|
||||
the LXC 102 findings, that's no longer true for the write direction
|
||||
(`oref0-upload-profile` already does it) or the analysis-time read direction
|
||||
(`nightscout-autotune` already does it). The frontend's only remaining NS-ish
|
||||
need is the profile-name dropdown, served via `autotune-service`'s
|
||||
`GET /v1/profiles` passthrough — so **no Nightscout client code is written in
|
||||
Python at all** for this feature. All NS-format handling stays inside the
|
||||
existing, already-proven Node/oref0 tooling.
|
||||
|
||||
## 7. Testing strategy (NFR3)
|
||||
|
||||
- **`service/`**: unit tests mock `child_process.spawn` and the filesystem —
|
||||
never invoke the real `nightscout-autotune`/`oref0-upload-profile` binaries
|
||||
or a real Nightscout in CI. Cover: job state transitions, the
|
||||
pending/running 409 guard, SQLite persistence (temp-file DB per test),
|
||||
route handlers, and the large-jump/merge-adjacent logic that lives here
|
||||
(none currently — kept in the frontend, see §4.3).
|
||||
- **`app/`**: `merge_profile`/large-jump pure-function unit tests; `AppTest`
|
||||
page tests mocking `requests` against `autotune-service` (same
|
||||
`FakeResponse`/`monkeypatch` pattern as `glucose-dashboard`'s suite);
|
||||
`auth.py` tests copied from the existing suite.
|
||||
- **UAT** (post-implementation, against the real LXC 102 service and the
|
||||
user's real Nightscout): trigger a manual analysis → review the
|
||||
comparison/flagged-jump view → reject a slot and confirm it stays at the
|
||||
current value → accept + custom-edit others → stage without switch →
|
||||
manually confirm in Nightscout the new profile appears correctly and the
|
||||
previously active one is untouched → separately test staging *with*
|
||||
switch → confirm a nightly run appears on the nightly page the next
|
||||
morning, including a deliberately-forced-failure case showing a clear
|
||||
reason.
|
||||
|
||||
## 8. CI/CD (NFR1, NFR4)
|
||||
|
||||
New Gitea repo `glucose-profile-tuning`, pipeline modeled on
|
||||
`glucose-dashboard/.gitea/workflows/pipeline.yml` with two differences:
|
||||
- **Two build targets**: `service/Dockerfile` (deploy → LXC 102) and
|
||||
`app/Dockerfile` (deploy → LXC 120, second compose service alongside
|
||||
`glucose-dashboard`), each independently Trivy-scanned and pushed to
|
||||
`gitea.zb-kw.duckdns.org`.
|
||||
- **`pull_request` trigger added to the test/lint job** (NFR4) — unlike
|
||||
`glucose-dashboard`'s current `push`-only pipeline, this one can end in a
|
||||
write to a real medical-data store, so tests must gate merge, not just run
|
||||
after the fact.
|
||||
|
||||
New secrets/variables to provision (mirroring the existing dashboard's
|
||||
setup): SSH deploy credentials for both target hosts, the new NS
|
||||
`careportal` token, and a freshly generated `AUTOTUNE_SERVICE_API_KEY`
|
||||
shared between the two new containers.
|
||||
|
||||
## 9. Implementation phasing
|
||||
|
||||
Same incremental, ship-behind-CI approach used for the reporting feature —
|
||||
each phase is a separate PR, merged only once green:
|
||||
|
||||
1. **Scaffold** — both Dockerfiles building, empty pipeline green end-to-end
|
||||
(test → Trivy → deploy to both LXCs), before any real feature — this
|
||||
phase alone would have caught the hostname/secret/token issues the
|
||||
reporting feature's deployment hit, ahead of time.
|
||||
2. **Microservice core** (FR2) — SQLite schema, job runner wrapping
|
||||
`nightscout-autotune`, `POST`/`GET /v1/analyses` only, unit-tested with
|
||||
mocked `spawn`. No frontend yet — verified via `curl` against the real
|
||||
LXC 102 (matches FR2's own acceptance criterion).
|
||||
3. **Frontend read path** (FR3 + FR4) — pages 1 + 2 up to the comparison
|
||||
view, no staging yet.
|
||||
4. **Staging** (FR5 + FR6 + FR8) — `st.data_editor` merge grid, `/stage`
|
||||
endpoint, confirmation gating, large-jump flags.
|
||||
5. **Nightly + audit** (FR9 + FR7) — scheduler, nightly-runs page, history
|
||||
page.
|
||||
|
||||
## 10. Open items carried into implementation (not blocking doc approval)
|
||||
|
||||
- Verify `oref0-upload-profile`'s secret-vs-token argument behavior (§3.3).
|
||||
- Verify the `careportal` role actually permits profile writes on this NS
|
||||
version before building against it (§5).
|
||||
- Confirm the default `NIGHTLY_AUTOTUNE_DAYS=7` after watching real output
|
||||
for a week or two; it's a tunable, not a fixed clinical constant (§3.4).
|
||||
Reference in New Issue
Block a user