glucose-dashboard/docs/requirements-profile-tuning.md

> **Moved.** Implementation now lives in the separate `glucose-profile-tuning`
> repo; the canonical, actively maintained copy of this file is at
> `glucose-profile-tuning/docs/requirements-profile-tuning.md`. This copy is
> kept for history and is no longer updated.

# Requirements: oref0-autotune profile tuning

Status: **final**  
Owner: willemjan  
Related: [requirements-reporting.md](requirements-reporting.md) (separate feature, same Nightscout data source)

## 1. Purpose & background

AndroidAPS (AAPS) only gained built-in Autotune in version 3.4, and even that
version tunes only a single ISF/carb-ratio value plus the basal schedule from
on-device history — it doesn't use Nightscout at all, and the user is still on
AAPS 3.3 regardless. Previously, the user ran a hosted tool (inspired by
`MarkMpn/AutotuneWeb`, now archived/discontinued) that analyzed Nightscout data
with `oref0-autotune` and proposed a new pump profile, which was then manually
entered into AAPS — a process the user describes as "a lot of trial and error." Before that, the user ran the tool from `MarkMpn/AutotuneWeb` but that was discontinued and cost a lot of effort to make changes to the profile by manually entering the changes into AAPS.

This feature builds a guided, human-in-the-loop workflow: trigger an
oref0-autotune analysis from an internally self hosted application, review the proposed profile against
the current one with the underlying estimations visible, selectively
accept/adjust values, and stage the result in Nightscout for manual activation.

The user has two LXC containers on Proxmox from earlier experimentation —
`autotune` (VMID 102, 192.168.2.13) and `oref0` (VMID 107, 192.168.2.14) —
currently stopped, of unknown/half-remembered working state. The working container can be used for the profile calculations, but if another solutions surpasses in terms of integratability a new solutions can be set up.

## 2. Scope

**In scope:**
- Booting and inspecting the two existing LXCs to determine (or establish) a
  working oref0-autotune setup.
- A small HTTP microservice on that LXC that runs oref0-autotune against the
  user's Nightscout data on demand. Either one of the two existing LXCs or a new one.
- A **separate** app/service (own deployment, own CI/CD pipeline) that
  triggers the analysis, displays old-vs-new profile comparisons with
  estimation detail, lets the user selectively adjust proposed values, and
  stages the result in Nightscout. The user can then choose if the profile is automatically activated in AndroidAPS (creating a
  `"Profile Switch"` treatment)
- This app could be based on the glucose-dashboard streamlit app, but streamlit is not a fixed requirement, another solution can be used. For example, shiny for python.
- A nightly scheduled autotune run of which the results are visible in the morning. The profile of each run should be stored in a database. The user can read those back from the database in the application and can send it to nightscout/androidAPS, but the nightly scheduled runs should not send profiles automatically to nightscout and/or AAPS, this should always be a user action.

**Out of scope (this iteration):**
- Any change to AAPS settings or on-device configuration.

## 3. Architecture decisions (made during spec review)

1. **Execution**: an always-on HTTP microservice on whichever LXC has (or gets)
   a working oref0-autotune setup, rather than SSH-triggering a one-off Docker
   run or bundling autotune into the dashboard's own container. The dashboard
   calls it over HTTP and polls for job completion.
2. **Profile push safety**: **stage-only**. The service writes a new named
   profile into the Nightscout profile store and does create a
   `"Profile Switch"` treatment when the user selected that option — final activation stays a manual
   step in AAPS when the option is not selected. This matters because AndroidAPS applies
   profiles via Profile Switch treatments (gated behind an explicit "accept
   profile switch from Nightscout" AAPS setting), not by simply reading
   `profile.json`.
3. **Deployment**: a separate app/service from `glucose-dashboard`, with its own
   repo-or-directory, Dockerfile, and Gitea CI/CD pipeline — isolating
   write-capable, dosing-adjacent code from the read-only dashboard.

## 4. Functional requirements

### FR1 — Infra investigation (first task, needs explicit go-ahead)
Boot LXC 102 (`autotune`) and LXC 107 (`oref0`) and inspect what's installed
(Docker containers, cron jobs, scripts, config, any README/notes left behind)
to determine which one — if either — has a working oref0-autotune setup.
Document findings (what's there, what's missing, what needs rebuilding).

*Acceptance:* a clear decision on which LXC hosts the microservice, and what
(if anything) needs to be installed/fixed there before FR2 can start.

**Status: done (2026-07-26).** Both LXCs were booted and inspected, then
stopped again (back to their original state — no state was left running).

- **LXC 102 (`autotune`, 192.168.2.13, 1 core / 1GB RAM / 16GB disk)** —
  Docker installed, with the image `ghcr.io/houthacker/nightscout-autotune:latest`
  (Alpine + Node 24, ~279MB) already pulled and previously run successfully
  (9 months ago) against the real Nightscout instance
  (`https://zb-ns.duckdns.org`). It bundles the full `oref0` npm toolkit
  (`oref0-autotune`, `oref0-upload-profile`, `convert-ns-profile`,
  `mm-format-ns-profile`, and more) under `/usr/local/bin`, not just the
  autotune algorithm itself. Its documented interface is a single container
  run with env vars: `NS_HOST`, `AUTOTUNE_DAYS`, `UAM_AS_BASAL` (mandatory),
  plus optional `NS_TOKEN`/`NS_API_SECRET` (as sha1 hashes), `NS_PROFILE`,
  `AUTOSENS_MIN`/`MAX`, `MIN_5MIN_CARBIMPACT`, `INSULIN_TYPE`,
  `OPENAPS_WORKDIR`, `HTML_EXPORT`. A prior run's output is still on disk at
  `/root/autotune-output/autotune/`: per-day `ns-entries`/`ns-treatments`
  fetched from Nightscout, the oref0-format `profile.json` (input) and
  `newprofile.YYYY-MM-DD.json` (recommendation), and a human-readable
  `autotune_recommendations.log` (a `Pump | Autotune | Days Missing` table per
  parameter/basal-slot — ideal source for FR4's estimation detail). Critically,
  the image also has `oref0-upload-profile` (`profile.json NSURL api-secret
  [--preview] [--switch]`) already installed — this is the piece that converts
  an oref0-format profile back into Nightscout's profile-store format *and*
  optionally issues the Profile Switch treatment, i.e. it already implements
  what FR6 needs. `--preview` (dry run, no upload) is also useful for FR8's
  confirmation step.
- **LXC 107 (`oref0`, 192.168.2.14, same specs)** — no Docker; a native
  npm/`openaps` install of the raw `oref0` toolkit. Bash history shows a
  messier path to a working run (locale errors needing `locale-gen`, several
  retries with different flag spellings) before landing on a working
  `oref0-autotune --dir=~/myopenaps --ns-host=... --start-date=...` invocation
  and a manual `oref0-upload-profile ... --switch=true` call. It reaches the
  same destination but through an unreproducible, hand-patched native
  environment rather than a pinned, documented container image.

**Decision: LXC 102 is the base.** It already has a working, versioned,
well-documented container with both the analysis direction (Nightscout →
oref0 profile → recommendation) and the upload direction
(oref0 recommendation → Nightscout profile store, with optional Profile
Switch) solved, via tools already installed in the same image. No new
autotune image needs to be built or evaluated — the open question in §8 about
choosing between `p5nbTgip0r/autotune-docker` /
`houthacker42/nightscout-autotune` / a from-scratch `oref0` build is moot:
`houthacker/nightscout-autotune` is what's already there, already proven
against this Nightscout instance, and turns out to be the most complete of
the three. LXC 107 will be left stopped/decommissioned — its only asset (a
successful manual run) is already superseded by 102's cleaner path. FR2's
microservice will be built as a thin wrapper *around* the CLI tools already
baked into the `nightscout-autotune` image (see
[design-profile-tuning.md](design-profile-tuning.md)), rather than
reimplementing Nightscout-profile ⇄ oref0-profile conversion from scratch —
directly resolving the "had to manually parse parts of the profile" pain
point named in FR2.

### FR2 — Autotune HTTP microservice
A small HTTP API (exact framework TBD during design — e.g. Flask/FastAPI
wrapping a Docker-based oref0-autotune image) exposing at minimum:
- `POST /analyze` — body: Nightscout URL/token, date range, (optionally) the
  starting profile to tune from. Returns a job ID.
- `GET /analyze/{job_id}` — status (`pending`/`running`/`done`/`failed`) and,
  once done, the recommended profile JSON plus the estimation/recommendation
  detail (per-hour deviations, the autotune log).
- In previous tries, the user had to manually parse parts of the profile as obtained from Nightscout to have it accpeted by the autotune service. This should be thoroughly investigated and subsequently automated

*Acceptance:* a manual `curl` against the running service, pointed at the
user's real Nightscout instance, returns a plausible recommended profile within
a reasonable time (minutes, not seconds — oref0-autotune is not instant).

### FR3 — Trigger analysis from the dashboard
A new app where the user picks a date range and selects a
current profile to tune from (fetched live from Nightscout), starts the
analysis, and sees progress while the microservice job runs (via polling).
Nightscout connection should be made using a token with a proper role.

*Acceptance:* starting an analysis with bad/unreachable Nightscout credentials
fails with a clear error rather than hanging silently.

### FR4 — Old-vs-new profile comparison view
Side-by-side display of the current and proposed profile: hourly basal rates,
ISF schedule, carb ratio schedule, DIA — plus the underlying estimation detail
(per-hour deviations, confidence notes) surfaced from the autotune log, so the
user can judge *why* a change is being proposed, not just see the new number.

*Acceptance:* every changed value is visually distinguishable from unchanged
values; the estimation detail for a given time slot is viewable alongside that
slot's proposed change.

### FR5 — Selective adjustment
Per time-slot controls (accept/reject each proposed basal/ISF/CR change
individually) plus editable numeric fields so the user can fine-tune any
accepted value before staging — matching the user's explicit request to
"select parts of the proposed profile and still adjust the values a bit."
This should look like a table with the columns 'time-slot' (per hour), 'current profile', 'suggestion', a input to select which column to select (current vs suggestion) and on the last column the final selected value, but as input numeric fields which can be changed by 0.01 steps.

*Acceptance:* rejecting a slot keeps the current value unchanged in the
staged profile; editing an accepted value's number is reflected in what
actually gets sent to Nightscout.

### FR6 — Stage to Nightscout
Write the resulting merged profile (accepted + manually adjusted values) as a
new named profile in the Nightscout profile store (e.g. `"Autotune
YYYY-MM-DD"` or `"Autotune YYYY-MM-DD ID"`) via a new POST to `/api/v1/profile`, with optionally creating a Profile
Switch treatment. After staging, show clear instructions for manual activation
in AAPS (or the Nightscout UI).

*Acceptance:* the new profile appears in Nightscout's profile list/editor with
exactly the merged values; the previously active profile is untouched and
still active until the user manually switches.

### FR7 — Audit / history
Keep a record of past analysis runs and what was accepted vs. rejected vs.
manually adjusted, for traceability (useful both for the user's own reference
and potentially for the hospital team). Could reuse Nightscout's own profile
history (each staged profile is already a new dated document) rather than a
separate store — **decide during design**.

### FR8 — Safety guardrails
- Surface oref0-autotune's own `autosens_min`/`autosens_max` bounds so the user
  can see the algorithm's own safety rails, not just the raw numbers.
- Flag any accepted value that represents an unusually large jump from the
  current profile.
- Require an explicit confirmation step (e.g. a review summary + confirmation
  checkbox) before the FR6 write happens — no silent/implicit staging.

### FR9 — Automated nightly runs
Each night at 4:00 the analysis is triggered with a fixed amount of days, whatever gives usually the most accurate results, and stores the result in a database. The app should have a page where the latest run is shown and should have the same downstream options as FR3 has (FR4, FR5 and FR6).

*Acceptance:* The analysis appears on the nightly analysis page and gets pulled from the database with autotune results. When the analysis failed a reason for failure must be shown.

## 5. Non-functional requirements

- **NFR1 — Isolation**: separate repository-or-directory, Dockerfile, and Gitea
  CI/CD pipeline from `glucose-dashboard`, modeled on the existing pipeline
  (test/lint → build+Trivy+push → deploy) but with its own secrets.
- **NFR2 — Nightscout token scope**: the existing dashboard token is documented
  read-only by convention (see `glucose-dashboard/README.md`). This feature
  needs a **separate, write-scoped** Nightscout token (Nightscout supports
  role-based tokens — e.g. a `careportal`-level role) rather than upgrading the
  shared read-only token, so the read-only reporting dashboard's blast radius
  doesn't change.
- **NFR3 — Test safety**: automated tests must never call a real Nightscout
  instance or the real autotune microservice — mock both, following the
  existing `monkeypatch`-on-`requests` pattern from `glucose-dashboard`'s test
  suite.
- **NFR4 — CI/CD gating**: given this pipeline can end in a write to a real
  medical data store, add a `pull_request` trigger
  for the test/lint job so changes are validated *before* merge, not only
  after — the current `glucose-dashboard` pipeline only triggers on `push` to
  `main`.
- **NFR5 — Framework**: make a descision on which framework the solution can be based best. For example streamlit, shiny for python etc.

## 6. Data requirements

- `GET /api/v1/profile.json` — read the current active profile store
  (net-new capability; nothing in `nightscout_client.py` handles profiles today).
- `POST /api/v1/profile.json` — write the new named profile (net-new).
- The autotune microservice's own job API (FR2).
- Read access to entries/treatments over the analysis date range (can reuse the
  reporting feature's date-range fetch capability once built, or the existing
  rolling-window fetch for shorter tuning windows).

## 7. Testing & acceptance criteria

- Unit tests for the profile merge/validation logic (accept/reject/adjust
  producing the correct final profile document) — pure functions, no I/O,
  fully testable without any live service.
- Unit tests for the new Nightscout profile read/write client functions, using
  the existing `FakeResponse`/`monkeypatch.setattr(requests, ...)` pattern.
- Integration-style tests against the autotune microservice using a mocked
  HTTP layer (never the real LXC service in CI).
- A UAT checklist (produced during the design phase) exercising the full flow
  end-to-end against the user's real Nightscout instance and the real
  microservice: trigger → review → adjust → stage → manually verify the new
  profile appears correctly in Nightscout, with no unintended Profile Switch.

## 8. Open questions for the user's edit pass

1. ~~Which LXC (102 or 107 or new) ends up hosting the service, once FR1 is
   done?~~ **Resolved (FR1 complete, see below): LXC 102 (`autotune`).**
2. ~~Which autotune Docker image to standardize on?~~ **Resolved: the image
   already installed on LXC 102 —
   `ghcr.io/houthacker/nightscout-autotune:latest`.** See findings below.
3. Should automatic Profile Switch activation become an opt-in "phase 2" later,
   once you've built trust in the recommendations — or should manual
   activation remain permanent policy regardless of trust? Can already start automate it as i've added in the requirements
4. FR7 (audit/history) — is reusing Nightscout's own profile-document history
   sufficient, or do you want a dedicated log outside Nightscout? Create a dedicated store of analysis and include proper production grade logging for the full application so debugging later will be easier.
5. Confirm the separate write-scoped Nightscout token is something you're
   comfortable creating/managing (NFR2) — this is a manual one-time setup step
   in the Nightscout admin UI, not something the app can do for you. Yes, i can create a new token, if you indicate which rights it needs.