glucose-dashboard/docs/requirements-profile-tuning.md

Moved. Implementation now lives in the separate glucose-profile-tuning repo; the canonical, actively maintained copy of this file is at glucose-profile-tuning/docs/requirements-profile-tuning.md. This copy is kept for history and is no longer updated.

Requirements: oref0-autotune profile tuning

Status: final
Owner: willemjan
Related: requirements-reporting.md (separate feature, same Nightscout data source)

1. Purpose & background

AndroidAPS (AAPS) only gained built-in Autotune in version 3.4, and even that version tunes only a single ISF/carb-ratio value plus the basal schedule from on-device history — it doesn't use Nightscout at all, and the user is still on AAPS 3.3 regardless. Previously, the user ran a hosted tool (inspired by MarkMpn/AutotuneWeb, now archived/discontinued) that analyzed Nightscout data with oref0-autotune and proposed a new pump profile, which was then manually entered into AAPS — a process the user describes as "a lot of trial and error." Before that, the user ran the tool from MarkMpn/AutotuneWeb but that was discontinued and cost a lot of effort to make changes to the profile by manually entering the changes into AAPS.

This feature builds a guided, human-in-the-loop workflow: trigger an oref0-autotune analysis from an internally self hosted application, review the proposed profile against the current one with the underlying estimations visible, selectively accept/adjust values, and stage the result in Nightscout for manual activation.

The user has two LXC containers on Proxmox from earlier experimentation — autotune (VMID 102, 192.168.2.13) and oref0 (VMID 107, 192.168.2.14) — currently stopped, of unknown/half-remembered working state. The working container can be used for the profile calculations, but if another solutions surpasses in terms of integratability a new solutions can be set up.

2. Scope

In scope:

• Booting and inspecting the two existing LXCs to determine (or establish) a working oref0-autotune setup.
• A small HTTP microservice on that LXC that runs oref0-autotune against the user's Nightscout data on demand. Either one of the two existing LXCs or a new one.
• A separate app/service (own deployment, own CI/CD pipeline) that triggers the analysis, displays old-vs-new profile comparisons with estimation detail, lets the user selectively adjust proposed values, and stages the result in Nightscout. The user can then choose if the profile is automatically activated in AndroidAPS (creating a "Profile Switch" treatment)
• This app could be based on the glucose-dashboard streamlit app, but streamlit is not a fixed requirement, another solution can be used. For example, shiny for python.
• A nightly scheduled autotune run of which the results are visible in the morning. The profile of each run should be stored in a database. The user can read those back from the database in the application and can send it to nightscout/androidAPS, but the nightly scheduled runs should not send profiles automatically to nightscout and/or AAPS, this should always be a user action.

Out of scope (this iteration):

• Any change to AAPS settings or on-device configuration.

3. Architecture decisions (made during spec review)

1. Execution: an always-on HTTP microservice on whichever LXC has (or gets) a working oref0-autotune setup, rather than SSH-triggering a one-off Docker run or bundling autotune into the dashboard's own container. The dashboard calls it over HTTP and polls for job completion.
2. Profile push safety: stage-only. The service writes a new named profile into the Nightscout profile store and does create a "Profile Switch" treatment when the user selected that option — final activation stays a manual step in AAPS when the option is not selected. This matters because AndroidAPS applies profiles via Profile Switch treatments (gated behind an explicit "accept profile switch from Nightscout" AAPS setting), not by simply reading profile.json.
3. Deployment: a separate app/service from glucose-dashboard, with its own repo-or-directory, Dockerfile, and Gitea CI/CD pipeline — isolating write-capable, dosing-adjacent code from the read-only dashboard.

4. Functional requirements

FR1 — Infra investigation (first task, needs explicit go-ahead)

Boot LXC 102 (autotune) and LXC 107 (oref0) and inspect what's installed (Docker containers, cron jobs, scripts, config, any README/notes left behind) to determine which one — if either — has a working oref0-autotune setup. Document findings (what's there, what's missing, what needs rebuilding).

Acceptance: a clear decision on which LXC hosts the microservice, and what (if anything) needs to be installed/fixed there before FR2 can start.

Status: done (2026-07-26). Both LXCs were booted and inspected, then stopped again (back to their original state — no state was left running).

• LXC 102 (autotune, 192.168.2.13, 1 core / 1GB RAM / 16GB disk) — Docker installed, with the image ghcr.io/houthacker/nightscout-autotune:latest (Alpine + Node 24, ~279MB) already pulled and previously run successfully (9 months ago) against the real Nightscout instance (https://zb-ns.duckdns.org). It bundles the full oref0 npm toolkit (oref0-autotune, oref0-upload-profile, convert-ns-profile, mm-format-ns-profile, and more) under /usr/local/bin, not just the autotune algorithm itself. Its documented interface is a single container run with env vars: NS_HOST, AUTOTUNE_DAYS, UAM_AS_BASAL (mandatory), plus optional NS_TOKEN/NS_API_SECRET (as sha1 hashes), NS_PROFILE, AUTOSENS_MIN/MAX, MIN_5MIN_CARBIMPACT, INSULIN_TYPE, OPENAPS_WORKDIR, HTML_EXPORT. A prior run's output is still on disk at /root/autotune-output/autotune/: per-day ns-entries/ns-treatments fetched from Nightscout, the oref0-format profile.json (input) and newprofile.YYYY-MM-DD.json (recommendation), and a human-readable autotune_recommendations.log (a Pump | Autotune | Days Missing table per parameter/basal-slot — ideal source for FR4's estimation detail). Critically, the image also has oref0-upload-profile (profile.json NSURL api-secret [--preview] [--switch]) already installed — this is the piece that converts an oref0-format profile back into Nightscout's profile-store format and optionally issues the Profile Switch treatment, i.e. it already implements what FR6 needs. --preview (dry run, no upload) is also useful for FR8's confirmation step.
• LXC 107 (oref0, 192.168.2.14, same specs) — no Docker; a native npm/openaps install of the raw oref0 toolkit. Bash history shows a messier path to a working run (locale errors needing locale-gen, several retries with different flag spellings) before landing on a working oref0-autotune --dir=~/myopenaps --ns-host=... --start-date=... invocation and a manual oref0-upload-profile ... --switch=true call. It reaches the same destination but through an unreproducible, hand-patched native environment rather than a pinned, documented container image.

Decision: LXC 102 is the base. It already has a working, versioned, well-documented container with both the analysis direction (Nightscout → oref0 profile → recommendation) and the upload direction (oref0 recommendation → Nightscout profile store, with optional Profile Switch) solved, via tools already installed in the same image. No new autotune image needs to be built or evaluated — the open question in §8 about choosing between p5nbTgip0r/autotune-docker / houthacker42/nightscout-autotune / a from-scratch oref0 build is moot: houthacker/nightscout-autotune is what's already there, already proven against this Nightscout instance, and turns out to be the most complete of the three. LXC 107 will be left stopped/decommissioned — its only asset (a successful manual run) is already superseded by 102's cleaner path. FR2's microservice will be built as a thin wrapper around the CLI tools already baked into the nightscout-autotune image (see design-profile-tuning.md), rather than reimplementing Nightscout-profile ⇄ oref0-profile conversion from scratch — directly resolving the "had to manually parse parts of the profile" pain point named in FR2.

FR2 — Autotune HTTP microservice

A small HTTP API (exact framework TBD during design — e.g. Flask/FastAPI wrapping a Docker-based oref0-autotune image) exposing at minimum:

• POST /analyze — body: Nightscout URL/token, date range, (optionally) the starting profile to tune from. Returns a job ID.
• GET /analyze/{job_id} — status (pending/running/done/failed) and, once done, the recommended profile JSON plus the estimation/recommendation detail (per-hour deviations, the autotune log).
• In previous tries, the user had to manually parse parts of the profile as obtained from Nightscout to have it accpeted by the autotune service. This should be thoroughly investigated and subsequently automated

Acceptance: a manual curl against the running service, pointed at the user's real Nightscout instance, returns a plausible recommended profile within a reasonable time (minutes, not seconds — oref0-autotune is not instant).

FR3 — Trigger analysis from the dashboard

A new app where the user picks a date range and selects a current profile to tune from (fetched live from Nightscout), starts the analysis, and sees progress while the microservice job runs (via polling). Nightscout connection should be made using a token with a proper role.

Acceptance: starting an analysis with bad/unreachable Nightscout credentials fails with a clear error rather than hanging silently.

FR4 — Old-vs-new profile comparison view

Side-by-side display of the current and proposed profile: hourly basal rates, ISF schedule, carb ratio schedule, DIA — plus the underlying estimation detail (per-hour deviations, confidence notes) surfaced from the autotune log, so the user can judge why a change is being proposed, not just see the new number.

Acceptance: every changed value is visually distinguishable from unchanged values; the estimation detail for a given time slot is viewable alongside that slot's proposed change.

FR5 — Selective adjustment

Per time-slot controls (accept/reject each proposed basal/ISF/CR change individually) plus editable numeric fields so the user can fine-tune any accepted value before staging — matching the user's explicit request to "select parts of the proposed profile and still adjust the values a bit." This should look like a table with the columns 'time-slot' (per hour), 'current profile', 'suggestion', a input to select which column to select (current vs suggestion) and on the last column the final selected value, but as input numeric fields which can be changed by 0.01 steps.

Acceptance: rejecting a slot keeps the current value unchanged in the staged profile; editing an accepted value's number is reflected in what actually gets sent to Nightscout.

FR6 — Stage to Nightscout

Write the resulting merged profile (accepted + manually adjusted values) as a new named profile in the Nightscout profile store (e.g. "Autotune YYYY-MM-DD" or "Autotune YYYY-MM-DD ID") via a new POST to /api/v1/profile, with optionally creating a Profile Switch treatment. After staging, show clear instructions for manual activation in AAPS (or the Nightscout UI).

Acceptance: the new profile appears in Nightscout's profile list/editor with exactly the merged values; the previously active profile is untouched and still active until the user manually switches.

FR7 — Audit / history

Keep a record of past analysis runs and what was accepted vs. rejected vs. manually adjusted, for traceability (useful both for the user's own reference and potentially for the hospital team). Could reuse Nightscout's own profile history (each staged profile is already a new dated document) rather than a separate store — decide during design.

FR8 — Safety guardrails

• Surface oref0-autotune's own autosens_min/autosens_max bounds so the user can see the algorithm's own safety rails, not just the raw numbers.
• Flag any accepted value that represents an unusually large jump from the current profile.
• Require an explicit confirmation step (e.g. a review summary + confirmation checkbox) before the FR6 write happens — no silent/implicit staging.

FR9 — Automated nightly runs

Each night at 4:00 the analysis is triggered with a fixed amount of days, whatever gives usually the most accurate results, and stores the result in a database. The app should have a page where the latest run is shown and should have the same downstream options as FR3 has (FR4, FR5 and FR6).

Acceptance: The analysis appears on the nightly analysis page and gets pulled from the database with autotune results. When the analysis failed a reason for failure must be shown.

5. Non-functional requirements

• NFR1 — Isolation: separate repository-or-directory, Dockerfile, and Gitea CI/CD pipeline from glucose-dashboard, modeled on the existing pipeline (test/lint → build+Trivy+push → deploy) but with its own secrets.
• NFR2 — Nightscout token scope: the existing dashboard token is documented read-only by convention (see glucose-dashboard/README.md). This feature needs a separate, write-scoped Nightscout token (Nightscout supports role-based tokens — e.g. a careportal-level role) rather than upgrading the shared read-only token, so the read-only reporting dashboard's blast radius doesn't change.
• NFR3 — Test safety: automated tests must never call a real Nightscout instance or the real autotune microservice — mock both, following the existing monkeypatch-on-requests pattern from glucose-dashboard's test suite.
• NFR4 — CI/CD gating: given this pipeline can end in a write to a real medical data store, add a pull_request trigger for the test/lint job so changes are validated before merge, not only after — the current glucose-dashboard pipeline only triggers on push to main.
• NFR5 — Framework: make a descision on which framework the solution can be based best. For example streamlit, shiny for python etc.

6. Data requirements

• GET /api/v1/profile.json — read the current active profile store (net-new capability; nothing in nightscout_client.py handles profiles today).
• POST /api/v1/profile.json — write the new named profile (net-new).
• The autotune microservice's own job API (FR2).
• Read access to entries/treatments over the analysis date range (can reuse the reporting feature's date-range fetch capability once built, or the existing rolling-window fetch for shorter tuning windows).

7. Testing & acceptance criteria

• Unit tests for the profile merge/validation logic (accept/reject/adjust producing the correct final profile document) — pure functions, no I/O, fully testable without any live service.
• Unit tests for the new Nightscout profile read/write client functions, using the existing FakeResponse/monkeypatch.setattr(requests, ...) pattern.
• Integration-style tests against the autotune microservice using a mocked HTTP layer (never the real LXC service in CI).
• A UAT checklist (produced during the design phase) exercising the full flow end-to-end against the user's real Nightscout instance and the real microservice: trigger → review → adjust → stage → manually verify the new profile appears correctly in Nightscout, with no unintended Profile Switch.

8. Open questions for the user's edit pass

1. Which LXC (102 or 107 or new) ends up hosting the service, once FR1 is done? Resolved (FR1 complete, see below): LXC 102 (autotune).
2. Which autotune Docker image to standardize on? Resolved: the image already installed on LXC 102 — ghcr.io/houthacker/nightscout-autotune:latest. See findings below.
3. Should automatic Profile Switch activation become an opt-in "phase 2" later, once you've built trust in the recommendations — or should manual activation remain permanent policy regardless of trust? Can already start automate it as i've added in the requirements
4. FR7 (audit/history) — is reusing Nightscout's own profile-document history sufficient, or do you want a dedicated log outside Nightscout? Create a dedicated store of analysis and include proper production grade logging for the full application so debugging later will be easier.
5. Confirm the separate write-scoped Nightscout token is something you're comfortable creating/managing (NFR2) — this is a manual one-time setup step in the Nightscout admin UI, not something the app can do for you. Yes, i can create a new token, if you indicate which rights it needs.