methodology

Circadia Methodology.

How Circadia captures and analyzes free-running sleep data for researchers evaluating the app's outputs.

Circadia ->

A description of how Circadia captures and analyzes free-running sleep data, written for circadian-rhythm researchers evaluating whether the app's outputs can be cited or used in secondary analysis.

For the patient-clinician handoff, see the Doctor's Report PDF exported from the app. Code paths below (src/lib/…) refer to the Circadia app repository unless they say otherwise.

Last revised: 2026-10-27. This revision re-checked every claim against the shipped code and corrected what had gone stale since 2026-06-17: one gate multiplier (1.3) for every τ; the 24.9 h prior; how naps, sleepless answers and long cycles enter the estimator; what the displayed τ actually is; the Adaptive model now served (v8, with per-person version pins); the report card; the Smart forecast's per-person layers; the current covariate list; the admin endpoints; and the cohort state (§12, as of 2026-10-27). Prior revisions: 2026-06-17 (per-row drift label unified on the gap-space series) and 2026-06-09 (gap-space Smart estimator with robust Huber location; timezone-correct epoch parsing).


1. What Circadia measures

Circadia is a self-tracking app for sighted Non-24-Hour Sleep-Wake Disorder (N24SWD) and adjacent free-running phenotypes. Each user logs sleep onsets and wakes (and optionally sleepless gaps), by hand or by import from a wearable or another sleep app. The app derives:

  • Daily drift — the shift in sleep onset from one cycle to the next, in hours.
  • τ (tau) — the estimated period of the user's observed sleep-wake rhythm (a behavioral period), in hours: τ = 24 × cycleDays + drift, where drift is a robust location (weighted Huber M-estimate, §3 step 5) of the per-cycle drifts and cycleDays is 1 for everyone whose median cycle is under about 39.6 h (§3 step 3). Under the limitations described in §13 (no DLMO, self-report, behaviorally derived), this approximates — but is not a direct measurement of — the intrinsic circadian period reported in lab forced-desynchrony protocols (Czeisler 1999; Duffy 2011).
  • σ_obs — observed variability of the daily drift.
  • σ_τ — standard error on τ.
  • Sleep per night — for a night logged in pieces, the sum of the pieces' own measured lengths (one value per night), not the length of each piece (live since 2026-10-19).
  • |drift| and |drift all| — unsigned drift magnitudes (cohort-side diagnostic, see §10).
  • Cumulative debt and Process S — homeostatic measures, included for completeness; not the primary contribution.

The primary research contribution is this behavioral τ estimate and its uncertainty, derived from continuous at-home logging over multi-week windows. It is intended as an ambulatory approximation of the intrinsic period measured under controlled lab conditions (Czeisler 1999; Duffy 2011), with the limitations of self-reported onset timing that that implies. See §13 for the full caveat list.


2. Drift between two sessions

For two consecutive sleep onsets onset_prev and onset_curr, the gap-space drift used by the Smart estimator is:

gap_h   = (onset_curr − onset_prev) in hours
drift_h = gap_h − 24 × cycleDays          (cycleDays = 1 for almost everyone)

drift_h is the per-cycle shift in onset relative to a 24-hour day. A positive drift indicates phase delay; negative indicates phase advance.

Gap-space drift is unwrapped: it retains the integer-day content of the gap instead of reducing the shift to a clock-face difference. This removes the +14 h-vs-−10 h aliasing that a modular representation suffers (a real +15 h forward shift is +15 h in gap-space, not −9 h), at the cost of requiring explicit gating of gaps that are not single body-cycles (§3, step 3).

The older wrapped-clock drift — d_clock = hour_of_day(onset_curr) − hour_of_day(onset_prev) wrapped to (−12, +12] (calcDrift(), now in src/lib/model-primitives.js and re-exported from src/lib/utils.js) — is not the basis of the headline τ. It survives as the basis of the legacy Clean and Raw comparison estimators (§4). Since 2026-09-08 it no longer wraps a clean gap of 36–48 h: there the clock-face difference picks the wrong branch (a 38 h day would read −10 h), so calcDrift returns gap_h − 24 instead. The two quantities are congruent mod 24 and identical for any gap in (12 h, 36 h].

Per-row labels. Since 2026-09-14 every per-row drift a user reads (Log, Calendar, Chart, day and entry sheets, and the analysis cards that cite one night) comes from one rule, nightDriftReading() in src/lib/nightDrift.js, presented by entryDriftChip() (src/lib/driftLabel.js):

  • gap under 14 h → a second sleep inside one day: the row shows the gap length, not a drift;
  • |gap-space drift| over 18 h → more than one body-day: the row shows the gap, unless the τ estimator counted that cycle (a manual include or a confirmed sleepless stretch), in which case it shows the drift;
  • otherwise → the gap-space drift.

Some secondary displays still rebuild drift with their own clock-face wrap; see §13 item 9.

Timezone handling (2026-06-09)

Onsets and wakes are stored as timezone-naive local strings plus a separate per-entry IANA tz column. At the data boundary, each entry is stamped with absolute epochs onsetMs / wakeMs via localToMs(naive_local, tz) (src/lib/model-primitives.js), which resolves the named zone with DST awareness. All gap and drift math reads these absolute epochs. Entries without a stored tz (legacy rows) fall back to parsing in the viewer's browser timezone. A wall-clock time typed inside a repeated (fall-back) hour is stored with an explicit UTC offset (since 2026-10-21), which the parser reads as-is.

Before this change, gap math implicitly parsed naive local strings in the viewer's timezone, so the same dataset produced different drift numbers depending on who was looking (an admin in one timezone saw different values than the user in another). The fix is a no-op for a user viewing their own data in their home timezone (verified across the cohort at the time: 0 of 44 users with entries moved) and corrects cross-timezone, admin, and cohort views.

Check-in records (e.g. 1–5 alertness ratings, table circadia_checkins) are stored differently: as absolute timestamptz. Analyses that place them on a wall clock must convert them, and must not convert twice.


3. τ estimator (Smart) — gap-space, robust

The production τ/drift estimator is a gap-space estimator with τ-relative cycle gating, adjacent anchoring, an EWMA-weighted fading prior, and a robust (Huber M-estimator) location. It is implemented as estimatePeriodGapSpace() in src/lib/gapSpaceTau.js and is called in production with adjacent anchoring ON and soft phase-cleanliness weighting OFF, through runGapSpaceEstimate() in src/App.jsx (mirrored for the report card by gapSpaceDriftEstimate() in the same module).

Dated transitions that matter for longitudinal comparisons of app outputs:

  • 2026-05-29 — gap-space estimator promoted to the headline, replacing the legacy wrapped-clock EWMA regression (estimatePeriod(); now a comparison surface, §4).
  • 2026-06-09 — reported location changed from the weighted mean to a weighted Huber M-estimate, and all gap math moved to timezone-correct absolute epochs (§2).
  • 2026-07-10 — population prior recentred from 24.7 h to 24.9 h (24.0 h for users on an entrained lens).
  • 2026-07-25 — users' own answers about long gaps (sleepless / "didn't log") enter the estimate; nap bridging becomes conditional (step 1); cycle-days and the widened long-cycle cutoff added (step 3).
  • 2026-09-14 — one upper-gate multiplier, 1.3, for every τ (it had dropped to 1.2 above τ 28 h).
  • 2026-09-21 — a nap can no longer be counted as a day's only sleep when that would create a day shorter than the estimator's own lower gate (step 1).

Inputs and exclusion flags

The estimator receives the full entry list with flags and handles exclusion internally:

  • xDrift — excluded from drift (nap auto-flag or user exclusion).
  • fragmented — auto-flagged continuation of a prior episode (markFragmented(), §7).
  • tooShort — sleep span below the user's nap threshold.
  • fragManual — user explicitly marked this as a fragment of the previous sleep; never a phase marker of its own.
  • xDriftManual — the user explicitly toggled this entry. Explicit intent wins: the include/exclude decision is then governed purely by xDrift, overriding the auto-flags, and a manually included entry also bypasses both cycle gates (step 3) in either direction.
  • Sleepless marks (circadia_wake_periods: a date plus the hours the person reports being awake) and "I slept but didn't log it" answers on specific gaps (step 3).
  • priorTau — 24.9 h, or 24.0 h when the user has chosen an entrained lens (step 4).

Step 1 — adjacent anchor timeline

Gaps are measured against the immediately preceding entry in time (adjacent anchoring). Excluded entries still anchor the timeline — they contribute no drift of their own, but the next entry measures its gap to them rather than bridging across them. Bridging was the source of a cascade bug in which one excluded sleep manufactured a phantom multi-day "cycle" and nulled the drift of an otherwise-clean neighbor.

Three kinds of entries are dropped from the anchor timeline:

  • user-marked fragments (fragManual);
  • leading false-starts — auto-flagged fragments whose episode's main sleep comes after them (e.g. a 45-minute doze right before the real sleep), which would otherwise make the real sleep measure a tiny gap to its own false-start;
  • bridged naps (classifyNaps()). A nap-length exclusion (shorter than the nap threshold) that the user marked always bridges. One the app marked bridges only if the merged gap it leaves (previous non-nap onset to next non-nap onset) is a plausible single cycle for this person: 0.9–1.1 × their median non-nap onset-to-onset gap (the window widens by a factor 1/0.85 on the short side when a neighboring row ended in a forced wake, and on the long side when one carries an illness, medication or stress flag). Otherwise the nap keeps anchoring and contributes no drift. The reference cycle is computed from the data before any bridging, not from τ, to avoid a feedback loop. A nap is never treated as the only sleep of its cycle if either day that would create falls under the lower gate (since 2026-09-21); counting such naps as phase markers ("promotion") is implemented but switched off.

An optional per-user setting ("manually excluded sleeps → measure from your last counted sleep", bridgeManualExclusions, default off) bridges every hand-excluded sleep regardless of length.

Step 2 — candidate transitions

Each surviving adjacent pair contributes a gap-space drift drift_i = gap_h − 24 (§2) with an EWMA recency weight

w_i = exp(−ln(2) × (t_now − t_onset_i) / halfLife),  halfLife = 28 days

Recent data is weighted more heavily than old data. The 28-day half-life is a design choice, not a fitted constant. For windowed views of the past (e.g. a four-week window a year ago), t_now is the window's own end, so old windows are not swamped by the prior.

Step 3 — two-pass τ-relative cycle gating

A gap-space drift is only meaningful if the gap is approximately one body-cycle. Two gates decide that, both scaled to the user's own τ:

lower_gate = max(18 h,  0.7 × τ_est)
upper_gate = max(34 h,  1.3 × τ_est)
  • gap < lower_gate → "snapback" — too short to be a real single cycle (a nap, fragment, or short recovery sleep that survived upstream filtering). Excluded from the point estimate.
  • gap > upper_gate → "crash" — a sleep-debt collapse or a logging hole, not a valid phase marker. Excluded from the point estimate entirely; it contributes zero rather than dragging τ.

The multiplier is 1.3 for everyone since 2026-09-14. Until then it dropped to 1.2 when τ_est exceeded 28 h, so a longer body-day bought a tighter ceiling. Measured across all 146 live logs with each participant's own settings and answers before the change, 3 logs moved (+0.56, +0.37, +0.15 h) and a walk-forward test moved their median next-onset error by 3, 11 and 0 minutes; the change removes a monotonicity violation rather than claiming an accuracy gain.

The estimator runs two passes: pass 1 gates with the prior τ (24.9 h free-running; 24.0 h under an entrained lens) and takes an EWMA-weighted mean of the surviving drifts as τ_est; pass 2 re-gates with that estimate. The τ-relative cap matters for fast drifters: a flat 30 h gate (the legacy behavior) systematically truncated their genuine long cycles and biased τ downward.

Four refinements sit on top of the gates:

  • The person's own answers win over the gate. If the hours a user marked sleepless inside a gap, plus the logged sleep that opened it, account for at least 85% of the gap, the gap is one cycle however long (it overrides the upper gate). A gap the user marked "I slept but didn't log it" is never counted, however ordinary it looks. A long gap is never subdivided into several shorter "cycles".
  • Widened cutoff when the gate is eating the rhythm. If pass 1 discards at least 25% of a person's transitions as too long, and at least 20 cycles survive, the upper gate's 34 h floor is replaced by 1.25 × the 90th percentile of their surviving gaps (when larger). At introduction this changed one log in the cohort.
  • Multi-day cycles. cycleDays = max(1, floor(median_gap/24 + 0.35)) from the pass-1 surviving gaps, so a person whose median cycle exceeds about 39.6 h is measured against the number of clock days a cycle actually spans (drift = gap − 24 × cycleDays). Everyone below that median has cycleDays = 1 and is unaffected.
  • Wake axis. The same included cycles are also measured wake-to-wake (wakeDrift); a user can choose that as their drift basis.

Excluded transitions are individually labeled (snapback / crash) and surfaced in the user's log as "not counted toward drift," with the manual-include override (xDriftManual, above) available per entry.

Step 4 — fading prior and weighted mean (seed)

A weak prior centers cold-start estimates on the published sighted-N24 population mean: priorPseudoCount = 3 pseudo-observations at drift = priorTau − 24 = 0.9 h (free-running default; entrained lenses use 24.0 h → 0.0 h), with prior SD 1.0 h. The prior mean is Hayakawa 2005 (n = 57 sighted N24: 24.9 ± 0.4 h). The SD is deliberately wider than Hayakawa's: our own cohort's between-person SD was 0.98 h at the 2026-07-10 recentring (51 users with at least 30 real sleeps; mean 25.05 h, median 25.00 h). The prior fades linearly —

effective_prior = max(0, priorPseudoCount − N_included / 3)

— so it is gone entirely by ~9 included cycles. The prior-anchored weighted mean of included drifts is still computed, but since 2026-06-09 it is the seed and fallback for the robust location below, not the reported number. It is returned as meanDrift for diagnostics.

Step 5 — robust location (weighted Huber M-estimator)

The reported drift is a weighted Huber M-estimate of location over the included cycles, using the same per-cycle EWMA weights:

m0        = weighted median of drift_i
scale     s = 1.4826 × median(|drift_i − m0|)     (classic, unweighted MAD)
location  μ = IRLS fixed point of Σ w_i ψ_c((drift_i − μ)/s),  c = 2.0
τ           = 24 × cycleDays + μ

The prior is not injected into the Huber location; it enters only the seed, the variance and the effective N. Residuals within c·s of the location get full weight; beyond that, influence grows only sub-linearly instead of one-for-one. The tuning constant c = 2.0 is deliberately larger than the classic 1.345: a cycle must sit more than two robust standard deviations out before it is down-weighted at all, which keeps the estimator a practical no-op for clean directional drifters while reining in the heavy two-sided tails that fragmented, bidirectional, or relative-coordination sleepers produce.

No-op guarantees: with fewer than 2 included cycles, or a degenerate (≈0) MAD, the estimator returns the plain weighted mean. With no included cycles at all it returns the prior-anchored mean. At deployment (2026-06-09) the change was validated against the live cohort: near no-op for clean drifters (one fast directional drifter moved +3.97 → +3.85 h/day), with material correction for heavy-tailed loggers (one bidirectional logger moved from a washed-out −0.33 mean to +0.51, matching her median cycle).

Step 6 — uncertainty

σ_obs²  = weighted variance of included drifts, centered on μ (the
          robust location), prior included
σ_floor = max(1.0, 0.5 × √(max(μ, 0.5)))
σ_obs   = max(σ_obs, σ_floor)
N_eff   = (Σ w_i)² / Σ w_i²            # Kish, with the prior's 3 pseudo-counts
σ_τ     = σ_obs / √max(N_eff, 1)

Explained cycles do not widen the band. When a user has marked some included cycles sleepless and at least 3 included cycles remain unexplained, σ_τ is computed on the unexplained cycles only; the explained ones still count toward the drift. A cycle with a stated cause is not evidence that the rhythm is unreadable.

The estimator also returns the median of included-cycle drifts (driftMedian, tauMedian) as a second robust summary. It is a diagnostic: no headline surface displays it (step 8).

Step 7 — structural signals and collapses

The estimate carries explicit flags:

  • insufficientData — fewer than 3 included cycles.
  • implausibleTau — τ < 21 h, a non-physiological period regardless of sleep structure. There is deliberately no symmetric upper bound: genuine N24 free-runners legitimately reach τ ≈ 27–29 h+, so a normal-population ceiling would wrongly gate exactly the users the app exists for.
  • polyphasic — a structural multi-sleep-per-cycle signal, independent of the gates. It trips on either (a) a manual-include tell — ≥ 4 included cycles of which > 40% are sub-18 h gaps the user forced in, asserting multiple sleeps per cycle — or (b) a density tell — ≥ 6 adjacent gaps with > 1.5 logged sleeps/day or a median adjacent onset-gap < 18 h. Both are vetoed by a consolidated-main-sleep guard: if the median non-fragment sleep is ≥ 4 h, the user has one main sleep per cycle (an occasional second-sleep logger, not a polyphasic schedule) and is not flagged.

The estimator also returns a combined trustworthy flag, but no display surface reads it. What a user is shown is decided by the tier logic in step 8, which uses the cycle count and the implausibility floor directly.

Two collapses run before the headline estimate, in this order:

  1. Multi-sleep logs (detectMultiSleep()): if most cycles (≥ 40%) contain a second sleep, the pattern holds across the log (required for logs over 180 days or under 28 days), the typical sleep is under 5 h, and one bout per cycle is clearly dominant, each cycle is represented by its first sleep and the estimate is re-run on those markers (at least 20 needed). Narrow by design: at introduction, 1 log in the live cohort qualified.
  2. User-merged split nights: if any row is fragManual, the estimate is also run with each merged night collapsed to one sleep at its first onset, and the collapsed estimate is adopted only if it does not raise σ_obs by more than 0.4 h.

Step 8 — what the user actually sees

Several surfaces consume the estimate, and they are not the same number:

  • The drift number (Predict's drift tile, "Smart" basis) is the robust location (step 5) over the user's selected window (7d/14d/30d/90d/all/custom; the default is all-time and the choice is not persisted between sessions). The windowed run uses identical options to the full-history run. The user may switch the basis to Clean, Raw (§4) or Wake (step 3); a typed manual drift overrides all of them.
  • The tier comes from classifyTau() (same module, run with the same σ-gated merged-night collapse). It assigns one of four display tiers: tier 4 onboarding (0–1 included cycles; no number), tier 3 varies (no number: τ below 21 h, or at least 60 cycles with σ_τ still above 4 h), tier 2 preliminary (2 cycles, or σ_τ above the established bar), tier 1 established. The bars scale with drift, because a fast drifter has more absolute spread for the same regularity: established if σ_τ ≤ 1.0 + 0.25 × |drift|, preliminary up to 1.3 + 0.35 × |drift|, and still preliminary above that unless the 60-cycle / 4 h condition holds. Logs flagged polyphasic are first collapsed to one main sleep per cycle (onsets gap-clustered at 14 h breaks; the longest sleep in each cluster becomes that cycle's marker), tiered on the collapsed series, and their band gets a 0.5 h floor added in quadrature.
  • The displayed τ (resolveDisplayTau() in src/lib/derived.js). Two candidate day lengths exist: the tier's τ (classifyTau, the Huber estimate or the polyphasic collapse) and the full-history estimate's τ (24 × cycleDays + drift). headlineDriftPlan() replays both over the person's last 60 onset-to-onset steps and shows the estimate's τ only if it lands at least 5 minutes closer per night (median) over at least 8 nights; otherwise the tier's τ. A collapsed polyphasic τ is never replaced, a typed manual drift always wins, and tiers 3–4 show no number. For clean single-sleepers the two candidates are the same number.

The Adaptive model (§5) has its own τ readout, described there.


4. Alternative drift estimators (for comparison)

Two simpler estimators are exposed in the UI as user-selectable drift bases. Both operate on the legacy wrapped-clock drift (§2), anchored to the most recent real sleep, over non-excluded rows:

  • Clean — unweighted arithmetic mean of drift over transitions that are not post-sleepless, i.e. gap_h ≤ postSleeplessThresholdH (default 30 h, per-user adjustable, 18–72 h). No recency weighting, no prior, no τ-relative gates, no robustness.
  • Raw — unweighted mean over ALL transitions, including post-sleepless ones. For post-sleepless transitions Raw reads driftRawLegacy, the onset-hour difference forced into [0, 24 h) (congruent to gap_h mod 24). Included for transparency; biases τ upward when users have many skipped sleeps.

Because both read the clock-face difference, both alias true per-cycle shifts beyond 12 h on gaps outside the range handled in §2 (Clean unwraps a clean 36–48 h gap since 2026-09-08; anything longer is post-sleepless on the default dial) and suffer sign cancellation in summary statistics. They are retained for transparency and historical comparability, not inference.

A third comparator exists in source: the legacy Smart estimator (estimatePeriod(), src/lib/model-primitives.js) — an EWMA-weighted wrapped-clock regression with the same fading prior and σ floor, plus a wraparound-unfolding pass and a bidirectional-drift median fallback. It was the headline estimator until 2026-05-29. It still supplies the cohort table's σ_obs column and unwrap diagnostics (§10), the Actogram's visible-window estimate, and the Model Lab. Its full description is preserved in this file's git history (revisions before 2026-06-09).

A fourth, the fold periodogram (src/lib/foldTau.js), is not a drift basis users choose: it sets the forecast rate for multi-sleep sleepers (§8).

When evaluating cohort data, the gap-space Smart estimator (§3) is the recommended reference. Raw remains useful as a divergence diagnostic — when Raw and Smart diverge by > 3 h on the same user, most of that user's transitions are being filtered (low cycle rate) and any single summary number undersells their real shift magnitude. See §10 for the cohort-side diagnostic.


5. Adaptive forecast (Kalman IMM)

The Smart estimator above returns a single best-fit τ and its uncertainty. The Adaptive prediction engine is a separate sequential state-space model that maintains a full phase/drift distribution for each user and projects it forward. It is an Interacting Multiple Model (IMM) Kalman filter, implemented as kalmanIMMPredict() in src/lib/kalman-dlm.js. It superseded the May-2026 particle filter (particleFilterPredict() in src/lib/model-lab.js, retained as a comparison surface in the Model Lab) when the IMM class was first promoted on 2026-06-03.

Its shared parameters are trained on opted-in, anonymized sleep histories (§6, §11): Adaptive is a machine-learned model in that sense. The default served configuration is version 8, promoted 2026-07-04 and still the default as of 2026-10-27. It is resolved at runtime from Circadia's own backend (/api/model-config), so a promotion is a reversible database write the app picks up on next load — no redeploy. On top of the default, each person can be pinned to a different catalog version chosen by an audit of their own held-out nights (/model-pin; §6). On 2026-10-21, 83 of the 93 logs graded for a report-card analysis carried a pin: 52 on v8, the rest on versions 1, 2, 3, 4, 6, 9, 10 or 19.

Architecture

The state is a local linear-trend Kalman filter: a level (the user's current onset phase) and a slope (their current per-cycle drift), with the slope free to evolve so the tracker follows acceleration as well as steady drift. Each historical entry is folded in by the standard Kalman predict → update step (levelNoiseH, slopeNoiseH, and observation noise obsSigmaH govern how fast level and slope are allowed to move and how much each logged onset is trusted).

The "IMM" part runs this filter as a bank of modes that share everything except the slope (drift) process noise:

ModeSlope process noise (v8)Behavior
CalmslopeNoiseH = 0.005tight drift center — a stable free-run
VolatileslopeNoiseH × volatileScale (≈ 35.7×)wide band — a user going "off the rails"

Each step the modes are Markov-mixed: a transition matrix (stickCalm = P(stay calm), stickVol = P(stay volatile), initModeProbVol = initial mass on the volatile mode) blends their estimates by how well each has been predicting.

Served v8 also switches on several deterministic and structural terms that earlier versions of this document described as off:

  • Process S (homeostatic pressure) — a battery that charges over time awake and discharges over sleep; its deviation from baseline shifts the predicted onset earlier when over-tired and lengthens the predicted sleep (pressureOnsetFactor ≈ 3.2, pressureDurationFactor ≈ 4.1).
  • Clock-time terms — a periodic wave with a fitted period of 2.32 days (triangle, square and saw components) and a weekly calendar term (cosine amplitude ≈ 0.63 h), both global, applied to every user.
  • Phase-response terms — light and medication phase-response curves gated to nights the tag is logged, a relative-coordination capture term (phase-dependent drift, amplitude 0.4 h), and a scallop-reset nudge when a strong positive drift suddenly reverses.
  • One shared harmonic with process noise (curvature) and a harmonic-energy band inflation.
  • Secondary-sleep streams (streamDecomposition = 1): nearby episodes are grouped into body-day cycles, the clock is fitted to each cycle's anchor episode, and repeatable secondary sleeps are forecast as additional streams instead of being deleted.
  • Per-user observation scale (obsAdaptRate ≈ 0.26): each user's observation noise adapts to their own residuals, within 0.36–5.65× the global value.

The exact served values are public at /api/model-config and inspectable in the Model Lab (#model-lab).

Session breaks and the τ readout (October 2026)

  • Session break (live 2026-10-17). An anchor gap longer than sessionBreakGapH = 336 h (two weeks unlogged) is not drift evidence: past about 14 unlogged cycles a free-runner's cycle count is not identifiable. On a break the level re-anchors one cycle before the returning sleep; the drift and its variance are kept; phase-locked harmonics, the transient and the scallop velocity are cleared. Before this, one consented log with a 3.45-year gap read an Adaptive τ of 15.07 h for about 20 sleeps. Logs without such a gap are unchanged.
  • Lived-rhythm τ readout (live 2026-10-18, display only). Adaptive's drift is deliberately stiff, so 24 + drift read hours below the rhythm fast drifters actually live. Each forecast row now carries tauReadout: the recency-weighted median (half-life 24 cycles) of uncontradicted anchor gaps, capped at 1.6 × the filter's period so unlogged sleeps do not read as long cycles. It is shown (source: "lived-rhythm") only when it differs from 24 + drift by more than 1 h. Forecasts are unchanged.

The band

The forecast band is read from the model's own predictive distribution (the mixture of the modes' Gaussians, plus the stream and harmonic inflation terms above), not from a fixed multiplier. It is wide (and can be multimodal) mid-transition when the modes disagree, tight when they agree, and it widens as process noise accumulates across long gaps. Its calibration is measured, not assumed: on the consented cohort on 2026-10-18, Adaptive's nominal 80% band covered 81.1% of next onsets (Smart: 76.8%).

Robustness and adaptation

  • Outlier gate (outlierGate) — an observation that lands too many σ from the prediction is downweighted instead of being allowed to yank the state, so one mis-logged night doesn't whip the forecast.
  • Online adaptation (adaptGain / adaptThreshold / adaptDecay) and the volatile mode exist to let the filter open up under sustained surprise. Measured on the consented cohort (2026-10-18), they are effectively inert in served v8: the volatile mode never wins and the drift slope sits at its floor. Unfreezing them (five candidates, globally or per person) cost more people than it helped at every lead time, so the stiffness is kept.
  • Neither model predicts better from less history, at any lead from 1 day to 6 weeks (history-truncation test, 2026-10-18).

Context inputs (tags and self-report)

Self-reported context does move the forecast. Past tagged nights inform the state, and a declared "right now" context shifts the upcoming sleep, decaying over the following cycles. Guardrails:

  • MNAR-gated and per-user-scaled. A tag you never log carries no weight. In v8 a tag needs at least 12 supporting nights (contextTagMinSupport) and is shrunk toward zero by a 26-night prior (contextTagPriorCount); tagAdaptGain (≈ 0.48) scales how much your tags move your forecast from your own evidence, so a tag that doesn't predict your sleep shrinks toward zero for you.
  • The effect is small. When tags first went live (June 2026, the tag-aware v3 model), the measured shift was about 10 minutes; an ablation on served parameters in October 2026 found the PRC, context and scallop terms together moved report-card scores by about zero.

The per-covariate weights cover illness, stress, medication, social obligations, screens, blackout, custom tags, mood, cognition and quality. Further zeitgeber channels (caffeine, alcohol, melatonin, workout, light therapy, blue blockers, morning light, food timing, menstrual cycle) are wired with zero weight until a trained candidate earns them.

Personalization

Per-person version selection is live (above). The deeper per-user tracks built in June 2026 — a skill-weighted blend of a global and a tuned track, and per-user MAP fitting of base parameters shrunk to the global prior — are implemented but off in every served config (blendLambda = 0, mapFitParams empty).


6. Tuning and validation

Training and promotion

The adaptive model is gated, not hand-waved. A trainer fits candidate parameter sets on fresh consented exports (each person's series kept separate, with mutation timestamps), and a candidate reaches anyone only through a catalog gate: it must beat the current served model on held-out nights, it is admitted as an inactive per-person alternate, and per-person pins are then republished from each person's own held-out audit. Automatic activation also requires a signed architecture approval of the model code's fingerprint. That approval sat pending from 2026-08-01 until 2026-10-18, so no trained version was promoted or admitted in that period. The first admission afterwards, v19 (projection damping of the harmonic, projHarmDamp 0.6, on v8 parameters), was admitted on 2026-10-18 for 2 people whose own audits favored it by more than 1%. The lineage to the IMM class is written up in research/2026-06-03-from-regimes-to-kalman.md; the June crop (legacy-feature harvest, the data-gap fix, tag-awareness) in research/2026-06-14-personalization-and-harvest.md. The model registry is research/model_registry.json.

No-time-travel rule

At each prediction step a model has access only to data preceding the predicted entry. Since 2026-09-09 the forecast's phase anchor and its observation cut-off are separate: rows logged after the forecast origin are withheld before fitting, and a sleep whose wake is already known can still anchor phase. For multi-day forecast tests, the model is frozen at a split point and asked what it would have predicted over the next 7 or 14 days (or 30 days for full-trajectory scoring) without learning from those future sleeps.

Disruption-slice testing

Average scores over a whole month can hide failure modes users actually feel — a forecast that becomes useless right after one skipped night. Candidates are also scored separately on the rows immediately following a disruption (defined as a residual > disruptionThresholdH ≈ 4 from the prior fit).

The report card (what users see about accuracy)

Every user can see how each model would have done on their own nights (Analysis › Accuracy; src/lib/reportCard.js). It is a walk-forward backtest on the served path: each graded night is predicted only from the history before it, by the same code and per-person settings that serve live forecasts (Smart with the blend and layers it actually served that night, §8; Adaptive with the person's pinned version). Three cards are graded:

  • Smart (§3, §8);
  • Adaptive (§5);
  • Plain guess — the no-learning baseline: the previous onset plus the median of the last 14 onset-to-onset intervals, over every non-fragment sleep.

The default grade is a composite: closeness (median miss), hit rate (share of nights within the person's tolerance, default 1 h), window width, calibration of the 80% window (no penalty within one standard error of 80%, and no penalty for over-covering), consistency (spread of misses), absolute accuracy (zero at five times the tolerance) and durability (a forecast frozen for two weeks). Each component is scored 0–100 against the person's own plain-guess baseline (matching it scores 50), so a chaotic rhythm is not graded on a steady sleeper's scale. A model whose median miss is worse than the plain guess is capped at C. Users can switch to a pure "how close" grade (median-miss bands, A+ ≤ 30 min … F > 245 min at the default 1 h tolerance), a skill grade, or a coverage grade. Grades are computed in each sleep's own logged timezone (fixed for Smart on 2026-10-13 and for Adaptive on 2026-10-14; before that, a card viewed from another timezone mis-graded by the zone gap).

Current results

All figures below are walk-forward on consented logs, per person first and then across people, each person on their own settings.

DateCohortResult
2026-10-14 export95 people, next-night onsetper-person median miss: Smart 109 min, Adaptive 120 min; Smart closer for 71 of 95
2026-10-18consented cohortnominal 80% band covers next onset: Adaptive 81.1%, Smart 76.8%
2026-10-2193 logs with ≥ 30 graded nights, served pinsAdaptive composite grades A 19 / B 20 / C 49 / D 4 / F 1; Adaptive misses by more than the plain guess for 54 of 93
2026-10-2295 people with ≥ 30 entries, default dialsSmart grades A 42 / B 35 / C 18, none below C; median miss 89 min

Two findings bound what any model can claim here. First, misses are dominated by skipped or unlogged cycles, not timing error: on 2026-10-22, the 8% of nights that missed by more than 6 h carried 41% of all miss-minutes. Second, on steady sleepers the plain guess is a hard baseline: the October 2026 analysis traced most of Adaptive's C grades to the global clock-time terms in v8 (switching off the 2.32-day periodic wave alone improved the composite score by +3.15 points, 95% CI 1.85 to 4.54, with 41 people up and 6 down). No replacement has been promoted as of this revision.

History: the June 2026 gate (tag-aware v3 vs v2)

Kept for the record; v3 is no longer the default. Scores are a skill number where lower is better (held-out forecast error normalized against a fixed reference):

StratumPrevious adaptive (v2)Tag-aware v3Change
Aggregate1.4621.142−22% (win)
Easy (steady) users0.9920.971−2% (win)
Hard cohort (free-runners)0.8370.890+6% (small regress)
Held-out chaotic free-runner tail0.5710.595+4% (small regress)
Easy-user band coverage (cov80)0.69 (overconfident)0.83honest

The v3-era training corpus was the tuning snapshot of 2026-05-17 (n ≈ 18 active sharers plus two longer externally shared datasets).

Honest qualifiers

The edge of either learned layer is not unconditional. On calm, very-steady patterns the plain guess already has little to fix. Tag effects are real but small. And the training-corpus shape matters — a few long histories shape the global fit disproportionately, which is why per-person version pins exist: a global knob tuned to the cohort average can quietly hurt an atypical user.


7. Exclusion rules

A session is excluded from drift math (but still counted toward sleep totals) under these conditions:

  1. Fragmentation — a session starts within fragmentationThreshold hours (default 6 h, per-user 2–12 h) of the previous wake. Consecutive sessions that each begin inside that window are grouped into one episode (a single circadian cycle; episodeGroups() in src/lib/model-primitives.js), bounded by a 14 h span cap measured from the episode's first onset — a run of short wake-ups stays one night, but a genuinely long gap opens a fresh cycle. If the prior part ended in a forced wake ("interrupted, not finished"), the cap stretches to 30 h, so an alarm-broken night with a later catch-up still reads as one cycle. A user-marked fragment always joins the episode before it. Within an episode the earliest real sleep anchors the cycle's phase: the first part that reaches the nap threshold, or that ended in a forced wake even if shorter. Every other part inside the window is a fragment — since 2026-08-22 a large later part is no longer released as its own cycle (that size test could prevent a short-then-long night from ever grouping, whatever the user's setting; at removal, 484 sleeps relabeled across 38 people and no displayed τ moved). Two exceptions keep a part out of the fragment set: a part the user excluded by hand, and an auto-excluded nap that comes before the anchor (it may be the only sleep logged in a long gap). Threshold is per-user configurable; polyphasic / ME-CFS / split-sleep users typically lower it to 3–5 h, while clean monophasic N24 users can raise it to 8 h. Reference: markFragmented().
  2. Manual fragment (fragManual) — the user explicitly marked the session as a fragment of the previous sleep. Never a phase marker; also removed from the Smart estimator's anchor timeline (§3 step 1) so the next real sleep measures its cycle against the previous real sleep.
  3. Nap auto-flag — a session shorter than napThreshold hours (default 4 h, per-user 0.25–8 h) is auto-flagged (tooShort). User can override per-entry; manual choice wins. Whether an excluded nap is bridged or still anchors the next gap is decided by §3 step 1.
  4. Cycle gates (Smart estimator only, per transition rather than per session) — a gap below max(18 h, 0.7 × τ) ("snapback") or above max(34 h, 1.3 × τ) ("crash") is not a valid single-cycle phase marker and is excluded from the τ point estimate (§3 step 3), unless the user's sleepless marks account for the gap. The legacy Clean estimator still uses the user's flat postSleeplessThresholdH (default 30 h) instead; Raw includes everything with modular wrap.

Manual exclusions via the per-entry xDrift toggle override fragmentation and nap auto-flag in either direction, and a manual include additionally bypasses both cycle gates. xDriftManual is preserved separately so threshold changes after the fact don't clobber the user's explicit choice. In the database, x_drift = excluded and x_drift_manual = set by hand in either direction: a manual keep is x_drift = false, x_drift_manual = true. The app's automatic nap and fragment classification is recomputed at runtime and is mostly not persisted.


8. Confidence and forecasting

The Smart forecast

The default forecast (buildForecast() in src/lib/forecast.js) projects whole cycles forward from a phase anchor — the most recent real sleep (not a fragment, not an auto-excluded nap; a hand-excluded real sleep still anchors) — at a constant period:

τ_f          = 24 × cycleDays + drift_f
center(i)    = anchor_onset + (cycles_to_now + i) × τ_f
σ_pred(i)    = σ × √i ,        σ = max(σ_obs, 1 h)
band shown   = 1.4 × σ_pred(i)   (80% window: ± 1.282 × 1.4 × σ_pred)

This is the standard model for accumulated jitter in a free-running oscillator. It is not a calibrated frequentist interval; the 1.4 widening was chosen so the displayed window matches empirical coverage (measured 2026-10-18: 76.8% inside the nominal 80% window). Real-world predictions degrade beyond ~7 cycles due to compounding τ drift and zeitgeber perturbations.

drift_f is the user's displayed drift basis (§3 step 8), with two forecast-only substitutions: when the user has sleepless answers, the forecast steps by the estimate made without them, provided both agree on cycleDays (measured on 101 logs: 17 improved, none worse); and for users the multi-sleep plan selects, the rate comes from a fold periodogram (Sokolove–Bushell χ² periodogram over the last 120 days in 30-minute bins, 28-day recency weighting, scanned 20–36 h) instead of the gap-space drift. A typed manual drift overrides both. The fold rate feeds the forecast only; it never changes the displayed τ.

Confidence %. Since 2026-09-21 the percentage beside each predicted sleep is the probability that the onset lands within the user's own "close means within" tolerance (default ±1 h) under a normal error with the personal, widened σ. It previously reported the mass inside ±σ_obs, which cancelled to the same 68 / 52 / 44 / 38% sequence for every user.

Per-person layers on the Smart forecast (October 2026)

Each layer below is switched on per person only when that person's own recent nights, replayed walk-forward through the same buildForecast, show it helps (one-sided paired Wilcoxon), with probation and a ledger that sends it back home if it starts losing. All three were validated on the shipped functions with zero people "truly worse" (median, mean and night count all agreeing) against placebo runs.

  • Bedtime/wake blend (live 2026-10-13; src/lib/smartAnchor.js). The next onset is counted from a blend β ∈ {0, 0.2, …, 1} of the last bedtime (0) and the last wake-up (1); a neighboring β takes over on p < 0.2 with 45+ nights of history. Measured on 153 logs: 28 people truly helped, 0 truly worse. This replaced the separate "Wake-anchored" model.
  • Tonight's awake curve (live 2026-10-13). Tonight's onset is moved part of the way toward "last wake-up + the awake stretch that followed this person's past sleeps of similar length" (20+ pairs), at the stricter bar p < 0.1 with 60+ nights. 6 truly helped, 0 truly worse.
  • Clock-hour shape (live 2026-10-13). Six 4-hour clock bins of how much longer or shorter this person's cycle runs when it starts there (relative coordination), applied to nights 2+ only, fading with a 7-day e-fold, at strength 0, 0.25 or 0.5. Each past sleep is binned in its own logged timezone. 9 truly helped at 3/7/14 days, 0 truly worse.

Predicted sleep length is the user's typical sleep, or for split sleepers the whole-night total when their own replay shows it predicts better.

Analysis views

The Analysis tab also surfaces a Phase Position scatter (Position or Residual mode) comparing predicted and actual onset for each historical entry. In Adaptive mode the per-entry predictions are the Kalman filter's one-step-ahead predictions made before observing each row (post-predict, pre-update) — genuine forward-in-time fits. In Smart mode the comparison line is a constant-drift projection at the active drift basis, which is retrospective, not walk-forward; use the report card (§6) for Smart's forward accuracy.


9. Co-variates captured per session

For research purposes, each sleep entry can carry:

Core covariates

FieldTypeMeaning
q1–5Self-rated sleep quality (an import's 0 means unrated)
wakeTypenatural / forcedWhether the user woke spontaneously
stressboolSelf-reported stress affecting this session
illnessboolSelf-reported illness
medicationboolSelf-reported medication change/use
socialboolSocial obligation affected timing
mood1–5Post-wake mood
cognition1–5Post-wake "brain fog → sharp" rating
lightOutdoorcomma-separated subset of {morning, midday, evening, none}Bright outdoor light timing (multi-select May 2026; legacy single-string rows parse to a 1-element set)
screensBeforeboolScreen exposure in the 2 h before onset
blackoutboolFull darkness during sleep
customTagIdsstring[]References into the user's custom-tag table
adHocTagsstring[]Embedded one-off tags (max 10, capped 32 chars each)

Zeitgeber bundle

Stored as JSONB on the zeitgebers column of circadia_sleep_entries (allow-list: ZEITGEBER_KINDS in O+K src/lib/circadia-zeitgebers.ts). All fields optional; missing means "not tracked" (not "false"):

FieldTypeMeaning
morningLight1hboolBright outdoor or 10,000-lux exposure within 1 h of waking
firstFood2hboolFirst food within 2 h of waking
workoutsubset of {none, morning, afternoon, evening, night}Exercise timing (multi-select; none is exclusive)
workoutTimeHH:MMOptional precise workout time
caffeineboolAny caffeine intake on this day
caffeineWhensubset of the same periodsWhen caffeine was taken
caffeineTimeHH:MMOptional time of last caffeine
lastFood3hboolLast food at least 3 h before sleep onset
melatoninboolTook melatonin on this day
melatoninTimeHH:MMTime melatonin was taken
alcoholboolAlcohol on this day
alcoholTimeHH:MMTime of last drink
lightTherapyboolUsed a light box (Jul 2026)
lightTherapyTime / lightTherapyMinHH:MM / 0–240 minWhen and how long
blueBlockersboolWore blue-light blockers (Jul 2026)
blueBlockersTimeHH:MMWhen they went on
windowLightboolDaylight through a window, indoors (Oct 2026)
windowLightWhensubset of {morning, midday, evening}When
windowLightNearwindow / room / farProximity to the window (asked instead of lux)
windowLightMin0–600 minHow long
periodlight / medium / heavyMenstrual flow this day (opt-in)
pmsboolPMS symptoms (opt-in)
onsetTypeforced / crash (absent = natural)Kept awake past the body's time, or an involuntary early onset

Two further optional bundles: a sleep diary (diary JSONB: into bed, tried to sleep, out of bed, number of wakes, WASO minutes, rested, inertia minutes), and per-day check-ins in circadia_checkins (e.g. 1–5 alertness, absolute timestamps; §2).

All co-variates are optional, user-reported, and intended as exploratory signals — not ground-truth zeitgeber measurements. Per-user hide controls let users opt out of any zeitgeber they don't track; hidden fields don't appear in either the log form or the Analysis correlation panels.

Provenance

Each entry records tz (§2), import_source (absent for hand-logged rows; otherwise the importer: Sleep as Android, Fitbit, Garmin, Garmin via a Guava CSV export, Eight Sleep, Oura file import or background Oura sync, Apple Health, AutoSleep, Sleepmeter, Huckleberry, Circadia CSV, or pasted text), logged_via (e.g. api for the personal write API), and subject_id when a caregiver logs for a child or dependent. A parent's log and a child's log are separate series even when they share nights.

Per-session derived covariates (computed, not user-reported)

FieldMeaning
postSleeplessWhether the gap to the prior onset exceeded the user's post-sleepless threshold
fragmentedWhether this session is a non-anchor part of an episode (§7 rule 1)
ambiguousPer-row marker that wrapped drift on a clean transition lands beyond ±ambiguousThresholdH (default 8 h; likely a wrap artifact). Surfaced in the log; not used by the Smart estimator

Sleepless periods

Sleepless periods (intentionally skipped sleeps, sometimes lasting 30–48 h in free-running patients) are logged in a separate circadia_wake_periods table (date plus hours awake) to preserve the actual onset/wake timeline. The Smart estimator reads them as testimony about long gaps (§3 step 3); sleep-debt math counts them as ordinary sustained wakefulness. The table has no soft-delete column.

Per-user settings (cloud-synced)

SettingDefaultRange
postSleeplessThresholdH3018 – 72
fragThresholdH62 – 12
napThresholdH40.25 – 8
ambiguousThresholdH86 – 18
bridgeManualExclusionsoffon / off (§3 step 1)

Settings sync across devices via the user_settings JSONB column on circadia_user_profiles. The Smart estimator and all derived stats honor the user's own thresholds, not a global default. The τ prior also follows the user's lens (24.0 h for entrained lenses).


10. Cohort-side diagnostics

For admins analyzing the shared cohort, several additional aggregates are computed in computeUserStats() (src/components/AdminPanel.jsx):

Per-user, recomputed at view time

StatMeaning
tauH, sigmaTauFrom classifyTau() (the tier τ and its band, §3 step 8); withheld for tiers 3–4
sigmaObsFrom the legacy estimatePeriod() (§4), not the Smart estimator
driftMean (Smart), driftClean, driftRawtauH − 24, and the Clean and Raw means (§4)
driftMagnitudeUnsigned mean of per-cycle drift magnitude (abs(drift_i)) over clean transitions only
driftMagnitudeAllUnsigned mean across raw drifts (includes post-sleepless wraps). For users whose pattern is dominated by long awake stretches, this is closer to lived per-cycle shift than driftMagnitude
lowCycleRateBoolean flag: Smart and Raw drift diverge by > 3 h. Indicates most transitions are filtered as post-sleepless and Smart silently undersells. Surfaced with ⚠ in the cohort table
unwrapApplied, wrapDetectedFrom the legacy estimator's wraparound-unfold pass (§4) — diagnostic only, no longer feeds the headline
tauOriginal, tauUnwrappedLegacy τ before vs after unfolding, when applied
postSleeplessCountNumber of transitions whose gap exceeded the user's post-sleepless threshold

The cohort tauH is the tier τ computed without the user's sleepless marks, gap answers or merged-night collapse, so it can differ slightly from the τ that user sees (§3 step 8). For research-grade per-person τ, reproduce the app's full display path.

Cohort-level views

  • τ distribution — histogram (0.25 h bins), mean, median, range
  • Profile aggregates — self-id, treatments, entrainment status counts
  • Median |drift| (unsigned shift magnitude across cohort)
  • Median σ_obs (cycle jitter across cohort)
  • % with any covariate flag — coverage indicator
  • Sleepless gaps — count + mean / max / total hours across cohort

Cohort-vs-individual toggle

The cohort table can be computed two ways:

  1. Per-user settings (default view) — each user's stats computed with the thresholds they chose. This matches what each user sees.
  2. Generic defaults — every user re-run with the same thresholds (30 h post-sleepless, 6 h fragmentation, 4 h nap, 8 h ambiguous). Useful for apples-to-apples comparisons.

Drill-down

Admin can load any sharing user's anonymized dataset into the normal Log / Chart / Predict / Clock / Calendar / Analysis views (read-only; the admin's own data is untouched). This is the recommended way to inspect individual users. The preview scores entries with the previewed user's thresholds and timezone (not the viewer's), so the numbers shown match what that user sees in their own app; an estimator-diagnostic panel can also dump the exact entry list and estimate the app computed, as ground truth for offline reproduction.


11. Data structure and sharing

A user who has opted in via "share my data" appears in the admin/research view under an anonymousId. Their identity-linked user_id is never surfaced to admin or research consumers. Per-session co-variates and onset/wake timestamps are exposed in full, but unlinked from any account-level identifier.

Free-text fields not exposed by the shared API:

  • Sleep notes
  • Wake-period notes
  • Profile free-text fields (region, comorbidities_other, treatments_other)

Custom and ad-hoc tag names are a separate opt-in (per-user "share my tag content", consent item tag_names). Users sharing data can keep tag content private; their tags then appear as opaque codes.

The opt-in is reversible. A user whose sharing is withdrawn drops out of every subsequent pull (the shared-entries API answers not_found_or_revoked). The underlying sleep data remains under the user's account and is not auto-deleted.

There are three consent items in Circadia that govern data use, each grantable and withdrawable independently:

  • Simple sharing — for the Circadia developer and internal product improvement only. Tag-tuning, model fitting, bug investigation.
  • Research-level sharing — same anonymized data as simple, plus pre-consent for future academic research collaborations under a data-use agreement (DUA), plus the structured research profile below.
  • Publication consent — a separate, stricter additive gate that permits a user's anonymized data to be included in publicly-deposited or publication-bound datasets. Publication consent is never implied by simple or research consent; it must be granted explicitly per item. Without it, a user's data is shareable under DUA but not publishable.

Effective consent is the latest row per (user, item) in circadia_consent_records, merged with the legacy circadia_share_consents table for users with no granular record (pre-May-2026 opt-ins), minus accounts in circadia_cohort_exclusions. Any consent-gated analysis must apply that merge; querying the granular table alone undercounts. Caregiver-managed profiles (e.g. a child) carry their own research opt-in and guardian consent.

The research profile (optional, per person or managed subject) records: age bucket, sex at birth, country and region, latitude, age or life stage at onset, what came right before onset, clinician diagnosis, entrainment status, comorbidities and suspected comorbidities, current treatments, and caregiver status. Server-side storage for "what made it worse", family history, a daylight-exposure bucket, longitude (rounded to 0.1°), a per-treatment history (now or past, and the result) and eleven further conditions was added on 2026-10-26; the in-app questions for those fields were not yet live on that date.

Important: Research-level pre-consent does not authorize ad-hoc data transfer. The maintainer will reach out to research-tier sharers individually before any specific collaboration begins.

The developer training export is collected by npm run pull:shared-datasets in the Circadia repository through the scoped admin API (an O+K key with circadia:shared-data:read or circadia:research-data:read). The former direct-DB scripts/export-circadia-shared.mjs is retired because it bypassed managed-person consent filtering. Simple and research tiers permit in-house algorithm tuning; simple-tier data is not authorized for external researcher sharing merely by signing a DUA. External research must use eligible research-tier data and its applicable agreement. Public deposits additionally require publication consent and a frozen, dated eligibility snapshot.

The observation export preserves pseudonymous per-person series, current consent read receipts, and row mutation times. Private plans/settings, checkins and lens payloads are not added to pooled training under the existing sleep-log grant. See docs/circadia-data-dictionary.md and docs/circadia-observation-export.md for scope and limitations.

The reference endpoints are:

  • GET /api/circadia/admin/overview — cohort-level aggregates (computeCohortSummary())
  • GET /api/circadia/admin/shared-entries?anonymousId=… — one sharing user's anonymized sleep log and wake periods, with covariates, zeitgebers, custom tag IDs and (when shared) tag names

12. Current cohort state (as of 2026-10-27)

Counts pulled from the production databases on 2026-10-27 (02:17 UTC) via computeCohortSummary() (O+K src/lib/circadia-cohort-summary.mjs, the function behind the admin overview). Accounts flagged in circadia_cohort_exclusions (operational test accounts) are excluded.

  • 256 signups, 225 email-verified.
  • 118 users sharing data (simple or research): 108 with research-tier consent, 10 with simple-tier only. Consent items counted independently: simple 118, research 108, publication 101, tag names 47. 7 sharers have no sleep entries yet; 111 sharing users have logged sleep, and all 111 are τ-ready by the summary's rule (≥ 3 onset-to-onset transitions of 18–36 h).
  • Shared-cohort entry totals: 46,958 raw / 46,944 distinct onsets (14 duplicate-import rows).
  • Largest individual logs in the shared cohort (distinct sleep onsets; spans are first to last onset):
    • 4,056 sleeps across 2,674 days (~7.3 years).
    • 3,872 sleeps across 3,932 days (~10.8 years).
    • 3,257 sleeps across 7,132 days (~19.5 years) — the longest span.
    • 2,972 sleeps across 3,195 days (~8.7 years).
    • 2,777 sleeps across 2,565 days (~7.0 years).
    • These long-history datasets are the strongest single contribution to the τ-estimation work.
  • Observation spans: 4 to 7,132 days per sharing user with at least 3 entries (median 130 days).
  • Sleepless gaps logged (shared cohort): 391 events (mean 31.1 h, median 30.4 h, max 65.9 h).
  • Caregiver-managed profiles: 6 (3 minors), 2 of them opted into research.
  • Cohort skews adult sighted N24SWD and DSWPD; some ME/CFS overlap; some self-described irregular-sleep-wake / polyphasic patterns.
  • Geographic distribution: US-majority but not US-exclusive. Country and free-text region are collected in the research profile; region is not exported in the shared dataset (see the re-identification caveat in the data dictionary).

Analysis cohort. The cohort research atlas (Circadia repository, research/cohort-atlas/) works from a narrower, cleaned set. Its latest pull, 19 October 2026 (methods re-audited 20 October), analyzed 90 research-consented people with 37,815 own-log nights, after record cleaning (wall-clock time zones, same-night re-import duplicates, nested and reversed rows), nights in pieces folded with the app's own 14 h onset-based episode rule and summed for sleep length, and each person's own thresholds applied.

These numbers move daily — they describe the cohort on the dates stated. Email if you want a current snapshot for a specific analysis.


13. Known limitations

  1. Self-report bias. Onsets and wakes are user-entered; some users log via memory after the fact. Imports from wearables are device estimates, not polysomnography. There is no PSG ground truth.
  2. No DLMO. Melatonin onset is not measured; τ is inferred from onset timing alone.
  3. Cycle counting in long gaps. When gap_h exceeds the upper cycle gate (Smart) or the post-sleepless threshold (Clean), the algorithm cannot infer how many body-cycles elapsed. Unless the user's sleepless marks account for the gap, these pairs are dropped from Smart/Clean drift math rather than imputed. Raw and |drift all| diagnostics include them with modular-wrap math. Skipped and unlogged cycles also dominate forecast error (§6).
  4. Sleep-debt model is rough. The 14-day cumulative debt is a linear shortfall vs target — it does not implement the allostatic slow variable of McCauley 2009. Process S uses Borbély 1982 parameters with no individual calibration.
  5. Co-variates are correlational only. The dataset does not support causal inference about, e.g., evening screens shifting tau, because exposure is self-reported and unblinded.
  6. The Adaptive model is fit on a small corpus. Long histories shape the global fit disproportionately, and served v8 applies global clock-time terms that measurably hurt steady sleepers (§6). Patterns dissimilar to anything in the training pool may take longer to fit.
  7. Tag-correlation panels use n ≥ 3 per group as their reporting floor. These are exploratory; they should not be treated as statistically calibrated.
  8. Polyphasic / multi-sleep rhythms are only partly handled. The gap-space model assumes approximately one main sleep per circadian cycle. Structurally polyphasic logs are detected (§3 step 7) and collapsed to one marker per cycle, which can misrepresent genuinely split schedules. A chi-square fold periodogram (§8) now sets the forecast rate for multi-sleep sleepers; the displayed τ still comes from the gap-space path.
  9. Not every surface uses the shared drift rule yet. The headline τ, the per-row drift labels and the forecasts use the rules in §2–§3. About a dozen secondary displays (e.g. the share-card drift line, the Actogram drift strip, some Analysis cofactor views) and the per-user rhythm summary contributed to cohort views still rebuild drift with a ±12 h clock-face wrap, so a day longer than 36 h can read as a backward shift there. A fix was built behind a preview on 2026-10-25 and is not live as of this revision.
  10. "Per calendar day" sleep averages in Analysis › Rest currently average only days on which a sleep started, so a free-runner's zero-sleep days are dropped and the figure reads high. A corrected version (every civil day counted, logging gaps over 72 h excluded) is in preview, not live.

14. Key references

Estimator priors and α

  • Borbély AA. 1982. A two process model of sleep regulation. Hum Neurobiol 1(3):195–204.
  • Czeisler CA et al. 1999. Stability, precision, and near-24-hour period of the human circadian pacemaker. Science 284(5423):2177–81.
  • Daan S, Beersma DGM, Borbély AA. 1984. Timing of human sleep: recovery process gated by a circadian pacemaker. Am J Physiol 246(2 Pt 2):R161–83.
  • Duffy JF et al. 2011. Sex difference in the near-24-hour intrinsic period of the human circadian timing system. PNAS 108 Suppl 3:15602–8.
  • Hayakawa T et al. 2005. Clinical analyses of sighted patients with non-24-hour sleep-wake syndrome: a study of 57 consecutively diagnosed cases. Sleep 28(8):945–52. (The τ prior, §3 step 4.)

Homeostatic / debt

  • McCauley P et al. 2009. A new mathematical model for the homeostatic effects of sleep loss on neurobehavioral performance. J Theor Biol 256(2):227–39.
  • van Dongen HPA et al. 2003. The cumulative cost of additional wakefulness. Sleep 26(2):117–126.

Zeitgeber correlation backing (used by Analysis tab panels)

  • Khalsa SB et al. 2003. A phase response curve to single bright light pulses in human subjects. J Physiol 549(Pt 3):945–52.
  • Damiola F et al. 2000. Restricted feeding uncouples circadian oscillators in peripheral tissues from the central pacemaker in the suprachiasmatic nucleus. Genes Dev 14(23):2950–61.
  • Stokkan KA et al. 2001. Entrainment of the circadian clock in the liver by feeding. Science 291(5503):490–3.
  • Burke TM et al. 2015. Effects of caffeine on the human circadian clock in vivo and in vitro. Sci Transl Med 7(305):305ra146.
  • Chang AM et al. 2015. Evening use of light-emitting eReaders negatively affects sleep, circadian timing, and next-morning alertness. PNAS 112(4):1232–7.
  • Mason IC et al. 2022. Light exposure during sleep impairs cardiometabolic function. PNAS 119(12):e2113290119.
  • Youngstedt SD et al. 2019. Human circadian phase-response curves for exercise. J Physiol 597(8):2253–68.
  • Ebrahim IO et al. 2013. Alcohol and sleep I: effects on normal sleep. Alcohol Clin Exp Res 37(4):539–49.

Robust estimation and periodograms

  • Huber PJ. 1964. Robust estimation of a location parameter. Ann Math Statist 35(1):73–101. (The M-estimator used for the headline drift location, §3 step 5.)
  • Hampel FR et al. 1986. Robust Statistics: The Approach Based on Influence Functions. Wiley. (MAD-based scale.)
  • Sokolove PG, Bushell WN. 1978. The chi square periodogram: its utility for analysis of circadian rhythms. J Theor Biol 72(1):131–60. (The fold periodogram, §8.)

Modeling family

  • Kalman RE. 1960. A new approach to linear filtering and prediction problems. J Basic Eng 82(1):35–45. (Lineage for the state-space update.)
  • Gordon NJ et al. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc F 140(2):107–13. (Particle-filter founding paper; the May 2026 Adaptive model.)
  • Doucet A et al. 2001. Sequential Monte Carlo Methods in Practice. Springer. (General reference for the retired particle-filter model.)

15. Contact

Research-level anonymized data may be made available to researchers under a written data-use agreement and the product's current privacy terms. Simple-tier data is developer-only.

Contact: Dayah Dover, dayahdover@gmail.com

Please reach out to discuss before any data flows. Research-tier consent only pre-approves the act of sharing in principle; specific collaborations require a separate conversation.

If you publish using Circadia-derived data, please cite as:

Dover D. Circadia: free-running sleep tracking for N24SWD. Open alpha, 2026. https://circadia.owlandkestrel.com

A formal DOI deposit on Zenodo is planned.