06 — Operations & self-observability

NoticeOS is itself one of the assets it monitors, and it is held to every standard it applies to the others. This doc describes the routine mechanisms that automation depends on: NoticeOS's own report, its spend meter, its kill switch and its plan for vendor failures.

The NoticeOS report

NoticeOS sends the same nightly report as every other asset, under the same contract (doc 02):

• Ingest freshness per asset per data source (the asset's report, GSC, GA4, revenue): a scheduled push that stops without an error becomes an error flag within two nightly cycles, so it is found in two days rather than three weeks later. Two cycles rather than one is deliberate: a single late night (a retried collector, a provider that published at noon) would raise an error that resolves itself by morning, and an operator who keeps seeing errors that clear by themselves starts ignoring the flag. The age is REPORT_MAX_AGE_HOURS — 48h, REPORT_CADENCE_HOURS × REPORT_STALE_MULTIPLIER in packages/contract/src/reporting.ts — and both places read this one number: the hourly ingest-freshness cron raises its error past that age and the Tower counts the property stale past the same age, so one payload never contains an open error and a "fresh" count for the same property. To flag a stopped data source sooner, change the constant, not this document. An asset whose Settings declare that it sends no nightly report (config/constants.json no_nightly_report) is not expected to send one: owesNightlyReport in the same file leaves it out of both counts, it is never flagged, a flag still open for it is resolved on the next hourly run, and its card shows a neutral "No report".
• Spend: inference + API dollars per stage (Sense/Attribute/Decide/Act), per asset, vs budget; cost-per-landed-change trailing 30d. Today only the API half is metered — see Budget enforcement below.
• Throughput & quality: hypothesis cards ranked, changes shipped by tier, accept rate, reliability-ledger movements, open deferrals, audit findings.
• Cron/system health: run successes, retries, queue depth, store size.

The Tower shows NoticeOS's own health first. If NoticeOS is unhealthy, none of the other data it displays can be relied on.

Running the installation

NoticeOS runs as one Docker Compose stack of four services: noticeos (the app: Tower, ingest and the runner), postgres, dolt and an optional backup worker (Docker guide). A development stack's app runs a checkout's live source; a production stack's app runs a prepared image. Running the runner directly on a host is not supported. .local/stack.json selects the stack, and agents and operators use one set of commands defined in this repository. An installation set up with Docker alone has no checkout: its folder holds the selector (stack.json), a copy of the stack definition and the noticeos launcher, which runs each command below from the app's image as <folder>/noticeos <command>, with no -- (Install with Docker).

Question / action	Command	Contract
Is the OS healthy?	pnpm os:status	Each service's Docker state and health, whether the database has every migration this code carries, the app's commit, whether main is ahead, and any service that does not run as installation.env declares. An open port alone does not count as healthy.
What just happened?	pnpm os:logs -- --lines 200	Recent logs from every service, or -- <service>; -- --follow keeps printing new lines while you reproduce a problem.
How big is the store, and how fast is it growing?	pnpm os:capacity	Rows, bytes and daily growth per table, insight snapshots, the raw archive and scheduled-job durations, read from the running OS; metadata only (doc 26). -- --json prints the raw result.
Recover the app	pnpm os:restart	Restarts the stack in dependency order, waiting for each layer's health; a failed step stops the sequence.
Apply an edit to installation.env	pnpm os:apply	Shows each change and the services it recreates, and applies it after you type apply: only those services, databases first, each waiting for health. If a service does not become healthy, the command returns the stack to the settings it last ran healthy with. It never points a database at other data, renames the project, builds or pulls an image, or applies a migration.
Put merged work live	pnpm os:update	Takes the image CI published for main's commit when it was built from exactly this checkout's source, or builds it here when none is published, shows the plan and applies it after you type update, replacing only the app container and rolling back automatically if the app does not become healthy again; pnpm os:rollback returns to the previous image. Without a checkout, noticeos update moves the image and the stack definition copied out of it together to main's newest published version, recreating each service whose declaration changed; noticeos rollback returns to the version before.

Merging is not deploying. A production app runs a fixed image that a merge never changes. pnpm os:update is the step after a verified merge: it stops and prints pnpm os:migrate when main contains a migration the database has not applied, and leaves a change to the Postgres roles to the operator as maintenance. Only the app container changes; the databases, the backup worker and the installation's state mount stay in place. How it works is in scripts/README.md.

The app container is healthy when the runner reports itself healthy, with its heartbeat (a regular signal that the runner is still running) at most 60 seconds old, its Tower ready and its scheduler armed; a configured backup worker responds; and the Tower and the ingest endpoint both respond inside the container. Output written to disk has common credential formats removed before it is written, and the commands remove them again when reading. Log lines are capped at 64 KiB; the runner log rotates at 5 MiB with five historical files; scheduled-job JSONL logs keep 30 days.

Status, logs and capacity are read-only. A routine restart or update after authorized ordinary runner/Tower work is within that work's scope. Operator approval is required before restarting or updating when a pending change affects auth, billing, security headers, DB migrations, consent, analytics/tracking, holdouts, or guardrail thresholds.

Startup, restart, apply and update never apply a migration. Postgres is the only operational store; applying a migration is operator-only:

1. pnpm os:status: what runs, whether main is ahead, whether the database is behind.
2. pnpm os:backup: a backup you can restore.
3. pnpm os:migrate, then pnpm os:migrate -- --apply (db/postgres/README.md).
4. pnpm os:update.

The app keeps running during step 3: migrations are additive unless the release's own instructions say otherwise (release policy).

For a provably new, empty installation only, pnpm os:setup may create its data volumes, apply the committed frozen schema and bootstrap one workspace (approved exception). Existing installations and production remain operator-only. The checks and steps are in the Docker guide.

Downtime recovery is bounded and based on the 30-day job-run record. Startup runs at most the latest missed run of each allowlisted scheduled job, one at a time: 15-minute jobs within one hour, hourly jobs within two hours, daily jobs within 36 hours, and the weekly DataForSEO collection within eight days. It never re-runs every missed run, and it runs the shortest-interval jobs first so the 15-minute Wall jobs never wait behind the weekly collection. Only the jobs in the catch-up plan have their regular runs paused, each until its catch-up run finishes; every other job keeps running on its normal schedule, and the log records when the catch-up pass is done. A recorded failure counts as an attempt rather than causing a restart loop. The task database health check and the task snapshot already run at startup and are not re-run. Unknown cron expressions are excluded, and the Worker rejects one anyway (unknown_cron, running nothing). Migrations, restore, seed/config apply, app updates, kill-switch work, and every surface that is permanently operator-only are not catch-up jobs. The catch-up pass runs only already-approved collection definitions; it cannot edit the measurement channel.

Budget enforcement (hard caps; alerts are not enforcement)

What is in force today. One cap, and it is enforced: config/constants.json monthly_caps.data_usd — $25/mo of metered data spend, reserved before every paid DataForSEO read and failing closed. There is no model gateway and no inference cap, because no process in this repo calls a model: the analyzer is rule-based, and every model call happens in an agent session billed outside the OS, so no settings page shows an inference cap for spending that nothing measures. Everything below is the design that takes effect when the OS makes its first model call of its own — it is a specification, not a description of the running system.

All model/API traffic flows through a gateway with dollar-denominated, fail-closed limits (Cloudflare AI Gateway-class, fitting the existing stack; LiteLLM-class proxy equivalent):

• Per-run cap (default modest; sized per change-class), per-day cap per stage, portfolio monthly cap. When a cap is exceeded, the gateway blocks the call or falls back to a cheaper model, so the system keeps running at reduced capability instead of stopping.
• Spend-velocity circuit breaker independent of totals ($/min beyond N× the planned rate trips it before the daily cap would; with alerts alone, a loop that keeps calling can run for days).
• Loop detectors: identical-call dedup, A↔B round-trip counters, recursion-depth caps, error-streak halts.
• The kill switch must also stop work at the provider: cancelling provider-side runs is part of the switch, because a provider keeps billing after the local process exits.
• Caps exist per credential, so spending on a leaked key is limited before anyone notices the leak.

The kill switch

Required before automated asset execution; not built in the first release. Asset agent execution remains manual; operators stop their external agents directly. Pausing NoticeOS's collection schedules does not pause an agent.

The future operating contract requires one documented, rehearsed action to stop the system: disable all NoticeOS crons

• revoke the GitHub App installation token + disable the gateway keys. Target: executable in under two minutes from a phone; drilled quarterly (the drill is a scheduled task in NoticeOS's own task list, and the drill result is a metric in NoticeOS's own report). Partial switches per stage (pause Act, keep Sense) allow smaller interventions.

Secrets & access

• The GitHub App private key: rotation cadence (quarterly), storage (CF secrets, never in repos), break-glass procedure written next to the key's location, scope audit at each rotation.
• Portfolio API credentials (Google signal service-account keys for GSC/GA4, DataForSEO, ad networks, affiliate networks): per-vendor rotation notes and expiry/revocation alarms (a disabled Google key stops Sense collection without raising an error; the ingest freshness check catches it).
• Least privilege: if the OS org is compromised, the exposure is limited to the App key and read credentials; asset production secrets are never stored in NoticeOS.

Bootstrap secrets vs. integration credentials

This section is the one place that defines the split, and other docs link here: four bootstrap secrets stay in the environment, and every other credential is stored in the product.

Four secrets stay in the environment permanently. The OS needs them to start, so the product cannot store them: a credential that can only be read once the OS is running cannot be the credential that starts it:

Secret	What it bootstraps
CREDENTIALS_KEY	the key every stored provider credential is encrypted under (openssl rand -base64 32). The store cannot hold its own encryption key.
OPERATOR_TOKEN	the bearer token the runner and the operator's own scripts present to the ingest's write routes. It authorizes the same process that would have to read it from the store.
ASSET_TOKENS	one token per asset, checked on an inbound report and presented on the outbound pull (doc 11). It is the OS's own shared secret with each asset, not a third-party account.
DATABASE_URL	the address of the installation's own Postgres: the application login's connection string, postgresql://noticeos_app:<password>@127.0.0.1:5432/noticeos?sslmode=disable (db/postgres/host sets one up). The store cannot hold its own address.

Where they are kept. Locally, all four are in the installation's secrets file, workers/ingest/.dev.secrets.json: under the stack's state mount (/state/workers/ingest/.dev.secrets.json in the app container), which pnpm os:setup writes. The first three reach the Workers as bindings (compiled into .dev.vars). DATABASE_URL never does: the runner reads it at every start, checks it as the application login, and passes it to the dev server's environment and nowhere else, as the POSTGRES binding's local address (CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_POSTGRES, scripts/database-address.mts). pnpm os:status reads the stack's application address from its Postgres secrets folder, only to read the database's migration record, and prints no part of it. pnpm os:setup takes the application address from the new installation's postgres-secrets/. A missing or unusable address, or a database behind the code, stops the start with a one-sentence message that names the file and does not print the address; only pnpm os:setup, on a new, empty installation, applies schema. Deployed on Cloudflare, the address is stored in the Hyperdrive configuration, not as a Worker secret.

Every third-party provider credential is connected in the product. Google Analytics / Search Console, Bing Webmaster, DataForSEO, calendar feeds, and whatever doc 11 adds: connected, tested and disconnected on the Tower's /integrations page, stored AES-GCM-encrypted in noticeos.connection_secrets, with the connection's public facts in noticeos.integration_connections. A Save inserts the next secret version and removes the previous version in one transaction, without rewriting ciphertext (Postgres model). Every collector reads the store first.

Env bindings remain the legacy fallback for existing installs. A provider with no stored row falls back entirely to its binding and keeps working — the card shows a neutral Legacy env chip and an Import from this machine button, which moves the whole secrets file into the store in one click; there is no import command. At startup, the runner logs one line naming each provider still on env, and /health reports the same in one line. The fallback is not deprecated and nothing requires the move; its cost is portability, because a fresh install would need those secrets copied by hand.

The legacy local source

Locally, an operator who has not moved to stored credentials edits the formatted, private workers/ingest/.dev.secrets.json. Nested maps such as GOOGLE_SIGNAL_ACCOUNTS remain real JSON instead of minified dotenv strings. The runner compiles that source to .dev.vars at every start, so an edit takes effect with pnpm os:restart. Google's property routing remains in GOOGLE_SIGNAL_ACCOUNTS, while each encoded service-account key is extracted to its own bounded GOOGLE_SERVICE_ACCOUNT_* binding. This keeps the readable account-centric source without exceeding a Worker text-binding limit. The generated file is derived output, not a second place to edit. Deployed Workers receive the routing map and per-account credentials as separate encrypted string secrets.

Vendor failure & degradation policy

For each signal source, the written policy is to skip the missing data and never estimate it:

Source	On failure	Never
DataForSEO	skip cycle, mark panel gap, retry next cadence	interpolate positions
GSC/GA4	mark unfinalized/missing; suppress dependent flags	alert on partial windows
Bing Webmaster	retain the last-good series, mark the source degraded, retry next daily cadence	convert an unreported lagging date into zero traffic
Ad/affiliate reporting	retain provider-reported estimates with an explicit estimated status; book payments on reconciliation (Mediavine)	invent revenue or label an estimate reconciled
Model provider	gateway fallback chain; if none, pause Act	swap models without recording an annotation

Vendor pricing/policy changes (model prices, API terms, network thresholds, crawler policies) are external annotations — they change ledgers and baselines and must be visible on every affected timeline.

The implemented Google and Bing collectors record every attempt in signal_runs, isolate provider/property failures, retain the last successful observations, and show the effective source state on each property card. GA4/GSC run every 15 minutes and degrade after two missed cadences (30 minutes). Bing Webmaster's rank-and-traffic dataset updates daily, so it runs at 02:30 UTC and degrades after two missed daily cadences (48 hours); polling it every 15 minutes would not make the provider data fresher. The broader flag and notification job for NoticeOS's own report is still outstanding. GA4 marks only today provisional in the chart. GSC also treats today as provisional when response metadata is absent and moves the boundary earlier when the provider supplies first_incomplete_date. Bing is final through the last date it returns, and the chart adds no zero values after that date. Dependent anomaly rules must ignore incomplete or absent dates.

GA4's current display data is fetched on demand, not by another cron or measurement-history collection. An open Home/Wall requests data from the Tower every 30 seconds; Tower calls ingest over a private Service Binding, and ingest concurrently makes two runRealtimeReport requests/property — both trailing 30- and 5-minute ranges, and the per-minute rows for the Wall's per-minute bar chart (MinutePulse) — shared by every display for one minute, plus one bounded Core hourly report for today and seven days ago. It reuses a short-lived read-only Google access token but stores nothing. The current-day line stops after GA4's newest reported hour; unreported future hours are null rather than zero, while the same-weekday reference line covers its complete day. These reads are isolated from /api/wall: a timeout, quota failure, or invalid response leaves the rest of the card available, shows no zero in place of missing data, and lets the client keep showing its last good values, labelled “Reconnecting.” The response requests property-quota telemetry so a future cadence change can be based on observed token consumption.

The analysis-grade provider collections run separately from those live chart collectors. They write bounded daily GA4/GSC/BWT and weekly DataForSEO gzip JSON to the private/local RAW_SIGNALS R2 binding and append-only attempt manifests to archive_runs. Google re-fetches the previous four completed dates; BWT retains one current provider snapshot for each of six report families; DataForSEO retains ranked-keyword, backlink, and Google/ChatGPT mention snapshots and exact metered cost. Identical content references the existing immutable object. A report failure never deletes a prior dump and never turns an absent provider row into zero. Locally, Wrangler persists local R2 under .wrangler/state; remote access and provisioning are explicit operator actions.

Offline analysis may publish one compact, content-addressed noticeos.asset_insight_snapshots row after operator review. That snapshot is the only way the Tower reads from the archive: it keeps rule, evidence, confidence, source, window, and caveat together, while the raw provider rows stay in R2. Publishing identical content is idempotent.

Property health and Needs Attention count only flags with both resolved_at IS NULL and disposition IS NULL. Mark read sets a disposition on this event; Resolve closes its underlying condition. A later recurrence is a new event, while a recovered metric auto-resolves its prior open flow event. This way the health indicator shows what needs the operator's attention now, not whether the property has ever produced a warning.

Reproducibility

• Models are pinned per stage; a version change is a deploy of the OS (annotation on every asset timeline), because it changes build quality and judge behavior with no other visible change, and therefore changes both attribution and the reliability ledger.
• Scoring-policy versions, prompt/context-pack versions, and gateway configs are all in version control; any ranked backlog or shipped change can be traced to the exact policy + pack + model that produced it.
• Where config lives: ALL config — including doc 02's per-asset anomaly thresholds and the Tower's layout/threshold edits — is a store document seeded from the installation folder (or the config/ defaults) and exported back to the installation folder (below); the store's other tables hold data, never tunables. "Tunable per asset" means per-asset values inside the versioned config.
• "Never tunables" means never measurement rules. The rule exists so that anything which can change a verdict is visible in a diff — thresholds, the scoring policy, the tracked-query list that determines spend. It does not apply to credentials. A provider secret is per-install operator state that must never enter version control, changes when the operator changes it rather than when the repo changes, and has to work in a deployed Worker with no filesystem — so it is stored encrypted in the store's noticeos.connection_secrets versions, alongside the public facts in noticeos.integration_connections, under the bootstrap secret CREDENTIALS_KEY. It is entered and tested on the Tower's Integrations page (Postgres model). Operator settings are kept in the store for the same reason. The measurement channel stays operator-only permanently: it is a hard invariant above. config/tower.json holds the shared Home/Wall countdown; applied values are sent in /api/wall, while the live clock always uses the display's local civil time. config/serp-panel.json defines the per-property tracked head terms the weekly DataForSEO collection pays to check — the one place a tracked-query list is written down, and the document that determines that spend, so it is a config document like every other tunable. config/entities.json defines the portfolio's legal entities and the asset ids each one owns — the one place which entity owns this asset is written down. It is config rather than a column on assets because it changes when legal ownership changes, not when the schema changes, and because a column would have needed a migration for a fact the product can store as a setting; it is edited on /settings → Ownership and on each asset's Settings → Identity card.
• The files are the seed and the export; once seeded, the store holds the authoritative copy. Both reasons for the rule still apply, and neither requires a Worker to read the file. pnpm config:seed loads each document from the installation folder, or from the generic default in config/ where the installation has none, into noticeos.config_documents as a whole JSON document keyed by its file name without directory or extension (config/tower.json → tower); APIs still name the config/ path. pnpm config:export writes them back to the installation folder with stable formatting, so a diff is still where a threshold change is visible and a commit is still the config version. noticeos.config_changes is the machine-readable record beside it — who changed what, why, and the version before and after (Postgres model). The result: a deployed Tower can save a setting. A field stored in a file is writable only where a Node process has the repo on the same machine; a store document is writable from any deployment. Seeding selects the stored document. Both Workers read a document from the store when it is there and fall back to the copy compiled into them when it is not. A reachable, initialized Postgres store with an unseeded document uses that compiled copy; an unavailable store fails the read instead of being treated as unseeded. The runner logs which state it is in, once, at startup. The measurement channel is unaffected: a guardrail threshold is operator-only whichever table stores it, and seeding changes where a value is read, never who may change it. The ingest's own collectors read it per run. One read per cron fire — readCollectorConfigs in workers/ingest/src/config-store.ts, resolved at dispatch.ts and passed to each collector as the parameter it already took — covers the six documents those jobs need: pull.json, counters.json, integrations.json, ga4-custom-dimensions.json, serp-panel.json, and constants.json's flag defaults (which writePulse resolves at its own two entry points, since one of them is an asset's POST /api/pulse rather than a cron). Each run's structured completion line includes configSource per file — the word only, never a document — so the configuration last night's run used is recorded in the run's own log line rather than inferred by a reader. This is how a data-source mapping saved on an asset's Data sources tab reaches the next collection run with no restart; an unseeded install records file for every one of them and runs exactly as it did the day before.

Legacy names

The product was once called ReindexOS and is NoticeOS now. Everything a person or a contributor reads, every package (@noticeos/*) and every code and wire name uses NoticeOS, and scripts/product-name.test.mjs fails if one does not. Two kinds of old name remain on purpose, and no other old name may remain.

Read for compatibility, never written. A running installation and the tasks it filed still contain these; the product reads the old name when the new one is absent, and each old spelling appears in exactly one module:

Old name	New name	Where it still appears	Read by
REINDEX_OS_INSTALLATION_DIR, REINDEX_OS_WORKER_CONFIG_ROOT, REINDEX_OPERATOR_TOKEN	NOTICEOS_INSTALLATION_DIR, NOTICEOS_WORKER_CONFIG_ROOT, NOTICEOS_OPERATOR_TOKEN	an operator's shell or script	scripts/product-env.mts
reindex_key, reindex_kind, reindex_asset, reindex_rule, reindex_source, the reindex-handoff label	noticeos_*, noticeos-handoff	handoff tasks in every project's tracker	packages/contract/src/task-metadata.mts
reindex_panel_asset, reindex_panel_date, reindex_push_asset, reindex_task_map_asset	noticeos_*	panel-review, unpushed-work and task-map tasks	packages/contract/src/task-metadata.mts

Created with NoticeOS names. Naming a resource that does not exist yet costs nothing, so everything a new installation creates is named in one place, scripts/resource-names.mts, and scripts/new-install-names.test.mjs checks that none of it uses the old name:

New resource	Name	Created by
the Tower and ingest Workers	noticeos-tower, noticeos-ingest	the checkout's Worker configs, for every installation
the Postgres database and R2 bucket	noticeos, noticeos-raw-signals	the checkout's Worker configs — what a stranger deploys and what pnpm os:setup sets up
the Postgres database and role	noticeos, noticeos_app	the Postgres profile

A store made under other names keeps them in its installation folder's resource-names.json (config/resource-names.README.md); the Tower's dev server applies them over both Worker configs.

Kept as they are. Renaming these would move or orphan live data, change a service only the operator may change, or break a reference stored in another system, for no benefit to anyone using the product:

Name	What it is	Why it stays
the product's old slug, as an id	the id of NoticeOS's own asset row (a historical row)	an id that every stored row, report and token is keyed on. The row is always shown as System and identified by is_os, never by this id; the ReindexOS its display_name still stores is legacy data that no screen displays
reindex-os-raw-signals	the R2 bucket an older store was made under, named in that installation's own resource-names.json, never in the checked-in Worker configs	the local R2 store is kept under the bucket's name; resource names are operator-only
an older checkout's path	the maintainer's checkout	the task database server keeps its Dolt data inside it
the Dolt task databases and the ro- task prefix	the task database server	every task id and every project's link to the task database server
the private repository's remote	the maintainers' repository	agents never rename or move a repository or a remote
the old name in docs/reports and docs/artifacts	dated records	they keep their original wording

The task database

Operator and agent work is tracked in Beads (bd), a Dolt-backed task tracker. One shared Dolt SQL server holds the task databases; each configured repository is a task project that connects to that server in server mode and has no task database of its own. The saved asset ↔ prefix ↔ database map is kept in the store; the product task-map default is empty. The task contract defines those identities.

The declared task service runs the task database server independently of the runner. It is the stack's dolt service (Compose profile), declared to the app in its protected /state/task-client.json. Health checks and backups use that declaration. A missing or invalid declaration fails with an explicit error. The runner checks the endpoint at startup, on its 15-minute health check and before each snapshot read; it logs state changes and backs up the declared service nightly. It does not start, stop or supervise the task service.

Recover the exact declared project, service and volume through the Compose recovery procedure, with approval for changes to existing services or stores. Never start the retired source to recover a Compose task database server. Homebrew recovery applies only to an installation still using the legacy native profile. Dolt holds an exclusive write lock per database; a second server must never open the same data directory.

Tasks are coordination state, not signals. A task records what someone intends to do; a report records what an asset observed. The task database is a second store, separate from the ledger — nothing in it enters the report envelope or the signal contract (doc 02), and no task is evidence of an outcome. A record of intent must never be counted as a measurement.

Durability. The runner's nightly 04:00 UTC job backs up all three stores into the same dated directory under .local/backups/, on the same 30-day retention: the declared Compose Postgres through a custom-format pg_dump, the R2 raw-archive store's sqlite via sqlite's online-backup API (a raw file copy of a live WAL database can be left inconsistent if it is taken mid-checkpoint), R2's content-addressed blobs as plain copies (immutable once written), and one consistent DOLT_BACKUP per task database into <date>/beads/<database>/. The task database backup is a call to the live server and requires the task service to be running (not the runner). Every required store is attempted, but a failed store blocks publication, offsite handoff and pruning. The previous complete backup remains available. Each published dated dir contains a generated RESTORE.md that lists what it holds and how to restore it. Each store is consistent on its own; the stores are not captured at one shared instant, and the restore proof also checks their archive references. Postgres requires the installation's explicit postgres/profile.json; a backup never infers or starts a project. See backup and restore.

Off-machine durability. The finished dated dir is copied to <offsiteBackupDir>/<date>/, the folder this host names in its installation's host-backup.json (config/host-backup.README.md), which a sync client copies offsite with no extra tool or credential. Only the completed copy is synced, never live stores, so the sync client never reads a partly written file; pruning removes offsite copies on the same 30-day window. The DOLT_BACKUP dirs in that copy restore via dolt backup restore file://…, which also provides the task databases' off-machine copy; a Dolt remote is not built unless a drill shows the copied backups are insufficient. A backup that has never been restored is not known to work, so the quarterly restore drill restores from the offsite copy alone, all three stores, into an isolated scratch location (procedure in scripts/README.md).

Data retention

Reports, normalized signals, raw provider archives and their manifests, ledger, annotations: append-only, kept indefinitely (they are the calibration corpus). Content hashes prevent unchanged rolling re-fetches from duplicating R2 bytes. Run transcripts: 90 days in hot storage, then sampled archives (audit trail for the reliability ledger). Storage costs are ledger entries for NoticeOS's own asset, like every other cost.