24 — Integration monitoring
An operator should learn about missing data from System Health before noticing an empty card. A connected account shows that access was configured. It does not show that every report, live feed, calendar, or notification is working.
One health record per capability
packages/contract/src/integration-health.ts is the coverage inventory and shared state projection. Its tests require every connected provider and every catalog entry in config/integrations.json to have a monitor or an explicit coverage gap. A new provider cannot be shown as green without a monitor. Family-based adapters expand the collector's actual report plan by property and date; they do not copy a list of report names into monitoring.
Archive adapters encode family as a canonical JSON tuple of report name, report date, query, and device. Current producers record aggregate reports, with the query/device slots empty: DataForSEO combines its tracked queries and devices into one report outcome and fails that report when a required member call is incomplete. Independent query/device incidents are not exposed. Yesterday’s successful report cannot clear today’s failed report. target identifies the property. Delivery targets identify the destination; real notifications and test sends have separate capability IDs. A later successful notification can demonstrate destination recovery, but never changes the failed receipt or automatically resends that earlier notification. Saved query-history reads use stored R2/Postgres evidence and are not new Google operations.
Each record is scoped to workspace, connection revision, capability, asset, target, and report family. Realtime active users, hourly traffic, daily Analytics, Search Console, and each archive family have independent outcomes. Connection tests and site discovery are separate capabilities. They cannot clear failed production reads. A credential replacement starts a new revision; old evidence remains in history and cannot confirm the replacement's access.
The workspace comes from the trusted server context. A standalone installation has one workspace; a column in this design does not implement tenant access control. Connection revisions and target identifiers are opaque IDs. Tokens, private calendar URLs, webhook addresses, account emails, raw error messages, and report payloads never enter health rows or grouping keys.
| State | Meaning |
|---|---|
| Working | The same capability returned a usable result for its current obligation. A valid zero or empty response is usable. |
| Failing | An observed failure has no later same-capability recovery. Aging does not clear it. |
| Stale | The last usable result no longer meets the collector's due obligation. |
| Never run | Configured and monitored, but no attempt has been recorded. |
| Idle | An event-driven action has no pending obligation, or an on-demand feed is not currently requested. Previous outcomes remain visible. |
| Paused / Disconnected | The operator disabled the capability or removed its connection. Neither is green. |
| Unmonitored | Enabled, but no reliable observation adapter is installed. |
| Unknown | The observer or inventory cannot be read, or its evidence is invalid. |
The reducer orders observations by attempt start, not arrival. A slow success from an older request cannot clear a newer failure. On a tie, the failure is kept until a later success records recovery. Invalid/future timestamps make availability unknown. Last attempt, last successful attempt, and the date covered by a report remain separate facts.
Evidence and failure detection
Scheduled collectors already keep signal_runs, archive_runs, and mediavine_runs. Their adapters read those records, including the failed family/date and incomplete reports. Source lag is distinct from transport failure: a successful response containing no new finalized date must not be labelled a provider outage or filled with zeroes.
Live/on-demand operations record their outcome after response parsing and semantic validation, inside the provider adapter. Catching only HTTP errors misses HTTP-200 error envelopes, malformed data, and partial reports. Failure categories are access, rate limit, exhausted budget, network, provider error, invalid report, incomplete report, configuration, and monitoring failure. Each category maps to reviewed product copy and an action; the provider's raw error text is never displayed as a diagnosis.
The recorder is awaited. If it fails, successful business data can remain visible, but the response explicitly reports monitoring unavailable. A recorder exception must not turn an irreversible successful action into a retryable action failure: outbound sends retain their delivery ID and success receipt, with a separate monitoring error. Monitoring never resends messages.
NoticeOS's scheduled runner continues to write job_runs. Those rows describe workflow execution, not arbitrary integration incidents. Configuration stays in config_documents; credentials stay in credentials; neither becomes an incident log. Account-wide credential status is setup evidence; capability health is not derived from it.
Durable state and recovery
The approved schema is implemented in the Postgres baseline. What is monitored — provider, connection revision, capability, site, target and family — is one capability_targets row that state and events refer to (0001_baseline.sql).
integration_capability_state keeps the latest attempt and latest successful attempt. integration_health_events keeps first failures, changes of failure category, and recovery transitions. Success polls update current state without creating an append-only event every 30 seconds. Existing report histories keep their own detailed attempts; event pointers identify those source records.
The writer uses one transaction. It first locks the monitored target, so observations of one target apply one at a time. It then inserts an incident event from the current row only when the incoming attempt is newer and changes the failure state/category, then conditionally upserts current state under the same ordering predicate. A deterministic event ID includes scope, connection revision and attempt ID, so retries are idempotent. A late success may advance last_success_started_at only if that success itself is newer than the stored success; it never overwrites the newer current failure. Updates and event insertion must share one transaction. The adapter tests must execute reordered, duplicate, and concurrent observations against this rule.
The configured inventory selects active connection revisions. Disconnecting or remapping removes old rows from current health without deleting their history. Reading health never substitutes the last good inventory if the current one is unavailable: completeness is then unknown. The observer's own unavailable state is explicit even when no incident could be persisted.
One presentation across the product
System Health, Integrations, the Wall, and asset details consume one health payload and state projection. The read endpoint reads stored evidence only; viewing System Health must not trigger API calls, quota consumption, paid research, or notification delivery. The payload includes monitoring coverage, affected assets, capability labels, last attempt, last success, next permitted retry, and the safe corrective action. All visible times use the viewer's local clock or relative age, with exact UTC references in tooltips.
Group failures only when evidence identifies the same cause and scope, such as the same account access failure or the same property quota bucket. Sharing an HTTP status or provider is not enough to group failures as one cause. A group expands to its individual capabilities and assets. Green means a recorded operational success, not a successful business outcome.
Notification delivery failures appear in System Health itself, so the broken notification channel is not the only way to learn of its failure. Adding new outbound outage notifications is separate from observing existing deliveries.
Retry and freshness ownership
The provider adapter holds one bounded retry policy and follows Retry-After. Pages share reads; hidden pages do not continue polling. Health reads and connection badges never start provider probes. Existing paid-request budgets, report cadence, measurement thresholds, and operator permissions remain set by their existing policies. Monitoring does not add new obligations for event-driven integrations or bypass provider quotas.
GA4's display read now shares successful live counts for 30 seconds and hourly Core reports for 15 minutes. Cache misses share a database lease; HTTP 429 responses pause frequent requests for five minutes by default, or for the period in Google’s explicit Retry-After when one is sent. The durable lease preserves cooldown even if the platform response cache is lost. Cache entries are local to a data center; another region may temporarily show “Checking traffic” until it can obtain a new reading. Actual observation time is retained on cached values and failures. Hourly failure does not blank successful realtime counts.
Google documents separate Core and Realtime budgets and daily replenishment; request allowance cannot be restored by retrying. The diagnostic category and next attempt do not state when the quota resets. References: Data API quotas and quota management guidance.
Implementation state
Migration 0035, the atomic recorder, operation adapters and read-only GET /api/integrations/health are implemented. Live GA4, calendar feeds, connection tests, discovery, account-credit reads and Discord delivery record validated outcomes. Scheduled and explicit report producers attach the exact credential snapshot to their own target and record the attempt ID as it is written. Overlapping runs are never associated by a time-window query. Recording failure is reported separately from a successful delivery or report.
The read model requires saved configuration. Missing or malformed inventory is unknown; bundled configuration is not a substitute for the active inventory. Stored credentials carry the edit revision from the same row as their fields. If a connection changes during an internally resolved operation, its result cannot verify the new connection. Unversioned report history remains available but does not establish the current connection’s health. A newly observed collection supplies that proof. Google’s revision includes its referenced service-account key and OAuth app dependency without exposing those values. An unmatched newer source attempt invalidates an older green result. A known failure remains open until recovery is recorded for the same connection.
Archive histories remain scoped by report and date. Incomplete or truncated responses cannot clear a failure. A SERP panel stopped at its own tracked depth is complete, not truncated in this sense. A DataForSEO report reads stale a day after it fell due, because a missed snapshot is never collected later; other archive reports read stale after two missed cadences. Each property archive read returns at most 256 latest report/date outcomes, prioritizing unresolved failures, and shows an explicit count and unknown coverage item when additional dates exist. Last success is selected by attempt start, independently of the current failure. Current-state reads select only the active provider revision. Non-archive operations are capped at 2,048 per provider; archive reads select current obligations plus unresolved failures before applying a 256-row limit. Excess rows make coverage unknown; resolved historical dates do not consume current coverage. Event history filters the latest 100 stored transitions to active scopes.
System Health, the Wall and Integrations consume the same saved-health projection. System Health never starts a live provider read. The Wall still requests the counts and calendar data it displays, and reports a failed monitoring write separately from those results.
The interface groups by provider and operation; grouping does not imply a shared cause. Expanding an operation shows its asset/report evidence, last attempt, last success, next eligible check and safe guidance. Unknown coverage has its own filter and stays gray. An operation with both a failure and a coverage gap appears in both filters. A recorded recovery is shown in the recent changes list. Times are relative or local, with UTC references.
Provider setup, explicit tests and production operation outcomes remain separate. A saved credential never promotes an unverified operation to green. An unavailable provider inventory does not erase another provider’s known results, while an empty inventory cannot produce an all-clear.