04 — Decision policy & context

For an optimizer with little oversight, the ranking function is the core of the product. If it is left as "scored on value/effort", the real policy is whatever order an agent happened to produce: not visible, not versioned and not auditable. This doc defines it explicitly, versions it and makes it the subject of the Learn stage.

The scoring function (policy v1.0 — versioned, logged on every ranking)

Every backlog item enters with a hypothesis card: predicted Δ ledger value (with a named revenue family and causal path), P(success), fully-loaded cost (inference + operator minutes at OPERATOR_RATE), time-to-signal, change-class, measurability tier (doc 03), and kill criterion (state + date), registered as the change's watch window when it ships. An item without a card is not ranked. Cards come from three sources: scheduled sweeps of the assets' own signals and anomalies, the operator, and scouting sweeps of outside sources (doc 13); cards from scouting record their sources and may have a window (opens/closes + decay) — the score is multiplied by a decay factor as the window closes, and a card with a window may skip the batch schedule but never the intake requirements.

score = (P_success × ΔV_predicted × F) / C_fully_loaded / max(T_signal, 2wk)

• ΔV_predicted — dollars per month + the enterprise-value delta that implies (24–40× durable monthly profit; concentration-discounted per doc 00).
• F — forecastability multiplier (0.5–1.2): revenue durability and source diversity. A gain that depends on a search algorithm scores below a durable gain with the same EV; the Onfolio case shows that the variance of a revenue stream is an allocation criterion, not a minor detail.
• C includes the predicted review minutes — review time, the scarcest resource, has a price, so ten trivial PRs no longer outrank one batch that includes its own verification.
• T_signal discounts long-delay bets (WSJF's cost-of-delay shape).
• P_success priors come from the ledger, per change-class, updated by Learn from the calibration report. This closes the loop: the system's estimates are graded against outcomes and the grading changes the estimates.
• The card is editable until the change ships. Its change entry then records ΔV per month, P_success, C and T_signal as they stood, and grading uses those values, never a later edit.

Why not a live bandit: SEO/revenue rewards arrive over weeks-to-months; at that delay, online bandit algorithms get too little feedback and end up optimizing noise. This policy is EV-scored batch prioritization with scheduled reallocation on the attribution schedule: it uses the bandit approach (posterior-weighted, with protected exploration) but runs in batches.

Portfolio allocation

• One queue across assets; scores are cross-asset comparable because they're denominated in dollars.
• The explore sleeve: 15% of build capacity (config), reserved for speculative, asymmetric bets that rank low on EV because of low P_success — justified as value-of-information and by the innovation-portfolio finding that the small transformational sleeve produces a disproportionate share of long-run returns. Sleeve bets require explicit kill criteria and bounded cost.
• Concentration risk is a standing portfolio input: the asset-level Google-organic share and single-revenue-family share appear on every Tower view; diversification work gets F > 1.
• Kill criteria are pre-committed and automated. Every bet's state+date is checked by cron; when the date passes, the bet is parked automatically and closed out in the ledger (sunk cost does not count in later decisions). Asset-class-level kill criteria exist too (the "content-site forecastability" question is periodically re-evaluated for every asset, with numbers).

Goodhart guards (the constraints that make optimization safe to run)

Goodhart's law: once a metric becomes the target of optimization, it stops being a reliable measure of the goal. The evidence is unambiguous: no proxy is immune to gaming; frontier agents game gameable harnesses at 25–100% rates; instructions do not prevent it, structural constraints do.

1. Guarded metric bundles. No target metric may be optimized without its paired counter-metrics: traffic ↔ thin-page ratio + originality gate; CTR ↔ post-click engagement + brand-rule compliance; RPM ↔ retention + CWV floor; positions ↔ revenue-per-session. Divergence between a rising proxy and flat/falling counter-metrics freezes the change class and sends the case to the operator, so reward hacking is detected while it happens.
2. Hard invariants, not scoring terms (blocking, non-negotiable, and enforced outside agent write scope where possible): brand/legal rules from context packs; no near-duplicate/scaled output; publishing-velocity caps per asset (the scaled-content abuse penalty depends on the proportion of low-value pages and applies to the whole domain, which can destroy the asset's value); churn caps per surface (frequent title changes look like manipulation to search engines and make measurement impossible); analytics/ consent/holdout/guardrail-threshold code is operator-only.
3. A sudden gain triggers a review. A sudden jump in gains from scaled changes is flagged for review, not treated as success: the same kind of change that makes a proxy jump is what Google's SpamBrain detects.
4. Rules for LLM judges. Rubric-scoring LLMs are calibrated against operator-labeled anchors, are never from the same run that authored the change, and are stress-tested before their scores are relied on (doc 05).

Context packs

Two tiers: per-asset AGENTS.md (positioning, brand and legal rules, do-not-touch, CI bar, deploy model, conventions) and the portfolio context in NoticeOS (current focus, scoring config, global guardrails).

The staleness mechanism (the facts in a pack become inaccurate within hours of being written unless they carry dates):

• AGENTS.md contains reviewed: <date>, so a reader knows how old its facts are; the pack is dated, with facts separated from rules.
• Stable rules vs volatile state are separated. Rules (brand, never-do, conventions) are in AGENTS.md; volatile facts (application states, pending reviews, temporary holds, dated postures) are in a STATE section with per-entry expiry dates — expired entries are treated as unknown, not as true.
• Drift detection: post-incident rules (doc 05) are added to the pack via PR; a quarterly pack review is a scheduled, ledger- costed task. One canonical source: what agents note in memory is written into the pack, and agents follow the pack, not memory or unwritten team knowledge. When the two conflict, the pack applies until it is amended.

Operator overrides (first-class inputs, not exceptions)

In the queue (doc 10) the operator can approve, boost, veto (reason required), edit, or park any card, and can hold a change in a T2+ class before it ships. Every override is recorded against the card and the scoring-policy version that ranked it. Vetoes and boosts are a primary input to the Learn stage: a class the operator keeps vetoing has a miscalibrated value model or a missing constraint, and the quarterly policy revision must fix the cause of the pattern, not only record the labels. Hypothesis cards must list their data sources from the integrations catalog (doc 11) — a card whose evidence can't be shown in the "why this rank" expander fails intake.

What Learn changes here

Quarterly (or after every 20 closed changes per class): P_success priors re-fit from realized outcomes; F multipliers re-examined against revenue durability; surrogate metrics re-validated or retired; scoring-policy version bumped with a changelog entry — the policy itself is under version control and its diffs are annotations on every asset timeline.