# Adaptive Visual Budget Study

## Scope

- Calibration set: 26 completed V6 episodes from P12-P22, 79 visual-technical GT rows after excluding pure lip-sync and explicit out-of-scope rows.
- Holdout set: 7 completed episodes from P23-P24, 36 manually scope-cleaned visual-technical GT rows.
- Baseline: 28 selected visual candidates per episode, 89/115 type-aware timestamp hits (77.4%).
- No paid VLM calls were made; all policies replay saved post-VLM events.

## Main Results

| Policy | Candidate rule | Average visual candidates | Hits | Coverage | Change vs baseline |
|---|---|---:|---:|---:|---:|
| Current baseline | fixed 28 | 28.0 | 89/115 | 77.4% | baseline |
| Stable adaptive | stable K28 master queue; count events with risk >= 0.64; clamp 23-28; take prefix | 26.24 | 89/115 | 77.4% | -6.3%, zero hit loss |
| Current-selector adaptive | risk >= 0.68; clamp 23-27; rebuild temporal queue at chosen K | 25.48 | 89/115 | 77.4% | -9.0%, zero hit loss, but non-monotonic |
| Fixed compact | fixed 20 under current selector | 20.0 | 87/115 | 75.7% | -28.6%, loses 2 historical hits |

## Calibration And Holdout

| Policy | P12-P22 | P23-P24 holdout |
|---|---:|---:|
| Baseline fixed 28 | 65/79 (82.3%) | 24/36 (66.7%) |
| Stable adaptive | 65/79 (82.3%), avg 26.19 | 24/36 (66.7%), avg 26.43 |
| Current-selector adaptive | 65/79 (82.3%), avg 25.42 | 24/36 (66.7%), avg 25.71 |
| Fixed 20 | 63/79 (79.7%) | 24/36 (66.7%) |

## Stable Adaptive Holdout Counts

| Episode | Visual candidates |
|---|---:|
| P23/EP18 | 27 |
| P23/EP19 | 28 |
| P23/EP20 | 27 |
| P23/EP21 | 27 |
| P23/EP22 | 24 |
| P23/EP23 | 27 |
| P24/EP16 | 25 |

## Findings

1. Fixed 28 is an operational fill target, not an evidence requirement. The current runner fills the queue until the budget is exhausted even when the tail has weak evidence.
2. Current temporal stratification is not a stable prefix. Recomputing at K=20, K=21, and K=28 can replace earlier items, so measured recall is non-monotonic. A production adaptive queue should first build one stable master ranking and only truncate its tail.
3. Detector strength and VLM confidence currently save only about 6-9% with zero observed hit loss. Most selected events are protected or have high risk, so the signals do not strongly separate a safe stopping point.
4. VLM confidence is confidence in the VLM decision, not a calibrated probability that the event is a real human-visible defect. Detector scores are also normalized differently across families. Neither should be used as a hard universal probability threshold yet.
5. The GT oracle needs only 12.0 candidates on average in calibration and 14.0 in holdout to preserve the baseline hits. This shows substantial theoretical headroom, but current scores cannot predict that stopping point reliably.

## Recommended Implementation

Use a two-stage adaptive queue:

1. Build a stable master visual queue with maximum 28 items.
2. Always include protected signals and configured family safety floors.
3. Choose the delivered prefix length using evidence count, initially:
   - `min_items: 23`
   - `max_items: 28`
   - `risk_threshold: 0.64`
   - `selection: stable-prefix`
4. Keep fixed 28 as a fallback when score calibration is missing or the episode has unusually dense protected signals.
5. Log both the selected count and the omitted tail so Recall@K can be replayed without paid calls.

A more aggressive `min_items: 20` mode should remain experimental: it retained all 24 holdout hits but lost 2 of 65 calibration hits.

## Longer-term Calibration

Replace the heuristic risk threshold with calibrated event probabilities learned from adjudicated positives and hard negatives. Stop at the smallest prefix where the estimated probability of any omitted true defect is below a configured risk budget. Calibrate separately by family, because blur, deformation, and judder detector scores are not directly comparable.