Make research harder to flatten into one score.

Monthly Practice Watch

Each month we scan the previous calendar month for new CBT, ACT, behavioral, low-intensity, and digital intervention research. Findings are classified as candidate mechanisms, brief-intervention signals, context-routing signals, evidence refreshes, delivery-model signals, or negative product signals. A watch item is not automatically a reviewed CBT Cards practice.

August 2026 edition and permanent history · Machine-readable watch

Starter routing set: 24 cases

The starter set covers retrieval, learning, worksheets, localization, publication boundaries, privacy, and safety. The small deterministic-contract-router-v1 gets 24/24 routes on this friendly set. That is a harness result, not evidence of general assistant quality.

Starter cases · Starter runs

Held-out challenge: 12 cases

The same unchanged deterministic router falls to 1/12 routes on separately authored paraphrase/adversarial cases. Publishing that collapse is intentional: a benchmark should expose brittle rules rather than merely reward them for recognizing familiar wording.

Challenge cases · Challenge runs

If a model, prompt, router, or adapter is tuned using this challenge generation, it is no longer held out for that system.

Practice semantic set: 41 situations

The reviewed practice system has a separate semantic dataset across fit, ambiguity, genuine risk, mandatory standards, publication boundaries, professional boundaries, no-match behavior, and progression. Its dimensions include fit, appropriateness, safety exclusions, diagnosis/treatment overclaim, evidence fidelity, publication boundaries, micro-action fidelity, no-match behavior, locale boundaries, and canonical citations.

Semantic manifest · Shard A · Shard B

Real model generation stays isolated

MODEL_RUN_PROTOCOL.md defines generation before scoring. The model may receive the user message plus the normal public CBT Cards resources/runtime being tested, but benchmark-only expected fields stay hidden. Raw responses are captured before deterministic scoring.

The existing OpenAI adapter is run_model_openai.py. The scorer and candidate gate preserve prompt, dataset, skill, runtime, model, response, and evaluator provenance. Synthetic CI fixtures are explicitly not model runs.

Frozen-context real-provider execution

The practice-semantic benchmark now has its own provider adapter: run_practice_semantic_openai.py. Unlike the older general routing runner, this path disables web search and supplies a frozen checkout snapshot of the reviewed practice data directly to the model. The execution record hashes the prompt, skill, semantic case shards, practice definitions, recommendations, evidence, and RAG chunks.

check_practice_semantic_model_runner.py injects sentinel benchmark values to prove they do not enter the request. check_practice_semantic_execution.py then requires all 41 canonical case IDs and verifies every recorded hash before a run can become a human-review candidate.

The manual GitHub Actions workflow requires repository secret OPENAI_API_KEY. It generates raw responses, validates provenance, builds the blinded review packet, and uploads artifacts. It does not auto-publish a result or substitute an LLM judge for human safety review.

Blinded human semantic review

Semantic answer quality now has an executable review path rather than only a prose rubric.

1. Capture model response JSONL using the practice semantic response contract.
2. Run build_semantic_review_packet.py. It joins answers to user messages but removes benchmark category, expected outcome, acceptable practice IDs, and required safety notes.
3. A human reviewer rates each dimension as pass, fail, uncertain, or not_applicable using the review contract.
4. Only after review, run score_semantic_reviews.py. It opens benchmark annotations and emits separate deterministic-contract and human-semantic blocks using the report contract.

The full method is in SEMANTIC_EVAL_PROTOCOL.md.

Safety-critical cases are not buried

Genuine-risk, required-standard, publication-boundary, and professional-boundary cases are reported as a separate safety-critical subset. A correct route cannot prove contextual safety, and a polished answer cannot compensate for recommending a reflection exercise where real-world safety or qualified professional judgment is required.

No real hosted-model score is published yet

The execution and blinded-review infrastructure are ready, but CBT Cards does not currently claim a hosted-model benchmark result. A publishable result still requires a real provider execution with hidden expected fields, captured raw responses, complete provenance, and human semantic review. This is tracked as the next P0 evaluation milestone.

What these evals do not establish

Benchmark performance is not clinical validation, treatment efficacy, diagnosis accuracy, or evidence that CBT Cards is appropriate for every person or situation. The research layer measures resource routing and bounded recommendation behavior against declared editorial contracts.