← back to all models ← RevenueBench home
deepseek

deepseek/deepseek-v4.1-flash-1017

Mean 0.956 · 34/40 perfect tests · $0.33 total · modalities in: text, image · out: text · each card: the prompt → the correct answer (gold) → this model's actual answer

What do the modalities mean?

Modalities in means what you can send this model: text, images, files, video, or audio. Modalities out means what it can send back. Bench tasks feed text extracts, so modalities do not change the score. They are here so you know what this model can handle in real work.

data-crm 5 tests
stage-hygiene-auditPASS 1.00
Plain-English question: How much of the open DS2+ pipeline is missing a substantive why-buys, and is anything missing DS1 timestamps or out of order?
Prompt — what the model saw
Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.0277 · 104s · in 50,831 / out 30,931 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

Plain-English answer: 81.5% of the 124 DS2+ deals are missing a substantive why-buys — Alex Franklin 48, Cole Ingram 19, Dana Mercer 18, Bryce Harmon 15, Farid Osman 1 — and every deal has a DS1 timestamp with none regressed.
The answer in one line: n_deals = 156; ds2plus = 124; noncomp_pct = 81.5; bryce_noncomp = 15; no_ds1 = 0; regressed = 0 (+1 more below)
n_deals156
bryce_noncomp15
ds2plus124
no_ds10
noncomp_pct81.5
regressed0
per_owner_noncompAlex Franklin=48 of 61 (78.7%); Cole Ingram=19 of 22 (86.4%); Dana Mercer=18 of 22 (81.8%); Bryce Harmon=15 of 18 (83.3%); Farid Osman=1 of 1 (100%)
graded checksnoncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
DEFINITIONS

- DS2+ set: deals with stage DS2, DS3, DS4 or DS5 (124 of 156 open deals; 32 DS1 deals excluded).
- why-buys compliant: non-blank, 15+ characters after trimming, and not a placeholder (tbd / n/a / see notes / none / -). A value of the form LONG(n) is present-and-substantive by rule.
- Non-compliant rate: non-compliant DS2+ deals ÷ all DS2+ deals, overall and per owner; owners with zero DS2+ deals are shown as n/a, not 0%.
- No DS1 timestamp: t_ds1 blank.
- Regressed: any pair where an earlier-stage timestamp is later than a later-stage timestamp (t_ds_i > t_ds_j for i < j, both populated). Deals with missing intermediate timestamps are not counted, since no inversion can be tested.
- Owners resolved via owners.csv owner_id → owner_name.

WHY-BUYS COMPLIANCE, DS2+ (101 of 124 non-compliant = 81.5%)

Arithmetic overall: 101 / 124 = 0.8145 → 81.5%. Of the 124, 23 have a LONG(n) value (all compliant), 101 are blank; zero were short-but-present or literal placeholders.

By owner (non-compliant / DS2+ deals = rate):
- Alex Franklin: 48 / 61 = 78.7%
- Dana Mercer: 18 / 22 = 81.8%
- Cole Ingram: 19 / 22 = 86.4%
- Bryce Harmon: 15 / 18 = 83.3%
- Farid Osman: 1 / 1 = 100.0%
- Elena Sinclair: 0 DS2+ deals → n/a

Sum check: 48+18+19+15+1 = 101; 61+22+22+18+1 = 124.

DS1 TIMESTAMP COVERAGE

0 of 156 deals are missing t_ds1. Every deal in the file carries a DS1 entry timestamp.

STAGE REGRESSION

0 of 156 deals regressed. Every populated earlier-to-later stage timestamp pair is monotonically non-decreasing (e.g. Deal-C26D20: DS1 2026-05-15 → DS2 2026-05-26 → DS3 2026-05-26 → DS4 2026-06-03). Note two deals in DS3+ have no DS3 timestamp (Deal-403845 at DS5, Deal-317E6F at DS3), so ordering cannot be tested for them, and all 10 DS5 deals have t_ds5 blank — the DS5 column appears unused, which caps what this check can see.

THREE PROCESS FIXES

1. Gate the stage transition, not the report. Add required-field validation on the CRM stage-move action: no move into DS2 or later without a 15+ character why_buys. All 101 failures are blank, not sloppy, which means nothing currently blocks the move. Then backfill the 101 open deals as a named backlog with dates, largest-owner first (Alex Franklin 48, Cole Ingram 19, Dana Mercer 18, Bryce Harmon 15, Farid Osman 1).

2. Make stage timestamps generated, not typed. 9 deals have t_ds1 earlier than create_date (Deal-403845, Deal-927338, Deal-DCA846, Deal-4F775F, Deal-898FC5, Deal-D8ABF7, Deal-F67D31, Deal-D7E999, Deal-819506) and 20 deals have entered_current_stage that disagrees with the matching stage timestamp (Deal-5408B0 entered DS4 2026-09-01 vs t_ds4 2026-06-24; Deal-0D0211 entered DS3 2026-08-31 vs t_ds3 2026-04-13). Hand-typed dates make the regression check unreliable; derive all stage timestamps from one stage-history record and stop accepting manual entry.

3. Tie forecast category to hygiene. Deals at DS2+ without a substantive why_buys should drop out of COMMIT and BEST_CASE roll-ups and appear on the Monday pipeline review until fixed, and populate the unused t_ds5 column at the same time so the top of the funnel is auditable. With 81.5% of DS2+ deals failing, this is a per-rep coaching loop, not a training deck: start with the four named owners above and their blank-deal lists.

⚠️ File-mutation verifier: 1 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv`: `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` exists but this task has not seen its full c…
owner-resolution0.83
Plain-English question: Every open deal has a blank owner name — resolve each one to a real person using the owners table.
Prompt — what the model saw
The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0174 · 72s · in 37,257 / out 18,327 tok
✓ resolved_count✗ unmatched_none✓ deactivated_flagged✓ bryce_amount✓ alex_amount✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: All 156 deals resolve to six named owners; jake_amount maps to Bryce Harmon ($1,054,144) and justin_amount to Alex Franklin ($624,310).
The answer in one line: jake_amount = 1,054,144; justin_amount = 624,310; resolved = 156
jake_amount1,054,144
justin_amount624,310
resolved156
deactivated
per_ownerBryce Harmon=1,054,144; Alex Franklin=624,310; Dana Mercer=341,195; Cole Ingram=288,161.43; Farid Osman=4,134; Elena Sinclair=2,100
graded checksresolved_count · unmatched_none · deactivated_flagged · bryce_amount · alex_amount · no_invented

Model's actual answer

show the model's full answer
OWNER RESOLUTION — deals_open.csv (156 deals)

Resolution rule: deals_open.csv owner_id joined to owners.csv owner_id. No use of amount-weighted or stage filters; every row in the file is included.

1) Deals resolved
156 of 156 open deals resolved to an owner name (100%). All 156 rows have a non-empty owner_id.

2) Owner ids with no match in owners.csv
None. Zero unmatched ids, so zero deals are reported as unattributed.
Distinct owner ids present in the deal extract: 119337721, 701163055, 716654662, 83155923, 83155924, 84342457 — all six exist in owners.csv.

3) Owner ids mapping to archived/deactivated owners
None of the owner ids appearing on open deals are archived (archived = false for all six).
For completeness, owners.csv does contain two archived owners, neither referenced by any open deal:
- Gavin Porter — 1520255671 (archived, 0 open deals)
- Hugo Lindqvist — 77260721 (archived, 0 open deals)

4) Pipeline amount per resolved owner (sum of `amount` across that owner's open deals)

Owner (alias)          owner_id    deals   pipeline amount
Bryce Harmon           119337721    35     1,054,144.00     (45.6%)
Alex Franklin          84342457     67       624,310.00     (27.0%)
Dana Mercer            83155923     24       341,195.00     (14.7%)
Cole Ingram            83155924     22       288,161.43     (12.5%)
Farid Osman            716654662     7         4,134.00      (0.2%)
Elena Sinclair         701163055     1         2,100.00      (0.1%)

Arithmetic check (sum of the six owner totals = sum of all 156 deal amounts):
1,054,144.00 + 624,310.00 + 341,195.00 + 288,161.43 + 4,134.00 + 2,100.00 = 2,314,044.43
Deal count check: 35 + 67 + 24 + 22 + 7 + 1 = 156

Supporting detail where the result could look wrong otherwise:
- Farid Osman: 6 of his 7 deals are booked at 1 (Deal-03BA69, Deal-117863, Deal-F17780, Deal-2B39B0, Deal-8FDCD2) plus Deal-499BF6 at 1,249 and Deal-8BA24E at 2,880. Sum: 1+1+1+2,880+1+1 = 2,885; 2,885 + 1,249 = 4,134.
- Cole Ingram's total carries cents (288,161.43) from rows such as Deal-D04904 (58,529.25), Deal-175395 (4,779.88), Deal-2F3A66 (3,334.80), Deal-42326B (2,480.40), Deal-FF809F (7,781.20), Deal-A71728 (6,947.50), Deal-AF932D (7,225.40).
- Deal-57FF13 (Elena Sinclair) is a malformed row in the source: the n_contacts and source columns are empty and there is an extra trailing field. deal_id, deal_alias, owner_id, stage, forecast_category, amount and the date fields parse cleanly, so it is included at 2,100.
gong-hubspot-join-integrity0.75
Plain-English question: How many call records point at deal ids that do not exist in the open pipeline?
Prompt — what the model saw
Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.0152 · 138s · in 20,067 / out 19,274 tok
✓ orphan_count✗ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

Plain-English answer: 30 Gong call rows are orphans keyed to dead deal ids, none are duplicates, and DS3 call coverage is 29.4%.
The answer in one line: orphans = 30; ds3_cov_pct = 29.4; dup = 0
orphans30
ds3_cov_pct29.4
dup0
orphan_aliasesDeal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
graded checksorphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
INTEGRITY CHECK — call-to-deal mapping (67 gong rows; 156 open default-pipeline deals)

(a) Orphan gong rows (hs_deal_id with no match in deals_open)
Count: 30 of 67 rows = 44.8% (30 ÷ 67 = 0.4478)
Share of call volume affected: 304 of 465 logged calls = 65.4%
Sample aliases (all from the gong table):
- Deal-8FA85D (60251290957; 46 calls) — largest single block in the file
- Deal-8FC3F9 (60251649055; 24 calls)
- Deal-42B265 (61227242540; 21 calls)
- Deal-38CA53 (60785759793; 5 calls)
- Deal-3B6668 (60251130117; 4 calls)
Pattern worth noting: the 30 orphans are one contiguous block (rows 39–67 of the gong file), all with calls_with_ds2_date = 0 and calls_with_sqo_date = 0 except where the date columns are blank. That is consistent with a second id space (different portal/pipeline) rather than random record drift — worth confirming before treating them as lost attribution.

(b) Duplicate conversation keys (calls_90d > distinct_conversation_keys)
Count: 0 rows. Every one of the 67 rows has calls_90d exactly equal to distinct_conversation_keys (ratio 1.00 across the file), so no conversation-key duplication is detectable in this extract.

(c) Call coverage among open deals at DS3 or later
DS3+ population: 61 DS3 + 14 DS4 + 10 DS5 = 85 deals
With ≥1 logged call: 25 of 85 = 29.4% (25 ÷ 85 = 0.2941)
Without any logged call: 60 of 85 = 70.6%
The 25 covered deals: Deal-1E2498, Deal-215CCA, Deal-25F752, Deal-333EBB, Deal-3974EB, Deal-40522D, Deal-5408B0, Deal-547B2B, Deal-584EE5, Deal-5EED42, Deal-60C2C2, Deal-8952F0, Deal-944310, Deal-A2B47C, Deal-B7EBD1, Deal-BA3DDC, Deal-C26D20, Deal-C61CF7, Deal-C6D97A, Deal-C6FE92, Deal-D348E1, Deal-D9A12F, Deal-DAF1D9, Deal-E53952, Deal-F9A3C1

Caveats on the data as given:
- The join was done on hs_deal_id ↔ deal_id only. Deal aliases were used as a cross-check: no matched id had a conflicting alias, and no orphan's alias exists in deals_open (so orphans are not alias-level renames).
- gong_owner_id is a Gong user id, not a HubSpot owner_id; it cannot be joined to deals_open.owner_id, so no owner-level reconciliation is reported.
- 5 gong rows map to deals sitting at DS1 (e.g. Deal-03BA69 at 5 calls, Deal-523604 at 4 calls) — calls logged ahead of stage progression, included in the counts above but excluded from (c).
snowflake-sql-trapped-catalogPASS 1.00
Plain-English question: Write one Snowflake query for first-month usage per customer using only the provided, trap-laden catalog.
Prompt — what the model saw
Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0040 · 46s · in 19,522 / out 1,799 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

Plain-English answer: A single correct SQL query that uses the right catalog tables and avoids the trap columns — graded on the exact tables and columns.
The answer in one line: use_table = PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
avoid_stalePRODUCTION.HUBSPOT.DEALS (stale)
avoid_unpopulatedV2_LIVE.OBJECTS_DEALS (unpopulated)
forbidden_filterdeleted-giver exclusion must NOT be applied
outputone fenced SELECT with per-FROM/JOIN table-choice comment
required_filtersucceeded redemptions only
use_tablePRODUCTION.PLG.COMPANY_COHORT_SUMMARY
graded checksavoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- ============================================================================
-- DATA-GAP DISCLOSURE (read before running)
-- The catalog provided contains NO row-level recognition/giving event table
-- (no giver id + company id + event timestamp), and no documented redemption
-- table with company/timestamp columns. PRODUCTION.DEPRECATED_RECOGNITION.
-- REDEMPTION_RECORDS_V2 is the documented redemption source and STATE='succeeded'
-- is its filter, but its columns are not listed in this catalog, so it cannot be
-- joined to a company without inventing a key. The company identifier column in
-- COMPANY_COHORT_SUMMARY is likewise not documented (only its metric columns are).
-- Consequently "unique givers / recognitions / successful redemptions for the
-- first calendar month" can only be answered from the pre-aggregated M1 block,
-- not recomputed from raw events. Caveats are stated inline below.
-- NOT used, per catalog notes: HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS
-- (unpopulated), PRODUCTION.HUBSPOT.DEALS (stale, last sync 2023-03),
-- PRODUCTION.HUBSPOT.GONG_HUBSPOT_MAP_FAST (retired). No deal, engagement or
-- Gong table is needed for this question, so none is referenced.
-- DELETED-GIVER FILTER: no giver-level exclusion is applied anywhere in this
-- query. The historical giving figures must not be filtered on deleted givers,
-- and the source here is already an unfiltered historical aggregate.
-- ============================================================================
SELECT
    -- NOTE: the company key column is not documented in the catalog excerpt.
    -- Substitute the real identifier (company id / name) in place of <company_key>.
    c.<company_key>                                   AS customer_company,
    DATE_TRUNC('month', c.SIGNUP_DATE)                AS first_customer_month,
    -- M1_USERS is "users" in month 1, NOT verified unique givers. The catalog
    -- does not document a unique-giver count, so this is the closest available
    -- field and should be labeled as users until confirmed.
    c.M1_USERS                                        AS m1_users_not_unique_givers,
    c.M1_GIVES                                        AS m1_recognition_count,
    -- M1_REDEMPTIONS: the catalog does not state whether this is already
    -- restricted to STATE = 'succeeded'; it cannot be re-filtered here because
    -- no raw redemption rows are joined.
    c.M1_REDEMPTIONS                                  AS m1_successful_redemption_count_unverified
-- FROM: PRODUCTION.PLG.COMPANY_COHORT_SUMMARY is the only documented
-- company-level table, and its M1_* columns are the only cataloged first-month
-- giving/redemption metrics. It covers self-serve companies only, so enterprise
-- customers are out of scope for this query.
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY AS c
WHERE c.SIGNUP_DATE IS NOT NULL
ORDER BY first_customer_month, customer_company
```
crm-completeness-enrichmentPASS 1.00
Plain-English question: Audit the CRM for missing and invalid fields, and propose the fixes.
Prompt — what the model saw
Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0076 · 81s · in 5,004 / out 11,031 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

Plain-English answer: 9 companies missing employee count, 6 missing HQ country, 4 invalid emails, 1 domain mismatch, 2 duplicate clusters, and 25 enrichment matches.
The answer in one line: n_contacts = 52; n_companies = 34; enrichment_matches = 25; invalid_email_count = 4; n_dup_clusters = 2; domain_mismatch_count = 1 (+4 more below)
n_contacts52
domain_mismatch_count1
enrichment_matches25
invalid_email_count4
n_companies34
n_dup_clusters2
company_missingindustry=0; employee_count=9; hq_country=6
domain_mismatchesCT-0011
duplicate_clusters['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
invalid_emailsCT-0010, CT-0080, CT-0081, CT-0192
graded checksdup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
DEALS: NOT ASSESSABLE — no deals file (or deal rows/fields) was included in the extract. Owner, stage, amount, close date, and why-buy completeness are all unknown, not 0%. Every pipeline-dollar statement below is therefore blocked; see Section 8.

## 1. Completeness per field (arithmetic shown)

Companies — 34 rows, 32 unique domains
- industry: 34/34 = 100.0%
- employee_count: 25/34 = 73.5% (9 blank)
- hq_country: 28/34 = 82.4% (6 blank)

Contacts — 52 rows
- email present: 52/52 = 100.0%
- email valid format: 48/52 = 92.3% (4 malformed)
- email valid AND domain-matches company: 47/52 = 90.4%
- title: 39/52 = 75.0% (13 blank)
- persona: 37/52 = 71.2% (15 blank)

Coverage
- Companies with ≥1 contact: 20/32 unique domains = 62.5%
- Zero-contact domains: 12 (C-2C60E5, C-B97B4E, C-2D7423, C-7BBDFA, C-50D386, C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20, acme-corp.com, globex.io)
- Enrichment coverage: 25/32 domains = 78.1%; 7 domains have no ZoomInfo row (ba969b.com, 332637.com, 93c8bf.com, ee9ffb.com, c9bb20.com, acme-corp.com, globex.io)

Deals — 0 rows: owner 0/0, stage 0/0, amount 0/0, close date 0/0, why-buy 0/0. Not computable.

## 2. Duplicate company clusters

Clustering was done on domain only — the extract has no company-name field (aliases only), so name-variant clustering beyond domain is not possible with this data.

Cluster 1 — acme-corp.com (2 records, identical domain and alias stem)
- C-0A092931: Technology, 500, US
- C-0A092932: tech, 510, USA
- Survivor: C-0A092931 (earliest record, canonical industry casing, ISO country code). Absorb C-0A092932.
- Conflict: employee_count 500 vs 510, 2.0% apart, no enrichment row to arbitrate → keep 500, log as unverified.
- Safe now: neither row has contacts or an enrichment row, so no re-parenting risk.

Cluster 2 — globex.io (2 records)
- C-0A092933: SaaS, 200, US
- C-0A092934: Technology, 200, US
- Survivor: C-0A092933 (retains the more specific industry "SaaS"; employees and country agree).
- No contacts, no enrichment row; merge is clean.

## 3. Invalid emails (4) and domain mismatches (1)

Invalid — missing domain part entirely:
- CT-0010 (C-66D1FC): "user0@"
- CT-0080 (C-92D97D): "user0@"
- CT-0081 (C-92D97D): "user1@"
- CT-0192 (C-425E2A): "user2@"
Fix: reconstruct to userN@<company domain> only with verification; otherwise quarantine. Do not auto-append — that would be inference, not data.

Domain mismatch:
- CT-0011 (C-66D1FC): email user1@other-domain.com, contact domain 66d1fc.com. Syntactically valid, wrong company domain. Recommend re-key to 66d1fc.com only if confirmed; otherwise mark as external/personal-domain contact and exclude from account-based sends.
- Note: the contacts.domain column matches the company domain on all 52 rows; the defect is in the email string, not the field.

## 4. Missing company fields filled from enrichment (8 changes, all employee_count)

Rule applied: fill only where CRM blank AND the enrichment row has a value.

| Alias | Domain | Field | CRM | ZI | Action |
|---|---|---|---|---|---|
| C-EC3025 | ec3025.com | employee_count | blank | 400 | fill 400 |
| C-96039F | 96039f.com | employee_count | blank | 400 | fill 400 |
| C-44EA29 | 44ea29.com | employee_count | blank | 400 | fill 400 |
| C-D04904 | d04904.com | employee_count | blank | 400 | fill 400 |
| C-B23205 | b23205.com | employee_count | blank | 400 | fill 400 |
| C-60C75F | 60c75f.com | employee_count | blank | 400 | fill 400 |
| C-7BBDFA | 7bbdfa.com | employee_count | blank | 400 | fill 400 |
| C-50D386 | 50d386.com | employee_count | blank | 400 | fill 400 |

Risk flag: all eight values are exactly 400 across unrelated companies, industries, and countries. That pattern is more consistent with a size band than a headcount. Accept with source=ZoomInfo and a verification date, and do not use these eight for tiering math until re-checked.

Post-fill: employee_count 33/34 = 97.1% (only C-93C8BF remains blank, no enrichment row).

No hq_country fills were possible: the 6 blanks all sit on rows where enrichment country is also blank (C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5) or absent (C-EE9FFB).

## 5. CRM vs enrichment disagreements — both values, recommended source

Industry (10 disagreements, all the same taxonomy collision):
- C-66D1FC "tech" vs "Computer Software"
- C-EC3025 "Technology" vs "Computer Software"
- C-44EA29 "tech" vs "Computer Software"
- C-92D97D "Technology" vs "Computer Software"
- C-D04904 "Technology" vs "Computer Software"
- C-77A95A "Technology" vs "Computer Software"
- C-AA8DDA "Technology" vs "Computer Software"
- C-B25F40 "Technology" vs "Computer Software"
- C-60C75F "tech" vs "Computer Software"
- C-425E2A "Tech " vs "Computer Software"

Recommendation: ZoomInfo as the stored industry value (it is a standardized taxonomy), CRM label preserved in a legacy free-text field. Effect: "Technology/tech/Tech " collapses to Computer Software on those 10 rows.

Employee count: no CRM-vs-enrichment disagreement where both have values. The only numeric conflict is intra-CRM (acme 500 vs 510).
HQ country: no true disagreement — all overlaps are US/USA vs United States. This is a normalization, not a conflict: 15 rows use US or USA and should be canonicalized to ISO "US" (or "United States" everywhere, one convention, not both).
Unresolvable by enrichment: globex.io "SaaS" vs "Technology" (no ZI row). Keep "SaaS" (more specific), mark source=CRM.

Also non-canonical, not a conflict: 10 industry labels ("tech" ×4, "Tech " ×4 with trailing space, "health care" ×2) and 4 country spellings (US ×9, USA ×6, United States ×2). "Health care" vs "Healthcare" also needs one form.

## 6. Gaps that cannot be filled from provided data

- hq_country: C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB
- employee_count: C-93C8BF
- No enrichment row at all: ba969b.com, 332637.com, 93c8bf.com, ee9ffb.com, c9bb20.com (C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20)
- Contacts: 13 missing title, 15 missing persona (concentrated on C-92D97D, C-AA8DDA, C-B25F40, C-60C75F)

## 7. Exposure proxy (declared)

Because no deal amounts were supplied, I rank by employee_count of the affected account as a stand-in for size. This is a proxy, not pipeline dollars.

Largest accounts with zero contacts: C-B97B4E (1,500), C-EE9FFB (1,500), C-332637 (1,500), C-2D7423 (900), C-C9BB20 (900), C-7BBDFA (400 est.), C-50D386 (400 est.), C-2C60E5 (340).

## 8. Top 10 fixes, ranked

AMOUNT AT STAKE: not computable for any of these — the extract contains no deals, no amounts, no owners, no stages, no close dates. Supplying deals.csv is a prerequisite for a dollar-ranked list. Ordering below is by record exposure (employee_count proxy + record count), not dollars.

1. Supply the deals extract. Until then owner/stage/amount/close date/why-buy are 0% assessable and no fix can be dollar-ranked.
2. Merge acme-corp.com — C-0A092931 survives, absorb C-0A092932; hold 500 vs 510 as a logged conflict.
3. Merge globex.io — C-0A092933 survives, absorb C-0A092934; keep SaaS.
4. Build contacts on the 12 zero-contact domains, starting with C-B97B4E, C-EE9FFB, C-332637 (1,500 each).
5. Repair or quarantine 4 malformed emails: CT-0010, CT-0080, CT-0081, CT-0192.
6. Resolve CT-0011 domain mismatch (user1@other-domain.com vs 66d1fc.com).
7. Apply the 8 ZoomInfo employee_count fills, source-tagged, with the uniform-400 flag noted.
8. Canonicalize hq_country on 15 US/USA rows to one convention; leave the 6 blanks explicitly null, not guessed.
9. Fill title/persona on 13 and 15 contacts respectively, prioritizing C-92D97D and C-B25F40 where both are missing on most rows.
10. Normalize industry to ZoomInfo taxonomy (10 rows) and chase enrichment coverage for the 5 accounts with no ZoomInfo row (C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20).
deal-intelligence 5 tests
closed-lost-classificationPASS 1.00
Plain-English question: Why did we lose the last six months of deals — one primary category per deal, and whose side is the reason on?
Prompt — what the model saw
Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0118 · 112s · in 8,325 / out 16,977 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

Plain-English answer: Every closed-lost deal gets exactly one category (pricing, competitor, no decision, timing, product gap, champion left, other) plus a side, per the computed classification.
The answer in one line: total_lost_6mo = 90; top_reason = competitor (26)
total_lost_6mo90
categoriespricing, competitor, no decision, timing, product gap, champion left, other
patternstwo most actionable, tag-vs-text disagreements counted
sideBonusly / buyer / unknown per deal
top_reasoncompetitor (26)
graded checkstotal_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Scope check first: the file is labeled 6 months but the close dates run 2026-07-29 → 2026-09-30 (63 days), 90 deals, $1,267,945.16 total. I classified what's there; nothing outside that window exists in the data.

Rules I applied (stated so you can audit):
- MIA/unresponsive → no decision (buyer never decided), not "other".
- "Doing nothing / not a priority" → no decision; → pricing only when the text cites cheapness/simpler-and-cheaper.
- Side = who the text says drove the outcome: Bonusly = our price/product/execution cited; buyer = their budget, timing, priorities, incumbent contract, procurement, or silence; unknown = no attributable cause given.
- Tie-break on multi-reason text = first-named reason.
- "Disagreement" = the free text names a different primary cause than the tag implies (vague text that merely fails to support a tag is not counted).

PER-DEAL CLASSIFICATION (alias — side)

no decision (32)
Deal-AC944F buyer · Deal-214060 buyer · Deal-13E9CF buyer · Deal-21B045 buyer · Deal-988493 buyer · Deal-F308CA buyer · Deal-70F704 buyer · Deal-4664E1 buyer · Deal-E74A73 buyer · Deal-D48E0B buyer · Deal-583ADB buyer · Deal-8E27DA buyer · Deal-E0441F buyer · Deal-7CB44D buyer · Deal-FAC17C buyer · Deal-50E5D8 buyer · Deal-AFA56C buyer · Deal-413C56 buyer · Deal-2A292B buyer · Deal-D1AABF buyer · Deal-2BBA21 buyer · Deal-7FBAC6 buyer · Deal-386F6E buyer · Deal-2FEDDB buyer · Deal-3F86A0 buyer · Deal-096750 buyer · Deal-ABD14C buyer · Deal-79E61A buyer · Deal-AE7C4E buyer · Deal-DAB4F1 buyer · Deal-B4B50F buyer · Deal-5885B9 buyer

competitor (24)
Deal-F7F635 unknown · Deal-F97C37 Bonusly · Deal-422BA6 buyer · Deal-381C8C unknown · Deal-F1E8A6 unknown · Deal-DDAB52 Bonusly · Deal-ACE061 unknown · Deal-2D2F8D unknown · Deal-0F96AA unknown · Deal-7CC678 unknown · Deal-242273 Bonusly · Deal-A2C349 Bonusly · Deal-C7156E unknown · Deal-8A0992 buyer · Deal-D0C698 buyer · Deal-EECC02 unknown · Deal-47F1A1 buyer · Deal-BF2A98 buyer · Deal-1E7DA9 unknown · Deal-286F9C Bonusly · Deal-369281 buyer · Deal-9FCD0D buyer · Deal-64B19A buyer · Deal-DC77FE Bonusly

timing (20)
Deal-DB0AAC buyer · Deal-91A056 buyer · Deal-29326C buyer · Deal-831B7B buyer · Deal-39E25C buyer · Deal-B3ABED buyer · Deal-ED9AE7 buyer · Deal-B6AC09 buyer · Deal-E6E80A buyer · Deal-B038F0 buyer · Deal-175756 buyer · Deal-BB78F3 buyer · Deal-15DA99 buyer · Deal-F4AF5D buyer · Deal-79B7A1 buyer · Deal-9F176A buyer · Deal-69CF3D buyer · Deal-ECBF89 buyer · Deal-D1A623 buyer · Deal-FEDBCB buyer

pricing (8)
Deal-7ED004 buyer · Deal-1BCA50 buyer · Deal-7B2236 buyer · Deal-C33D91 buyer · Deal-5E64CE buyer · Deal-5AD03E buyer · Deal-DAFB82 buyer · Deal-8A119B buyer

product gap (3)
Deal-9048EB Bonusly · Deal-3618CC Bonusly · Deal-981AD4 Bonusly

champion left (1)
Deal-F325A5 buyer

other (2)
Deal-5DB9B0 unknown · Deal-55867E unknown

CATEGORY COUNTS
no decision 32 + competitor 24 + timing 20 + pricing 8 + product gap 3 + champion left 1 + other 2 = 90

Composition of the 32 no-decision: 21 from MIA tags + 8 from "Doing nothing/Not a priority/Cost" + 2 from "Lost DM" + 1 from "Feature Request" = 32.
Timing is 18 "Lost-Timing" tags + 1 "Doing nothing" (Deal-FEDBCB) + 1 "Lost DM" (Deal-ED9AE7) = 20.
Competitor is 26 "Competitor" tags − 2 recoded to pricing (Deal-1BCA50, Deal-5AD03E) = 24.

Dollar weight (context, not asked): no decision $302,264.20 · competitor $367,234.96 · timing $256,991.00 · pricing $214,810.00 · product gap $94,245.00 · other $18,000.00 · champion left $14,400.00 = $1,267,945.16.

SIDE SPLIT
buyer 69 · unknown 12 · Bonusly 9 → 69 + 12 + 9 = 90 (76.7% / 13.3% / 10.0%).
The 9 Bonusly-side: Deal-F97C37, Deal-DDAB52, Deal-242273, Deal-A2C349, Deal-286F9C, Deal-DC77FE, Deal-9048EB, Deal-3618CC, Deal-981AD4.

TAG vs FREE-TEXT DISAGREEMENTS: 5 of 90
1. Deal-8E27DA — tag Feature Request; text: "moved forward with just a swag provider and didn't want R&R, currently" → no decision (not a feature request).
2. Deal-3618CC — tag Lost DM; text: "Wanted Surveys" → product gap (no champion departure mentioned).
3. Deal-5AD03E — tag Competitor; text: "Wanted more defined budget access" → pricing (no competitor named anywhere).
4. Deal-1BCA50 — tag Competitor; text: "mostly about the budget and details regarding gift cards… other stakeholder was already way down the path with another vendor" → pricing (text names budget as the main factor).
5. Deal-9048EB — tag MIA; text: "…bad fit based on their desired setup and multiple feature gaps" → product gap (text cites fit/features, not only silence).

Soft mismatches I did not count (no stated cause in the text, so nothing to contradict): Deal-55867E (tag Timing, text "I don't think we'll be moving forward with Bonusly at this time"), plus Deal-7B2236 / Deal-5E64CE / Deal-FEDBCB where the tags are catch-alls that absorb the text.

TWO PATTERNS WORTH ACTING ON

1. Most of the loss volume was never a loss — it was a deferral or a silence, and it is sitting unattended.
52 of 90 deals (57.8%) ended in timing or no decision: 20 + 32 = 52. That is $559,255.20 of $1,267,945.16, or 44.1%. Inside it, 22 deals / $254,142.00 are pure MIA (tag-level), and most timing texts name a specific future date ("early 2027", "Q2 next year", "new year", "reconnect early in 2027") — Deal-91A056, Deal-B6AC09, Deal-E6E80A, Deal-B038F0, Deal-175756, Deal-15DA99, Deal-39E25C, Deal-831B7B, Deal-F4AF5D, Deal-B3ABED. These are date-stamped, so they are follow-up work, not lost demand. Every one of the 20 timing deals is buyer-side; zero are Bonusly-side. Treat the 2027 cohort as a nurture list with owner and reconnect month, and separately treat the 22 MIA deals as an outreach-quality problem (Deal-F308CA and Deal-7CB44D both died on "ignored multiple pieces of outreach from me and the ADR").

2. Competitive losses are mostly unmeasured, and where they are measured the gap is surveys and points flexibility.
24 competitor deals = $367,234.96 (29.0%). Only 6 of the 24 name a reason attributable to our offer; 10 of 24 give no reason at all — "went in another direction" / "not going to be moving forward" / "Nothing specific provided" (Deal-F7F635, Deal-381C8C, Deal-F1E8A6, Deal-ACE061, Deal-2D2F8D, Deal-0F96AA, Deal-7CC678, Deal-C7156E, Deal-EECC02, Deal-1E7DA9). You cannot fix a competitive narrative you did not record. The 8 remaining are buyer-circumstance losses that no product work would recover: ADP PEO partnership (Deal-422BA6), Canadian provider for CEO alignment (Deal-8A0992, Deal-9FCD0D), incumbent renewals (Deal-47F1A1 WorkTango, Deal-BF2A98 HiThrive, Deal-369281 Paylocity, Deal-64B19A Motivosity, Deal-D0C698 Kudos). The named product signals cluster twice: surveys/surveying (Deal-A2C349 — Awardco plus surveying; Deal-3618CC — "Wanted Surveys") and points flexibility/customization (Deal-DC77FE — "label points as dollars", price explicitly not the factor; Deal-242273 — onsite redemption of points; Deal-F97C37 — diversified offerings; Deal-DDAB52 — Rippl "a lot more at the same cost").

One caveat on the side split: "buyer" at 69 of 90 is real but partly an artifact of MIA being counted buyer-side. If you'd rather MIA be its own "no signal" bucket, buyer drops to 47 and no-signal rises to 34. The category counts do not change.
pipeline-tieringPASS 1.00
Plain-English question: Score all 156 open deals into exactly one of LOCK / ACTION / BUILD / REVIVE / WATCH / RISKY.
Prompt — what the model saw
Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.0343 · 139s · in 49,758 / out 39,829 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

Plain-English answer: All 156 open deals tiered exactly once across the six tiers using forecast category, stage, meetings, contacts, and recency.
The answer in one line: total = 156
total156
graded checksjson_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{
 "tier_counts": {"LOCK": 5, "ACTION": 24, "BUILD": 34, "REVIVE": 6, "WATCH": 65, "RISKY": 22},
 "tier_examples": {
  "LOCK": ["Deal-25F752", "Deal-C26D20", "Deal-944310"],
  "ACTION": ["Deal-5408B0", "Deal-D348E1", "Deal-B7EBD1"],
  "BUILD": ["Deal-D73B89", "Deal-523604", "Deal-CA7DC0"],
  "REVIVE": ["Deal-2D1F1B", "Deal-012CB1", "Deal-F0EBBB"],
  "WATCH": ["Deal-6787C2", "Deal-66D1FC", "Deal-950043"],
  "RISKY": ["Deal-E53952", "Deal-9AAE5F", "Deal-547B2B"]
 },
 "risky_deals": ["Deal-E53952", "Deal-9AAE5F", "Deal-547B2B", "Deal-A2B47C", "Deal-2465CE", "Deal-62D607", "Deal-F9A08A", "Deal-0660B4", "Deal-FD9F4E", "Deal-BA571A", "Deal-FC22A3", "Deal-7BBDFA", "Deal-4A13AD", "Deal-8AD4A5", "Deal-15D24F", "Deal-9D0060", "Deal-635B8E", "Deal-ED725A", "Deal-3BA5EA", "Deal-5FDCE4", "Deal-F336B6", "Deal-7599B8"],
 "lock_violations": 0,
 "pipeline_shape": "156 open deals, $2.31M total; 105 PIPELINE, 40 BEST_CASE, 11 COMMIT ($485,655, 21% of open dollars). Stage mass sits mid-funnel: DS1 32, DS2 39, DS3 61, DS4 14, DS5 10. Engagement is thin against that shape: 101 of 156 deals (65%) have zero meetings_30d and only 53 logged any meeting at all, so email volume (recent for most rows) is doing the work of proof. The 51 COMMIT/BEST_CASE deals split badly: 5 clear LOCK quality (2+ meetings, 3+ contacts, touched within 14 days) versus 22 with no meeting and no call in 30 days despite the high forecast, i.e. $150,470 of the $485,655 high-confidence pool is unsupported by live conversation. Two deals, Deal-3EED2C and Deal-57FF13, have no engagement rows at all and are tiered REVIVE on absence of evidence rather than measured staleness; several rows also carry last_meeting dates after the 2026-09-04 as-of date, so recency there is read as recent and not inferred further."
}

⚠️ File-mutation verifier: 1 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv`: `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` exists but this task has not seen its full c…
call-transcript-extractionPASS 1.00
Plain-English question: Extract the CRM write-back fields from each call transcript as JSON.
Prompt — what the model saw
For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0058 · 35s · in 21,662 / out 4,197 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

Plain-English answer: Per transcript: prospect-only why-buys, pain points, stakeholders, budget and timeline signals, and competitors — e.g. TX-001: '$40k earmarked', live before open enrollment.
The answer in one line: rep = Alex Franklin
allowed_newTX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
per_transcript{"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
repAlex Franklin
rep_sourced_nulls{"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
graded checksjson_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "Prospect (VP People): automating anniversary and birthday awards would be the big win — HR team of three cannot keep up manually"
    ],
    "pain_points": [
      "Prospect (VP People): HR team of three cannot keep up with manual anniversary/birthday awards",
      "Prospect (HR Admin): everything tracked in a spreadsheet; people slip through the cracks"
    ],
    "stakeholders": {
      "on_call": ["Prospect (VP People)", "Prospect (HR Admin)"],
      "mentioned_by_prospect": ["IT lead (security sign-off required; not on call)"]
    },
    "budget_signal": {"stated": true, "amount": "$40k earmarked for engagement tools this fiscal year", "by": "Prospect (VP People)"},
    "timeline_signal": "Live before open enrollment in November (Prospect (VP People): \"Ideally we would have this live before open enrollment in November\")",
    "competitor_mentioned": ["Achievers (raised by Prospect (VP People): \"We looked at Achievers last year, but it was too heavy for a team our size\")"],
    "next_step": "Security review with prospect IT lead on September 12 — prospect agreed: \"Yes — let's do the security review on September 12.\"",
    "objections": [
      "Prospect (HR Admin): needs SSO and audit logs for IT sign-off"
    ],
    "confidence": {"level": "high", "basis": "why-buy, pain, budget, timeline, competitor, objection and next step are all prospect-stated verbatim; only the IT lead's identity is unnamed"},
    "arithmetic_notes": "No employee headcount provided, so $40k cannot be converted to a per-employee or per-seat figure. No price quoted by rep in this transcript, so budget fit is not computable."
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "Prospect (Head of Total Rewards): want to tie recognition to retention for hourly workforce where regretted turnover is over 30%"
    ],
    "pain_points": [
      "Prospect (Head of Total Rewards): regretted turnover in hourly workforce over 30%",
      "Prospect (CFO): Workday integration must be rock solid — stated as \"my one condition\""
    ],
    "stakeholders": {
      "on_call": ["Prospect (Head of Total Rewards)", "Prospect (CFO)"],
      "mentioned_by_prospect": ["Legal (pilot agreement routed to legal; not on call)"]
    },
    "budget_signal": {"stated": true, "amount": "$25k pilot budget approved for this quarter", "by": "Prospect (CFO): \"Finance has approved a $25k pilot budget for this quarter.\""},
    "timeline_signal": "Decision wanted by end of September (Prospect (CFO)); pilot agreement to legal this week (Prospect (CFO))",
    "competitor_mentioned": [],
    "next_step": "Send pilot agreement; prospect will route to legal this week — prospect agreed: \"Yes — send the pilot agreement and we'll route it to legal this week.\"",
    "objections": [
      "Prospect (CFO): Workday integration must be rock solid — \"that's my one condition\""
    ],
    "confidence": {"level": "high", "basis": "budget, timeline, next step and condition are all explicit prospect statements; no competing vendor raised"},
    "arithmetic_notes": "Turnover stated as \">30%\" without headcount or cost-per-hire, so no savings or ROI figure is computable. $25k pilot vs. no quoted price: coverage not determinable."
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "Prospect (People Ops Manager): need recognition visible across 12 retail locations",
      "Prospect (People Ops Manager): CEO used Bucketlist at her last company and liked it (prospect-stated buying context)"
    ],
    "pain_points": [
      "Prospect (People Ops Manager): recognition not visible across 12 retail locations",
      "Prospect (People Ops Manager): store managers have zero budget autonomy for on-the-spot recognition today"
    ],
    "stakeholders": {
      "on_call": ["Prospect (People Ops Manager)"],
      "mentioned_by_prospect": ["CEO (decides anything people-related; not on call)"]
    },
    "budget_signal": null,
    "timeline_signal": "No rush on prospect side until Q1 (Prospect (People Ops Manager): \"Honestly there's no rush on our side until Q1.\")",
    "competitor_mentioned": ["Bucketlist (raised by Prospect (People Ops Manager): CEO used it at her last company and liked it)"],
    "next_step": "Call with prospect CEO — agreed: \"Yes, let's schedule a call with our CEO — I'll send two times.\"",
    "objections": [
      "Prospect (People Ops Manager): no urgency until Q1",
      "Prospect (People Ops Manager): CEO has to be sold first — she decides anything people-related"
    ],
    "confidence": {"level": "medium", "basis": "need, timeline, competitor, blocker and next step are prospect-stated; no budget figure exists (rep's $8/employee-per-month quote is not a prospect signal and was excluded)"},
    "arithmetic_notes": "Rep quoted \"around $8 per employee per month\" but no employee count was stated by the prospect, so total annual cost is not computable. 12 locations is the only count given."
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "Prospect (VP People): consolidate three separate recognition tools into one"
    ],
    "pain_points": [
      "Prospect (VP People): paying for three tools and none of them talk to the HRIS"
    ],
    "stakeholders": {
      "on_call": ["Prospect (VP People)", "Prospect (IT Security Lead)"],
      "mentioned_by_prospect": ["CFO (follow-up requested by rep; prospect said maybe — calendar check, no commitment)", "Procurement (cycle referenced by IT Security Lead; not on call)"]
    },
    "budget_signal": {"stated": true, "amount": "no amount; approval authority threshold under $15k annually without board approval", "by": "Prospect (VP People): \"If it's under $15k annually, I can approve it without going to the board.\""},
    "timeline_signal": "Procurement cycle six to eight weeks minimum (Prospect (IT Security Lead)); prior vendor security review took three months",
    "competitor_mentioned": [],
    "next_step": null,
    "objections": [
      "Prospect (IT Security Lead): prior security review took three months — stated as \"my hesitation\"",
      "Prospect (IT Security Lead): procurement cycle runs six to eight weeks minimum",
      "Prospect (VP People): follow-up with CFO not committed — \"Maybe — I need to check her calendar, no promises.\""
    ],
    "confidence": {"level": "medium", "basis": "why-buy, pain, budget authority and timeline are prospect-stated; no next step agreed, so none recorded. Rep's \"I'll follow up\" is not a prospect commitment"},
    "arithmetic_notes": "$15k annual threshold stated, but no price or employee count given, so whether the deal lands under the approval threshold is not computable. Procurement floor of 6 weeks = ~1.5 months; stated prior review 3 months — both prospect-stated, not added together because they may overlap or be sequential (transcript does not say)."
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "Prospect (HR Director): automate service milestones (stated as required: \"Two things\")",
      "Prospect (HR Director): analytics on recognition equity across departments (stated as required: \"Two things\")"
    ],
    "pain_points": [
      "Prospect (People Ops Coordinator): night-shift teams feel invisible; their engagement scores run 20 points lower",
      "Prospect (HR Director): exec team skeptical after a failed rollout two years ago"
    ],
    "stakeholders": {
      "on_call": ["Prospect (HR Director)", "Prospect (People Ops Coordinator)"],
      "mentioned_by_prospect": ["Exec team (skeptical after prior failed rollout; presentation agreed)"]
    },
    "budget_signal": {"stated": true, "amount": "$12k approved under the engagement line", "by": "Prospect (HR Director)"},
    "timeline_signal": "Running before January all-hands (Prospect (HR Director))",
    "competitor_mentioned": ["Nectar (raised by Prospect (HR Director): \"We're mid-pilot with Nectar right now, so you'd need to beat that experience.\")"],
    "next_step": "Present to prospect exec team on October 2 — prospect agreed: \"Yes — come present to our exec team on October 2.\"",
    "objections": [
      "Prospect (HR Director): mid-pilot with Nectar — incumbent experience must be beaten",
      "Prospect (HR Director): exec team skeptical after a failed rollout two years ago"
    ],
    "confidence": {"level": "high", "basis": "both required capabilities, budget figure, deadline, live competitor, objection and dated next step are prospect-stated"},
    "arithmetic_notes": "Engagement gap stated as \"20 points lower\" for night-shift teams — no scale, baseline or headcount given, so no impact figure is computable. $12k vs. no quoted price: coverage not determinable."
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "Prospect (HR Manager): wants to cut admin time on service awards"
    ],
    "pain_points": [
      "Prospect (HR Manager): personally spends five hours a month ordering and shipping plaques"
    ],
    "stakeholders": {
      "on_call": ["Prospect (HR Manager)"],
      "mentioned_by_prospect": ["COO (usually prefers building things in-house; not on call)"]
    },
    "budget_signal": {"stated": true, "amount": null, "by": "Prospect (HR Manager): \"Budget isn't the issue — time is.\" (no amount stated)"},
    "timeline_signal": "Fiscal year resets in February; Q1 start is realistic (Prospect (HR Manager))",
    "competitor_mentioned": [],
    "next_step": "Send one-page overview; prospect will forward to COO this week — agreed: \"Yes — send the one-page overview and I'll forward it to our COO this week.\"",
    "objections": [
      "Prospect (HR Manager): COO usually prefers building things in-house"
    ],
    "confidence": {"level": "medium", "basis": "need, quantified pain, timeline, objection and next step are prospect-stated; budget is only a directional statement with no amount. Rep's pricing-flex offer was excluded as a rep statement"},
    "arithmetic_notes": "5 hours/month × 12 months = 60 hours/year of the HR Manager's own time on plaque ordering/shipping. No loaded hourly cost given, so no dollar figure computed. Alternative in this deal is internal build (prospect-stated), no named vendor."
  }
]
```

Missing data, stated explicitly:
- Budget amount: absent for Deal-530B50 (only a rep quote, excluded) and Deal-EE195F (direction only, no number).
- Headcount: not stated in any transcript, so no per-employee or total annual cost is computable for any deal.
- Competitors: no named vendor raised by the prospect in Deal-70BB30, Deal-180D02, Deal-EE195F.
- Next step: none agreed in Deal-180D02 — rep's "I'll follow up" and the prospect's "Maybe… no promises" do not qualify.
- Stakeholders named by prospects but not on the call (IT lead, legal, CEO, CFO, COO, exec team) are listed separately and not attributed as call participants.
next-to-close0.75
Plain-English question: Which 3 deals are closest to signature right now?
Prompt — what the model saw
Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0038 · 24s · in 21,449 / out 992 tok
✗ exactly_three✓ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

Plain-English answer: Deal-547B2B ($11,200), Deal-B7EBD1 ($9,000), and Deal-403845 ($9,000) — all DS5 COMMIT deals closing September 10-11, with what is left on each.
blocked_excludeddeal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
exactly_three1
ranking_ruleSlack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
top3rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
graded checksexactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
**1. Deal-547B2B — $11,200 (Alex Franklin, DS5/COMMIT, close 2026-09-11)**
Why close: Slack 09/04 09:12 — redlines back clean, signing page sent, VP People says they sign tomorrow.
Left: signature only.

**2. Deal-403845 — $9,000 (CRM owner Alex Franklin; Slack attributes it to Dana Mercer — data conflict, flagging rather than resolving)**
Why close: DS5/COMMIT, close 2026-09-11; Slack 09/04 10:02 — order form is with their finance team. Order form at finance is post-legal, pre-signature.
Left: finance approval and countersignature.

**3. Deal-A2B47C — $6,360 (Alex Franklin, DS5/COMMIT, close 2026-09-11)**
Why close: DS5/COMMIT, same 09/11 cohort; Slack 09/04 09:44 — "still warm, just normal legal-review pace."
Left: remaining legal review, then signature.

Arithmetic: 11,200 + 9,000 + 6,360 = $26,560 combined.

Excluded: Deal-2465CE ($5,400, COMMIT) is blocked — champion left, procurement freeze, Dana pulled it from commit, now Q4. Deal-D348E1 ($13,770, COMMIT) is warm but closes 2026-10-15, normal legal pace.

Note: Deal-B7EBD1 ($9,000, Dana, DS5/COMMIT) closes earliest (2026-09-10) but has no Slack update, so I did not rank it above a deal with a confirmed late-stage signal.
product-gap-detectionPASS 1.00
Plain-English question: Which prospect complaints are true product gaps versus rollout or enablement issues?
Prompt — what the model saw
Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0035 · 24s · in 19,120 / out 1,047 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

Plain-English answer: Two real gaps (SCIM provisioning, ADP Workforce Now integration) separated from plan gates and enablement issues, each with the prospect quote and deal.
allowed_newTG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
enablement_issuestranscript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voicetranscript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
plan_gatestranscript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
real_gapstranscript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
graded checksreal_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
PRODUCT GAP ANALYSIS — 5 transcript candidates

Amounts: the provided data contains no deal amounts or ARR figures for any deal. I cannot report a dollar value for Deal-EC3025, Deal-D0D6B5, Deal-CFE7F4, Deal-84DBA6, or Deal-36C33F. Missing data, not zero.

1) TG-001 — Deal-EC3025 — REAL GAP
Prospect (IT Security Lead): "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
Classification: real gap. Docs list "SSO / SAML and audit logs" as Enterprise-only but state plainly: "SCIM user provisioning and ADP Workforce Now integrations are NOT currently listed as supported capabilities." Not a tier limitation — no tier offers it.
Amount: not provided in data.

2) TG-002 — Deal-D0D6B5 — REAL GAP
Prospect (HRIS Manager): "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
Classification: real gap. Supported HRIS list is Workday, BambooHR, Gusto, Rippling (Pro and above). ADP Workforce Now is explicitly NOT listed as supported at any tier.
Amount: not provided in data.

3) TG-003 — Deal-CFE7F4 — PLAN GATE (not a gap)
Prospect (VP People): "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?"
Classification: plan gate. Docs: "Custom report builder" is Enterprise-only (— on Core and Pro). The capability exists; the prospect is below the tier. Alex Franklin's reply ("sits on our Enterprise tier") is consistent but is rep language and was not used as evidence.
Amount: not provided in data.

4) TG-004 — Deal-84DBA6 — ROLLOUT/ENABLEMENT ISSUE (not a gap)
Prospect (People Ops Manager): "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it."
Classification: rollout/enablement. Docs: "Slack and Microsoft Teams integration" is available on all plans, so the product capability is present and already switched on. The failure is adoption/training, not missing functionality.
Amount: not provided in data.

5) TG-005 — Deal-36C33F — NOT ADMISSIBLE (rep voice only)
The only product-gap statement is from Alex Franklin: "Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap." That is rep language, not prospect voice, so it does not qualify. The prospect's own line is not a gap: "Good to know. The web version should be fine for our office staff for now."
Amount: not provided in data.

SUMMARY — REAL PRODUCT GAPS ONLY (2)

1. SCIM user provisioning — Deal-EC3025
   Prospect: "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
   Docs: SCIM not listed as supported at any tier. Security-approval blocker.

2. ADP Workforce Now integration — Deal-D0D6B5
   Prospect: "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
   Docs: HRIS support limited to Workday, BambooHR, Gusto, Rippling (Pro+). ADP not supported. Stated dealbreaker.

Excluded from summary: TG-003 (plan gate — Enterprise-only custom report builder), TG-004 (enablement — Slack integration live on all plans), TG-005 (rep-stated, no prospect voice).

Data gap flag: 0 of 5 deals have amounts in the provided files.
rep-performance 5 tests
stale-pipeline-by-repPASS 1.00
Plain-English question: Which open deals have had no email, call, or meeting in the last 7 days?
Prompt — what the model saw
The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.0090 · 37s · in 39,560 / out 4,470 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ bryce_top_amount✓ cole_stale_amt✓ dana_stale_amt✓ stale_count

✓ Correct Answer (ground truth)

Plain-English answer: 65 open deals are stale; the largest is Deal-2D1F1B at $240,000, with Bryce Harmon carrying the most stale dollars at $626,243.
The answer in one line: jake_amt = 626,243; jenn_amt = 261,645; kevin_amt = 252,905.03; top_amount = 240,000; stale_count = 65; top_alias = Deal-2D1F1B
jake_amt626,243
jenn_amt261,645
kevin_amt252,905.03
stale_count65
top_amount240,000
top_aliasDeal-2D1F1B
graded checksstale_recall_ge_80 · grouped_by_owner · bryce_top_amount · cole_stale_amt · dana_stale_amt · stale_count

Model's actual answer

show the model's full answer
Staleness rule: snapshot 2026-09-05, cutoff = 2026-08-29. A deal is stale if max(last_email, last_call, last_meeting) < 2026-08-29. Days = 2026-09-05 − that max date. Deal-level date fields ignored.

156 open deals in the file; 2 have no row in the engagements table (recency not computable — flagged at the end); 63 are stale. Ethan note: 0 deals were dropped from the file as archived — owners.csv has no archived flag applied to these four.

BRYCE HARMON (119337721) — 13 stale deals, total $626,243.00
Deal-2D1F1B | DS1 | $240,000 | 81 days (2026-06-16)
Deal-66D1FC | DS1 | $99,000 | 16 days (2026-08-20)
Deal-950043 | DS1 | $70,000 | 19 days (2026-08-17)
Deal-B23205 | DS1 | $45,000 | 16 days (2026-08-20)
Deal-7BBDFA | DS3 | $37,440 | 46 days (2026-07-21)
Deal-332637 | DS2 | $36,000 | 9 days (2026-08-27)
Deal-1BEEBF | DS1 | $31,500 | 19 days (2026-08-17)
Deal-C5658B | DS1 | $23,400 | 16 days (2026-08-20)
Deal-40522D | DS3 | $21,000 | 19 days (2026-08-17)
Deal-F0EBBB | DS3 | $11,400 | 24 days (2026-08-12)
Deal-E25A09 | DS1 | $6,000 | 9 days (2026-08-27)
Deal-C9C286 | DS2 | $5,502 | 9 days (2026-08-27)
Deal-012CB1 | DS1 | $1 | 23 days (2026-08-13)

DANA MERCER (83155923) — 14 stale deals, total $261,645.00
Deal-44EA29 | DS2 | $60,000 | 10 days (2026-08-26)
Deal-E51FB7 | DS2 | $43,875 | 12 days (2026-08-24)
Deal-B42F46 | DS1 | $27,000 | 19 days (2026-08-17)
Deal-BA3DDC | DS3 | $23,400 | 15 days (2026-08-21)
Deal-9DDE86 | DS2 | $20,000 | 15 days (2026-08-21)
Deal-215CCA | DS3 | $18,900 | 17 days (2026-08-19)
Deal-5EED42 | DS3 | $16,250 | 11 days (2026-08-25)
Deal-57887A | DS2 | $15,000 | 8 days (2026-08-28)
Deal-B7EBD1 | DS5 | $9,000 | 16 days (2026-08-20)
Deal-3974EB | DS4 | $9,000 | 8 days (2026-08-28)
Deal-F40F04 | DS2 | $8,100 | 15 days (2026-08-21)
Deal-87DDD1 | DS1 | $5,000 | 19 days (2026-08-17)
Deal-F336B6 | DS3 | $4,200 | 15 days (2026-08-21)
Deal-0660B4 | DS4 | $1,920 | 16 days (2026-08-20)

COLE INGRAM (83155924) — 18 stale deals, total $252,905.03
Deal-D04904 | DS2 | $58,529.25 | 11 days (2026-08-25)
Deal-B25F40 | DS3 | $40,000 | 8 days (2026-08-28)
Deal-813836 | DS2 | $32,175 | 11 days (2026-08-25)
Deal-1BA595 | DS2 | $31,750 | 11 days (2026-08-25)
Deal-CFE1E8 | DS3 | $18,000 | 11 days (2026-08-25)
Deal-CD47A6 | DS2 | $12,168 | 11 days (2026-08-25)
Deal-627646 | DS3 | $11,193 | 11 days (2026-08-25)
Deal-FF809F | DS2 | $7,781.20 | 11 days (2026-08-25)
Deal-AF932D | DS2 | $7,225.40 | 11 days (2026-08-25)
Deal-A71728 | DS2 | $6,947.50 | 11 days (2026-08-25)
Deal-8BC9F5 | DS2 | $5,616 | 10 days (2026-08-26)
Deal-175395 | DS3 | $4,779.88 | 11 days (2026-08-25)
Deal-481E24 | DS3 | $4,140 | 10 days (2026-08-26)
Deal-C7F9BF | DS2 | $3,360 | 11 days (2026-08-25)
Deal-2F3A66 | DS3 | $3,334.80 | 11 days (2026-08-25)
Deal-342E96 | DS2 | $2,700 | 24 days (2026-08-12)
Deal-E568D5 | DS3 | $1,875 | 11 days (2026-08-25)
Deal-FD9F4E | DS5 | $1,330 | 10 days (2026-08-26)

ALEX FRANKLIN (84342457) — 18 stale deals, total $102,336.00
Deal-CC08D1 | DS1 | $24,000 | 16 days (2026-08-20)
Deal-E73427 | DS3 | $18,000 | 10 days (2026-08-26)
Deal-885F45 | DS2 | $9,300 | 12 days (2026-08-24)
Deal-C2FF3C | DS1 | $8,316 | 10 days (2026-08-26)
Deal-0D2F7A | DS3 | $5,100 | 12 days (2026-08-24)
Deal-6C60D4 | DS3 | $4,800 | 12 days (2026-08-24)
Deal-13FEBD | DS2 | $4,680 | 12 days (2026-08-24)
Deal-9D0060 | DS3 | $3,840 | 12 days (2026-08-24)
Deal-690476 | DS2 | $3,600 | 18 days (2026-08-18)
Deal-C6D97A | DS4 | $3,240 | 8 days (2026-08-28)
Deal-EE195F | DS3 | $3,120 | 8 days (2026-08-28)
Deal-278DEC | DS3 | $2,700 | 8 days (2026-08-28)
Deal-635B8E | DS3 | $2,600 | 18 days (2026-08-18)
Deal-6883F3 | DS1 | $2,400 | 16 days (2026-08-20)
Deal-4A13AD | DS3 | $2,160 | 26 days (2026-08-10)
Deal-F67D31 | DS2 | $1,800 | 8 days (2026-08-28)
Deal-5FDCE4 | DS3 | $1,600 | 12 days (2026-08-24)
Deal-BA571A | DS4 | $1,080 | 18 days (2026-08-18)

FARID OSMAN (716654662) — 0 stale deals, $0
ELENA SINCLAIR (701163055) — 0 stale deals, $0

Totals: 63 stale deals, $1,243,129.03 (626,243.00 + 261,645.00 + 252,905.03 + 102,336.00 = 1,243,129.03).

Data gaps, stated explicitly:
- Deal-3EED2C (Alex Franklin, DS2, $7,200) and Deal-57FF13 (Elena Sinclair, DS1, $2,100) have no row in engagements_by_deal_90d.csv. Recency cannot be computed; both excluded from the counts and totals above. Neither is confirmed contacted or stale.
- 17 deals have a last_meeting dated after the 2026-09-05 snapshot (e.g. Deal-3795AD 2026-10-02, Deal-62D607 2026-09-22, Deal-944310 2026-09-15). Per the max-date rule they count as recent and are not listed as stale, but those are scheduled-future dates, not logged past contact. Excluding them would add 3 deals to Bryce Harmon, 3 to Dana Mercer, 2 to Alex Franklin, 3 to Farid Osman, and change Farid from 0 to 3 stale.
- Bryce Harmon's Deal-2D1F1B ($240,000, 81 days) is the single largest stale exposure; last touch 2026-06-16 (email), no logged call or meeting.
activity-mix-vs-outcomePASS 1.00
Plain-English question: Per rep, what did the last 30 days of emails, calls, and meetings look like next to the deals that advanced?
Prompt — what the model saw
The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.0175 · 150s · in 20,925 / out 23,052 tok
✓ alex_ds2_30d✓ alex_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

Plain-English answer: Alex Franklin is the standout signal with 18 deals entering DS2 in 30 days (next closest: Bryce Harmon with 4), alongside per-rep activity totals.
The answer in one line: alex_ds2_30d = 18; bryce_ds2_30d = 4; note = legacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
alex_ds2_30d18
bryce_ds2_30d4
notelegacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
graded checksalex_ds2_30d · alex_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Snapshot: 2026-09-05. 30-day window = 2026-08-06 → 2026-09-05 inclusive.

Per-rep 30-day activity (summed from engagements_by_deal_90d.csv across deals owned in deals_open.csv):

Rep              Emails  Calls  Meetings  Total  Mix % (E/C/M)      DS2 entries in 30d  Total acts per DS2
Alex Franklin      307     36      41      384   79.9 / 9.4 / 10.7   18                  21.33
Bryce Harmon       162      0      43      205   79.0 / 0.0 / 21.0    4                  51.25
Cole Ingram         96     14       1      111   86.5 / 12.6 / 0.9    2                  55.50
Farid Osman         38      0      34       72   52.8 / 0.0 / 47.2    1                  72.00
Dana Mercer         84     18      11      113   74.3 / 15.9 / 9.7    1                 113.00
Elena Sinclair       0      0       0        0   n/a                  0                  n/a
Gavin Porter         0      0       0        0   n/a                  0                  n/a
Hugo Lindqvist       0      0       0        0   n/a                  0                  n/a

Mix arithmetic: Alex 307/384=79.9%, 36/384=9.4%, 41/384=10.7%. Bryce 162/205=79.0%, 0/205=0%, 43/205=21.0%. Cole 96/111=86.5%, 14/111=12.6%, 1/111=0.9%. Farid 38/72=52.8%, 0/72=0%, 34/72=47.2%. Dana 84/113=74.3%, 18/113=15.9%, 11/113=9.7%.

Efficiency ranking (activities per DS2 entry, lowest = most efficient):
1. Alex Franklin — 384 / 18 = 21.33
2. Bryce Harmon — 205 / 4 = 51.25
3. Cole Ingram — 111 / 2 = 55.50
4. Farid Osman — 72 / 1 = 72.00
5. Dana Mercer — 113 / 1 = 113.00

Most efficient rep: Alex Franklin (21.33 activities per DS2 entry).
Highest-volume rep: Alex Franklin (384 total activities; also most emails, calls, and meetings individually).
They do not differ — same rep on both.

DS2 entries counted by owner (t_ds2 in window): Alex Franklin 18; Bryce Harmon 4 (Deal-25F752, Deal-CA7DC0, Deal-1CCE5C, Deal-D73B89); Cole Ingram 2 (Deal-1BA595, Deal-42326B); Dana Mercer 1 (Deal-57887A); Farid Osman 1 (Deal-499BF6); Elena Sinclair 0.

Data notes:
- Elena Sinclair's only deal (Deal-57FF13) has no row in engagements_by_deal_90d.csv, and Alex Franklin's Deal-3EED2C (a DS2 entry on 2026-09-03) also has no engagement row — both are counted as 0 activities. Alex's 18 DS2 entries therefore include one deal contributing zero measured activity.
- Gavin Porter and Hugo Lindqvist are archived and own no open deals: 0 activities, 0 DS2 entries.
- No causal attribution made.
ae-qtd-scorecardPASS 1.00
Plain-English question: Build Alex Franklin's QTD scorecard: bookings versus quota, pipeline, and conversion.
Prompt — what the model saw
Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0156 · 73s · in 38,592 / out 15,277 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

Plain-English answer: 75% attainment ($150,000 on a $200,000 quota), $113,500 new versus $36,500 expansion, pipeline by stage, and the rolling DS2-to-won rate.
The answer in one line: quota = 200000; bookings = 150000; new_bookings = 113500; expansion_bookings = 36500; open_deals_count = 125; attainment_pct = 75; owner = Alex Franklin (+12 more below)
quota200,000
attainment_pct75
bookings150,000
expansion_bookings36,500
expansion_count3
losses_qtd27
new_bookings113,500
new_count5
open_deals_count125
wins_qtd8
activity_30d_totalsemails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
allowed_new18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
ds2_to_wonwindow_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
ownerAlex Franklin
pipeline_by_stage{"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
prior_quarter_excludeddeal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
quarter2026-Q3
snapshot_date2026-09-05
top_loss_reasonreason=Lost- Timing (1 year or more); count=13
graded checksbookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
ALEX FRANKLIN — QTD SCORECARD AS OF 2026-09-05 (2026-Q3, quarter = 2026-07-01 to 2026-09-30)

Data caveats up front: activity columns (emails_30d/calls_30d/meetings_30d/notes_30d) are the only activity data supplied — no per-activity dates, so "last 30 days" is read as those supplied fields. No quota, segment, or product detail beyond quota.csv. One closed-won deal excluded per instruction.

1) BOOKINGS VS QUOTA
   Quota (2026-Q3):            $200,000
   Q3 closed-won to date:      $150,000  (8 deals)
   Attainment:                 150,000 / 200,000 = 75.0%
   Remaining gap:              $50,000  (25.0% of quota)
   Arithmetic check: 40,000 + 20,000 + 35,000 + 21,000 + 12,000 + 11,000 + 6,500 + 4,500 = 150,000
   Monthly: Jul $95,000 (A1C3E5 40,000 + F2C7D8 20,000 + B7D2F4 35,000); Aug $44,000 (C9E1A6 21,000 + A8B4D6 12,000 + D4B8C2 11,000); Sep 1–5 $11,000 (E6F3A9 6,500 + C5D9E2 4,500).
   Excluded: Deal-B3E6F1, $24,000, closed 2026-06-20 — before the quarter.

2) NEW VS EXPANSION (Q3 won)
   New:         $113,500  (5 deals, 75.7% of bookings) — Deal-A1C3E5 40,000; Deal-B7D2F4 35,000; Deal-C9E1A6 21,000; Deal-D4B8C2 11,000; Deal-E6F3A9 6,500
   Expansion:   $36,500   (3 deals, 24.3%) — Deal-F2C7D8 20,000; Deal-A8B4D6 12,000; Deal-C5D9E2 4,500
   Check: 113,500 + 36,500 = 150,000. New-deal avg $22,700; expansion avg $12,167.

3) ACTIVE PIPELINE BY STAGE (all 125 open deals, $1,260,390)
   Stage   Deals   Amount      Avg deal
   DS1      20     $284,621    $14,231
   DS2      28     $353,760    $12,634
   DS3      67     $552,705    $8,249
   DS4       5     $23,574     $4,715
   DS5       5     $45,730     $9,146
   Note: DS1 deals carry no entered_ds2 value (not yet in DS2) — expected, not a data gap.
   Subset with close_date inside the quarter (≤2026-09-30): 21 deals, $108,088 — DS2 2/$5,760; DS3 13/$68,124; DS4 3/$7,644; DS5 3/$26,560. Coverage on the $50,000 gap = 108,088 / 50,000 = 2.16x.
   Past-due: Deal-7A2454 (DS3, $1,275) has a 2026-09-04 close date and is still open.
   Aging: 29 open deals worth $282,960 entered DS2 more than 90 days before the as-of date.

4) ROLLING 90-DAY DS2-TO-WON RATE
   Window: 2026-06-07 to 2026-09-05. Cohort = 111 deals with entered_ds2 in window: 8 won, 27 lost, 76 still open.
   Closed-outcome rate: 8 / (8 + 27) = 8 / 35 = 22.9%
   Companion timing: DS2→close for those 8 wins ran 23–41 days, median 28.5, mean 28.6. Losses closed out in 20–52 days from DS2, median 33. No loss resolved in ≤14 days.
   Wins: Deal-A1C3E5, Deal-F2C7D8, Deal-B7D2F4, Deal-C9E1A6, Deal-A8B4D6, Deal-D4B8C2, Deal-E6F3A9, Deal-C5D9E2.
   Deal-B3E6F1 is outside the 90-day cohort (entered DS2 2026-05-10) and already excluded from bookings.

5) WIN / LOSS COUNTS AND TOP LOSS REASON
   Q3 wins: 8   Q3 losses: 27   Count win rate: 8 / 35 = 22.9%
   Lost amount: $329,272 (vs $150,000 won — lost dollars are 2.2x won dollars)
   Loss reasons (count / amount):
     Lost- Timing (1 year or more)   13 / $184,681   (48.1% of losses, 56.1% of lost $)
     Competitor                       5 / $49,020
     MIA                              5 / $45,831
     Lost DM                          2 / $17,940
     Feature Request                  1 / $21,000
     Lost- Does not fit ICP (write in notes) 1 / $10,800
   Top loss reason: Lost- Timing (1 year or more) — 13 of 27 losses, $184,681, including the three largest losses: Deal-B3ABED $40,001, Deal-D1A623 $25,200, Deal-79B7A1 $25,000.

6) ACTIVITY VOLUME, LAST 30 DAYS (sum across all 161 deals)
   Emails: 807   Calls: 112   Meetings: 128   Notes: 50
   Total logged: 1,097 activities; 73.6% email, 10.2% calls, 11.7% meetings, 4.6% notes.
   Per-deal averages: won 8 deals = 11.12 emails / 3.88 calls / 2.88 meetings / 2.62 notes; lost 27 = 4.04 / 0.93 / 0.48 / 0.93; open 125 = 4.79 / 0.43 / 0.72 / 0.01.
   Open pipeline with zero calls in the window: 103 of 125 deals ($1,024,660 of $1,260,390).
   Zero-activity deals in the window: Deal-3EED2C (DS2, $7,200) has 0 in all four fields; Deal-0D2F7A and Deal-690476 have 0 emails and 1 call each.

7) COACHING OBSERVATIONS (grounded in the above)
   a) "Timing" is being recorded as an outcome, not managed as an objection. 13 of 27 losses and $184,681 — 56.1% of lost dollars — land on Lost- Timing (1 year or more), and those deals sat 20–52 days in DS2 before closing lost, so they were worked, not stalled-forever. The pattern with a 22.9% DS2 conversion and 2.2x lost-to-won dollars says qualification on compelling event is the constraint, not effort. Next quarter's biggest lever is disqualifying no-trigger deals at DS2 entry instead of carrying them to a "timing" loss 30+ days later.
   b) The motion is email-heavy where it is winning on calls. 807 of 1,097 activities (73.6%) are email versus 112 calls; won deals averaged 3.88 calls and 2.62 notes, open deals 0.43 calls and 0.01 notes, and 103 of 125 open deals got zero calls in 30 days. Zero notes on the open book means no recorded multi-threading or next-step documentation. Converting even part of the email volume into calls on the top 21 in-quarter deals ($108,088) is the highest-yield activity shift available.
   c) Late-stage and mid-pipeline are thin relative to the gap. Only 10 deals sit in DS4/DS5 ($69,304 combined) versus a $50,000 gap, coverage of 2.16x rests on 21 in-quarter deals of which 13 ($68,124, 63.0%) are DS3 and untested at DS4, and 29 deals worth $282,960 have aged past 90 days in DS2. Add the past-due Deal-7A2454 and the risk is a Q3 that stops at 75% because there is no DS5 cushion — the fix is advancing DS3 volume now and cleaning aged DS2 rather than adding DS1 logos (20 deals, $284,621, none dated to close in-quarter).
multithreading-gap0.75
Plain-English question: Which open deals are single-threaded or under-threaded?
Prompt — what the model saw
Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0076 · 41s · in 23,721 / out 6,511 tok
✓ flagged_recall✓ persona_language✗ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

Plain-English answer: Deals with fewer than 2 active contacts (single-threaded) or fewer than 3, or all in one persona (under-threaded), each flagged with the missing persona.
The answer in one line: active_window_start = 2026-07-07
active_window_start2026-07-07
allowed_new59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
best_persona_ruleDS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flag_ruleflag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
flaggeddeal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flaggedDeal-4B0BEB, Deal-84DBA6, Deal-D348E1
persona_universeeconomic buyer, champion, HR admin, IT security, finance
graded checksflagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Data gaps up front: neither file has an amount or stage column, so amount and stage are reported as "not in data" for every deal. No deal amounts or stages exist anywhere in the supplied files; I have not estimated them. Persona recommendations are therefore based on deal shape, not stage.

Activity rule applied: reference date 2026-10-19 minus 60 days = cutoff 2026-08-20; active = last_engaged_date >= 2026-08-20 and is_former = false. 15 deals scanned, 12 flagged, 3 clean (Deal-84DBA6, Deal-4B0BEB, Deal-D348E1).

1) Deal-EC3025 — company C-FDD0C7
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 2 — CT-047C54 (champion, 2026-09-02). Dropped: CT-F2C1AE (economic buyer, former).
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Add: economic buyer. On file: CT-6827DB, Chief People Officer, economic buyer.

2) Deal-92D97D — company C-E23238
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 2 — CT-01F5B4 (HR admin, 2026-08-28). Dropped: CT-A902AE (champion, 2026-06-01, 140 days stale).
   Personas present: HR admin
   Personas missing: economic buyer, champion, IT security, finance
   Add: champion (no active champion; single HR-admin thread). On file: none on file.

3) Deal-50D386 — company C-EB10E4
   Amount: not in data | Stage: not in data
   Active contacts: 2 of 2 — CT-AA41B2 (champion, 2026-09-01), CT-B9C35B (HR admin, 2026-08-25).
   Personas present: champion, HR admin
   Personas missing: economic buyer, IT security, finance
   Add: economic buyer. On file: CT-A1C4B3, Chief People Officer, economic buyer.

4) Deal-D0D6B5 — company C-32918E
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 3 — CT-87CED4 (champion, 2026-09-02). Dropped: CT-DE6D7C (champion, 2026-08-19), CT-FD70B2 (champion, 2026-08-07).
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Add: economic buyer. On file: CT-1FA4DB, Chief People Officer, economic buyer.

5) Deal-5BFE3B — company C-535D36
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 2 — CT-57123B (champion, 2026-08-31). Dropped: CT-5CE757 (champion, 2026-08-12).
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Add: economic buyer. On file: none on file.

6) Deal-36C33F — company C-077A0E
   Amount: not in data | Stage: not in data
   Active contacts: 0 of 3 — only CT-4FE556 (IT security, 2026-08-15) is non-former and it is 65 days stale; CT-405B45 and CT-86B22F are former.
   Personas present: none active
   Personas missing: economic buyer, champion, HR admin, IT security, finance
   Add: champion (no engaged contact at all; needs a live thread before anything else). On file: CT-1DB73E, Chief People Officer, economic buyer — but a champion is the gating hire here.

7) Deal-885F45 — company C-5E8EFB
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 2 — CT-51C81E (economic buyer, 2026-08-26). Dropped: CT-D9A0E8 (champion, 2026-08-11).
   Personas present: economic buyer
   Personas missing: champion, HR admin, IT security, finance
   Add: champion (buyer is engaged, but the champion has gone quiet since 2026-08-11). On file: CT-B3F25D, IT Security Lead, IT security — does not fit the gap.

8) Deal-FCBE5B — company C-737030
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 1 — CT-4A5317 (champion, 2026-08-29).
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Add: economic buyer. On file: none on file.

9) Deal-5408B0 — company C-2AE3AA
   Amount: not in data | Stage: not in data
   Active contacts: 1 of 2 — CT-D33AE4 (champion, 2026-09-01). Dropped: CT-8742FD (HR admin, 2026-08-18).
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Add: economic buyer. On file: CT-07FA76, Chief People Officer, economic buyer.

10) Deal-C6D97A — company C-5A8FC2
    Amount: not in data | Stage: not in data
    Active contacts: 2 of 3 — CT-223DDC (champion, 2026-08-31), CT-B03555 (champion, 2026-08-20). Dropped: CT-4E8A2B (champion, 2026-08-05).
    Personas present: champion only (all three contacts are the same persona)
    Personas missing: economic buyer, HR admin, IT security, finance
    Add: economic buyer. On file: none on file.

11) Deal-F9A08A — company C-0D15DF
    Amount: not in data | Stage: not in data
    Active contacts: 1 of 2 — CT-931B10 (champion, 2026-09-03). Dropped: CT-913581 (economic buyer, 2026-06-20, 121 days stale).
    Personas present: champion
    Personas missing: economic buyer, HR admin, IT security, finance
    Add: economic buyer. On file: CT-697541, Chief People Officer, economic buyer.

12) Deal-D348E1 — wait, that one is clean. Rechecking: Deal-D348E1 (C-804F29) has 4 active contacts across 4 personas, so it is not flagged.

Corrected count: 11 flagged deals, not 12. Arithmetic: 15 deals total; clean = Deal-84DBA6 (3 active, 3 personas), Deal-4B0BEB (3 active, 3 personas), Deal-D348E1 (4 active, 4 personas); 15 − 3 = 12. But item 12 above was an error in numbering only — the list contains 11 entries (items 1–11). Let me recount the entries: EC3025, 92D97D, 50D386, D0D6B5, 5BFE3B, 36C33F, 885F45, FCBE5B, 5408B0, C6D97A, F9A08A = 11. Plus clean Deal-84DBA6, Deal-4B0BEB, Deal-D348E1 = 14. One deal is unaccounted: Deal-D348E1 counted once; total deals in file = 15? Deal IDs: 61032318100, 59728118877, 61055128146, 60081655042, 51674270311, 63929535929, 63739413805, 61038797752, 60686135564, 62639586615, 60182332309, 61750885954, 62121783047, 49757401138 = 14 deals. So 14 − 3 clean = 11 flagged. Final: 11 flagged, 3 clean, 14 total.
talk-track-from-transcriptsPASS 1.00
Plain-English question: From this rep's last 10 calls: how do they open, handle objections, and how often do they lock a next step?
Prompt — what the model saw
From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0048 · 31s · in 21,065 / out 2,766 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

Plain-English answer: 8 of 10 calls open with the same customer-win story; objection handling, next-step agreement rate, and every competitor raised, with quotes.
The answer in one line: n_transcripts = 10; objection_count = 3; rep = Alex Franklin (+3 more below)
n_transcripts10
objection_count3
allowed_new13 items — e.g. TT-001, TT-002, TT-003, TT-004
competitor_quotesAwardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
competitors_prospect_raisedAwardco, Kudos
next_stepagreed=7; total=10; rate=0.7
objectionsobjection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
opener_themetheme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
repAlex Franklin
rep_raised_competitors_excludedWorkhuman
graded checksobjection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
Read all 10 transcripts (TT-001 → TT-010, 2026-07-28 → 2026-09-03).

DATA CAVEAT: the extract is sparse — most calls carry a rep line at minute 0, a prospect objection at minute 4–6, a rep reply at 5–8, and a close at 12–15. There are no rep lines at minutes 1–3 or 5 in 9 of 10 calls, so "first five minutes" can only be judged from the minute-0 line plus anything landing at minute 4–5. I am not inferring content that isn't in the file.

1) WHAT THEY LEAD WITH (first five minutes)
8 of 10 calls (80%) open with the identical, word-for-word 400-person retailer / 18% regretted-turnover case study: TT-001 Deal-D348E1, TT-002 Deal-5408B0, TT-003 Deal-547B2B, TT-005 Deal-C61CF7, TT-006 Deal-60C2C2, TT-007 Deal-EDC141, TT-008 Deal-D9A12F, TT-010 Deal-84DBA6.
  "a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards"
They do not vary the story by objection history or persona. The two deviations:
- TT-004 Deal-403845, agenda-first: "I put together a short agenda — security review first, then pricing."
- TT-009 Deal-1E2498, pricing-first, and the only call that acknowledges a prior call: "You asked for straight pricing last time, so let's start there."
In TT-005 Deal-C61CF7 the only other early content is a rep-initiated competitive jab at minute 2: "And unlike Workhuman, our pricing includes the full rewards catalog with no extra margin."

2) THE THREE MOST COMMON OBJECTIONS AND HOW THEY'RE HANDLED
Objection counts sum to 10 across 10 calls (4 + 3 + 3), so on average one objection per call, and no call shows two distinct objection types to the same rep.

a) Budget locked / no new line item — 4 calls: TT-001 Deal-D348E1, TT-003 Deal-547B2B, TT-006 Deal-60C2C2, TT-010 Deal-84DBA6.
   Handle: reframe as self-funded from turnover savings, with a hard dollar figure, verbatim the same in all four:
   "Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."
   Outcome: 3 of 4 (TT-001, TT-003, TT-006) then closed on a next step; TT-010 did not.

b) Timing / defer to next quarter, open enrollment — 3 calls: TT-002 Deal-5408B0, TT-005 Deal-C61CF7, TT-008 Deal-D9A12F.
   Handle: shrink scope instead of arguing urgency — a single-department 90-day pilot, verbatim in all three:
   "What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"
   Outcome: 3 of 3 closed on a next step.

c) Status-quo / "we use a spreadsheet and gift cards" — 3 calls: TT-004 Deal-403845, TT-007 Deal-EDC141, TT-009 Deal-1E2498.
   Handle: scale-and-automation reframe, verbatim in all three:
   "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."
   Outcome: 1 of 3 (TT-009) closed; TT-004 and TT-007 did not.

3) CONCRETE NEXT STEP AGREED — 7/10 = 70%
Agreed (7): TT-001, TT-002, TT-003, TT-005, TT-006, TT-008, TT-009 — all seven on the same ask, "Should we lock the next step — a working session with your team this week?", answered with "Yes, Thursday at 2pm works."
Not agreed (3): TT-004 Deal-403845 ("I'll leave it with you."), TT-007 Deal-EDC141 ("Fair enough."), TT-010 Deal-84DBA6 ("Understood, thanks for the candor.").
Ask rate equals close rate here: the rep made that specific ask in 7 of 10 calls and closed 7 of 7. Arithmetic: 7 ÷ 10 = 0.70. In the 3 calls with no ask, 0 of 3 produced a commitment — all three ended on a prospect-side deferral the rep acknowledged and stopped on.

4) EVERY COMPETITOR A PROSPECT RAISED
- Awardco — Deal-547B2B, TT-003, minute 4: "We're also in late talks with Awardco — their rewards catalog looks bigger than yours." Rep countered on automation/analytics, not catalog breadth.
- Kudos — Deal-EDC141, TT-007, minute 4: "How are you different from Kudos? Our CEO used them at her last company." Rep answered social recognition vs. automated milestones + analytics.
That is the complete list of prospect-raised competitors: 2 in 10 calls.
Not counted, flagged for accuracy: Workhuman appears in Deal-C61CF7 (TT-005) but the rep raised it, not the prospect. "The committee" / "budget committee" (TT-004, TT-010) is a buying process blocker, not a competitor.

TWO COACHING NOTES
1. The close is binary and fully within the rep's control: 7 asks, 7 yeses, 0 yeses when no ask was made. The three misses (TT-004, TT-007, TT-010) all ended on a prospect deferral the rep absorbed with a one-line acknowledgement — "Understood — I'll leave it with you." Ask for the working session on every call, including stalled ones; there is no evidence in this data that asking costs anything.
2. The budget-"locked" handle is one fixed script, and it did not move the one call where the real block was approval authority, not funding: in TT-010 the prospect restated the committee wall immediately after the $210k math ("We'll have to wait for the committee — I can't commit to anything today."). Separate "no money" from "no signer" — when the blocker is a committee, the ask should be a step that helps the committee decide, not a savings calculation.
reporting-analytics 5 tests
q3-weighted-forecastPASS 1.00
Plain-English question: Build the Q3 2026 weighted forecast from the open pipeline, and show the math deal by deal.
Prompt — what the model saw
Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0107 · 90s · in 33,307 / out 8,797 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

Plain-English answer: A stage-weighted Q3 forecast number computed from the open pipeline, with per-stage math that reconciles to the deal data.
The answer in one line: excluded_amount = 227575; best_case_total = 203565; weighted_forecast = 115,976.75; commit_total = 44729; excluded_out_of_quarter = 32; best_case_count = 24; flags = blank owner column + empty why_buys fields must be surfaced (+2 more below)
excluded_amount227,575
best_case_count24
best_case_total203,565
commit_count7
commit_total44,729
excluded_out_of_quarter32
weighted_forecast115,976.75
flagsblank owner column + empty why_buys fields must be surfaced
top5_best_caseDeal-2D7423, Deal-25F752, Deal-E53952, Deal-5EED42, Deal-FA32A0
graded checkscommit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
Q3 2026 FORECAST — deals in default pipeline, close date 2026-07-01 to 2026-09-30 (extract pulled 2026-09-05; 86 open deals total)

In-quarter deal counts
  COMMIT      7 deals
  BEST_CASE   24 deals
  PIPELINE    23 deals  (counted at zero)
  In-quarter total: 54 deals; 449,931.40
  Check: 44,729.00 + 203,565.00 + 201,637.40 = 449,931.40

COMMIT total (100%)
  11,200.00  Deal-547B2B  DS5  2026-09-11
   9,000.00  Deal-B7EBD1  DS5  2026-09-10
   9,000.00  Deal-403845  DS5  2026-09-11
   6,360.00  Deal-A2B47C  DS5  2026-09-11
   5,400.00  Deal-2465CE  DS5  2026-09-10
   2,520.00  Deal-A5E80A  DS1  2026-09-11
   1,249.00  Deal-499BF6  DS2  2026-09-30
  COMMIT total = 44,729.00

BEST_CASE total (35%)
  38,935 + 24,000 + 19,656 + 16,250 + 11,116 + 10,800 + 10,500 + 9,890 + 9,720 + 9,000
  + 7,200 + 5,400 + 5,160* (excluded — outside quarter, see below) ...
  In-quarter BEST_CASE sum = 203,565.00
  Weighted contribution = 203,565.00 x 0.35 = 71,247.75

Weighted forecast
  44,729.00 x 1.00 = 44,729.00
  203,565.00 x 0.35 = 71,247.75
  PIPELINE 201,637.40 x 0 = 0.00
  Weighted Q3 2026 forecast = 115,976.75

Excluded — close date outside the quarter (2026-10-01 to 2026-10-15)
  Count: 32 deals
  Total amount: 227,575.00
  Composition: 22 PIPELINE, 9 BEST_CASE, 1 COMMIT
  Largest exclusions: Deal-E51FB7 43,875.00 (10-01, PIPELINE); Deal-B936FE 18,000.00 (10-09, PIPELINE); Deal-D9A12F 17,000.00 (10-15, PIPELINE); Deal-D348E1 13,770.00 (10-15, COMMIT); Deal-4062CF 10,800.00 (10-15, PIPELINE)
  Note: the only excluded COMMIT (Deal-D348E1, 13,770.00) and the excluded BEST_CASE (Deal-C61CF7 5,400.00) contribute zero to this quarter.
  Reconciliation: 449,931.40 in-quarter + 227,575.00 excluded = 677,506.40 = full extract.

Top 5 BEST_CASE deals by amount inside the quarter
  1. Deal-2D7423   38,935.00  DS3  2026-09-30
  2. Deal-25F752   24,000.00  DS4  2026-09-25
  3. Deal-E53952   19,656.00  DS4  2026-09-30
  4. Deal-5EED42   16,250.00  DS3  2026-09-30
  5. Deal-FA32A0   11,116.00  DS3  2026-09-25
  Sum of top 5 = 109,957.00, which is 54.0% of in-quarter BEST_CASE (109,957.00 / 203,565.00)

## Data quality
Owner is blank on 85 of 86 rows, so no deal can be attributed or challenged by rep. why_buys_chars is 0 on 71 of 86 rows (including the two largest in-quarter BEST_CASE deals, Deal-2D7423 and Deal-25F752), so the categorization carries no stated qualification rationale. Category and stage contradict each other — Deal-A5E80A is COMMIT at DS1 and Deal-499BF6 is COMMIT at DS2, together 3,769.00 of the 44,729.00 COMMIT total — and four deals are still open past their close date at the 2026-09-05 pull (Deal-333EBB 2026-08-28, Deal-57FF13 2026-09-02, Deal-31AD2C and Deal-7A2454 2026-09-04), meaning slip/close-date hygiene is unmaintained. The extract has no account names, no currency, and no as-of/version field, and its window runs to 2026-10-15 while the pull date is 2026-09-05, so late-quarter amounts cannot be tied to a single consistent snapshot if run unattended.

⚠️ File-mutation verifier: 1 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/deals.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/deals.csv`: `/Users/amaniphipps/.hermes/cache/scratch/deals.csv` exists but this task has not seen its full current con…
aha-moment-2x2PASS 1.00
Plain-English question: Do companies that hit 5+ givers and one redemption in month one actually retain better at 24 months?
Prompt — what the model saw
Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0161 · 86s · in 43,918 / out 14,960 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

Plain-English answer: The four cohort cells are graded against computed 24-month retention; the numbers come from the scorer's cohort table, not opinion.
The answer in one line: both_5plus_givers_and_redemption = 66.0% (31 of 47)
both_5plus_givers_and_redemption66.0% (31 of 47)
exclusionsnone — claim bounded as correlation, not causation
givers_only46.9% (23 of 49)
largest_single_signal_liftunique givers
neither40.0% (38 of 95)
redemption_only31.0% (9 of 29)
graded checksboth_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
**Verdict:** the hypothesis holds directionally but only partially. Both signals is the best cell by a wide margin (66.0% vs 45.9% base, +20.0pp). But 5+ givers is the only *single* signal that lifts retention (+6.9pp); a redemption alone is **negative** (−9.0pp vs neither). Redemptions only pay off on top of givers.

**Full 2×2 (n=220, denominator = all companies)**

```
                     retained   n    24-mo retention   95% CI (Wilson)
both signals           31      47       66.0%          52–78%
givers-only            23      49       46.9%          34–61%
redemption-only         9      29       31.0%          17–49%
neither                38      95       40.0%          31–50%
---------------------------------------------
all companies         101     220       45.9%
```

Arithmetic: 31/47 = 66.0%; 23/49 = 46.9%; 9/29 = 31.0%; 38/95 = 40.0%. 31+23+9+38 = 101; 47+49+29+95 = 220; 101/220 = 45.9%.

Marginal effects (this is where the answer actually lives):
- Adding 5+ givers among redemption-only companies: 66.0% − 31.0% = **+34.9pp**
- Givers alone vs neither: 46.9% − 40.0% = **+6.9pp**
- Adding a redemption among givers-only companies: 66.0% − 46.9% = **+19.0pp**
- Redemption alone vs neither: 31.0% − 40.0% = **−9.0pp**

**Exclusions: none.** All 220 rows were used. No duplicate company_key, no blank or non-numeric m1_users / m1_redemptions / current_status. Signup months run 2023-01 through 2023-07, so at October 2026 the cohort is 39–45 months old, all past the 24-month mark — `current_status = 'active'` is a valid 24-month retention proxy. I did not exclude the three `non_renewing` companies (C-0B2078FB, C-0A96134F, C-0BEAF685); they count as not retained, since the test is 24-month retention, not ARR survival.

**Largest single-signal lift: 5+ unique givers.** +6.9pp standalone (46.9% vs 40.0%), and the largest marginal swing anywhere in the table (+34.9pp when layered onto redemption companies, +7.3pp better than a redemption layered onto givers). Redemption is the weaker and, alone, worse signal — 31.0% is below the 40.0% no-signal baseline.

**What this proves:** in this 2023 cohort, the conjunction predicts materially better 24-month retention than either signal alone or neither — both-signals vs everyone else is 66.0% vs 42.8% (70/173), chi-square 9.67, p ≈ 0.002. Givers ≥5 as a standalone flag is also statistically distinguishable from <5 givers (56.3% vs 37.9%, p ≈ 0.007). Redemption alone is not (52.6% vs 42.4%, p ≈ 0.15). So the operational read is defensible: a redemption without a giver base does not indicate health, and treating "redeemed once" as an activation milestone will overcount healthy accounts.

**What this does not prove:**
- Causality or direction. Companies that accumulate 5 givers are plausibly larger, better-funded, better-fit, better-implemented accounts; the signal may be a proxy for company quality rather than a lever. No randomization, no propensity matching, no control for tier, ARR, industry, country, or Slack/Teams/HRIS presence (all available in the extract and unadjusted).
- Generalizability. One cohort, one signup vintage (2023 H1–H2), single extract date, 220 companies. `redemption-only` rests on 9 retained companies — the CI (17–49%) overlaps the baseline, so that −9.0pp could be noise.
- Threshold sensitivity. 5 givers and 1 redemption were the specified cutoffs; I did not test alternatives, so nothing here says 5 is the right line.
- Mechanism. Nothing separates "givers cause retention" from "retained accounts kept giving."
- Renewal structure. `current_status` collapses 12- and 24-month renewal behavior into one flag; a company that lapsed at month 13 and one that lapsed at month 24 look identical.

**Data caveats (affecting, not excluded):** C-F0A562FF reports m1_redemptions=10 with m1_users=0 — a redemption with zero users is internally impossible as stated, which suggests redemptions and users are counted on different definitions or the redemption count is not unique-user. Twelve companies report more redemptions than gives (C-0B8435BE, C-0B2213A9, C-0FD34543, C-0B057828, C-0D78BCB2, C-108C5CE4, C-0FEB5AAF, C-F0A562FF, C-0E8B0E96, C-0D66DF9E, C-0F58EBD7, C-0D1104A2). If redemptions are non-unique events rather than users, the redemption signal in this test is weaker than the label implies. Dropping C-F0A562FF would move redemption-only to 8/28 = 28.6%, which strengthens the direction of the finding rather than weakening it.
arr-rollforward-reconciliationPASS 1.00
Plain-English question: Does CRM ARR match billing ARR — and if not, exactly where is the variance?
Prompt — what the model saw
Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0070 · 54s · in 22,198 / out 5,824 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

Plain-English answer: Billing $604,739.28 vs CRM $603,581.76: a $1,157.52 variance decomposed into status-mismatch, rounding, and missing-record buckets.
The answer in one line: n_companies = 40; n_mismatched_accounts = 7; as_of = 2026-09-05 (+4 more below)
n_companies40
n_mismatched_accounts7
as_of2026-09-05
billing_arr_total604739.28
bucket_sum_equals_variance1
buckets{"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
crm_arr_total603581.76
ruleterm_months != 12 requires cf_agreement_end_date
variance1157.52
variance_definitionbilling_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
violationssubscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
graded checksbilling_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
ARR RECONCILIATION — as of 2026-09-05

Basis: billing ARR = MRR × 12, active subscriptions only (cancelled subscriptions carry no recurring revenue and are booked as a status discrepancy, not ARR). The gross alternative that multiplies every row including cancelled carries is shown at the end so the choice is auditable.

TOTALS
  CRM (company records) .......... $603,581.76   (39 companies)
  Billing (active subs) .......... $604,739.28   (37 active subs; 604,739.28 = 50,394.94 MRR × 12)
  Variance (Billing − CRM) ....... +$1,157.52    (Billing is higher)

DECOMPOSITION (sums exactly to +1,157.52)

  Bucket                    Amount        Accounts
  Status mismatch        -13,158.48      2
  Missing records        +11,952.00      2  (+28,449.24 billing-only, −16,497.24 CRM-only)
  Other (rate mismatch)   +2,400.00      1
  Rounding                   -36.00      2
  --------------------------------------------
  Total                   +1,157.52

  Check: 28,449.24 − 16,497.24 = 11,952.00
         28,449.24 − 16,497.24 + 2,400.00 − 36.00 − 13,158.48 = 1,157.52  ✓

MISMATCHED ACCOUNTS

1) C-21629AA4 — Missing record (billing only)  +$28,449.24
   SUB-0004, MRR 2,370.77 × 12 = 28,449.24. No company record exists.
   Suggested owner: RevOps / CRM admin — create company record, then assign to the AE of record. (No owner field in supplied data, so no named owner can be cited.)

2) C-0D5BBE3A — Missing record (CRM only)  −$16,497.24
   CRM ARR 16,497.24 (implied MRR 1,374.81 = 16,497.24/12) with no subscription row. Either a churned/never-provisioned account still carrying ARR, or the sub is absent from this extract.
   Suggested owner: CS/account manager for the account plus RevOps to confirm churn — write-down on CRM unless a live sub is found.

3) C-0F7269D7 — Other: rate mismatch  +$2,400.00
   Billing 2,233.00 × 12 = 26,796.00 vs CRM 24,396.00 (CRM implied MRR 2,033.00). Gap = 200.00/mo × 12 = 2,400.00.
   Suggested owner: AE of record with deal desk — determine whether a 200/mo discount or a rate change was never written back to the CRM.

4) C-0C8323BF — Status mismatch  −$4,905.24
   SUB-000E is cancelled (MRR 408.77) but CRM still carries 4,905.24 = 408.77 × 12 at full ARR.
   Suggested owner: CS/account manager to confirm churn date; RevOps to zero the CRM ARR in the period the cancellation took effect.

5) C-0DC4FB8C — Status mismatch  −$8,253.24
   SUB-000F is cancelled (MRR 687.77) but CRM still carries 8,253.24 = 687.77 × 12.
   Suggested owner: CS/account manager to confirm churn date; RevOps to zero the CRM ARR.

6) C-0D66DF9E — Rounding  −$16.00
   Billing 1,932.00 × 12 = 23,184.00 vs CRM 23,200.00.
   Suggested owner: RevOps (data hygiene) — CRM appears rounded to the nearest $100; fix by deriving CRM ARR from billed MRR.

7) C-14D70CE0 — Rounding  −$20.00
   Billing 1,515.00 × 12 = 18,180.00 vs CRM 18,200.00.
   Suggested owner: RevOps (data hygiene) — same nearest-$100 rounding pattern.

All other 32 shared accounts reconcile to the cent.

GROSS ALTERNATIVE (every subscription row × 12, cancelled included)
  Billing gross = $617,897.76; CRM = $603,581.76; variance = +$14,316.00.
  Same items, decomposed: missing records +11,952.00, other +2,400.00, rounding −36.00, status mismatch 0.00 → 14,316.00 ✓. Status mismatch is 0.00 here only because the two cancelled subs are billed at exactly the MRR still sitting in the CRM (4905.24 and 8253.24), so the discrepancy is a status flag, not a dollar gap.

TERM-RULE VIOLATIONS (term ≠ 12 months and cf_agreement_end_date empty)

  SUB-0002  C-1794A52C  24 months  active  cf_agreement_end_date: blank
  SUB-0019  C-22170CA1  36 months  active  cf_agreement_end_date: blank

  Compliant: SUB-000C (C-0DB48281, 24 mo, 2027-11-30) and SUB-001A (C-0FC4DBB8, 36 mo, 2027-11-30).
  ARR exposure on the two violations: 591.00 × 12 = 7,092.00 and 649.00 × 12 = 7,788.00 (both reconcile to CRM today, so no dollar variance — the gap is term/end-date data quality and renewal-risk visibility).
  Suggested owner: RevOps/contract admin to backfill from the signed order form; AE of record to confirm end date.

DATA GAPS: no owner, segment, or close-date fields were provided, so owners above are suggested by accountability type, not derived from source records.
kvm-month-over-monthPASS 1.00
Plain-English question: How did the four core KVMs move in August versus July?
Prompt — what the model saw
Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0028 · 40s · in 3,932 / out 3,438 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

Plain-English answer: Giving rate 60.27% (up 0.07%), redemptions per user 1.7302 (up 0.01%), with 1:1 meetings and pulse engagement each up fractionally — direction reported for all four.
The answer in one line: n_company_rows_per_month = 30; aggregation = simple mean of company-level values per month
n_company_rows_per_month30
aggregationsimple mean of company-level values per month
biggest_moverpulse_engagement
driving_segmentsize_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
kvms{"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
segment_detail{"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
graded checksfour_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Equal-weighted mean across all 30 companies (all `tier_three`, so `plan_tier` cannot differentiate); 30 companies in each month.

| KVM | 2026-07 | 2026-08 | Abs change | Rel change | Direction |
|---|---|---|---|---|---|
| Giving rate | 0.6023 | 0.6027 | +0.0004 | +0.07% | Up |
| Redemptions per user | 1.7300 | 1.7302 | +0.0002 | +0.01% | Up |
| 1:1 meetings engagement | 0.4469 | 0.4472 | +0.0003 | +0.06% | Up |
| Pulse check engagement | 0.6006 | 0.5086 | −0.0920 | −15.31% | Down |

Arithmetic: sum of the 30 monthly values ÷ 30; relative = abs ÷ July value × 100.

Largest relative move: pulse check engagement, −15.31% (−0.0920, from 0.6006 to 0.5086).

Size band drives it. Every `enterprise` company fell from 0.5331–0.5661 to 0.2619–0.2864; the enterprise band average moved 0.5500 → 0.2743 (−0.2757, −50.13%). `smb` moved 0.6588 → 0.6573 (−0.22%) and `mid_market` 0.5930 → 0.5942 (+0.21%). Example aliases: C-0B2895EF (0.5619 → 0.2864), C-0B827671 (0.5331 → 0.2698), C-0BA71F12 (0.5588 → 0.2723).

Plan tier does not explain it: all rows are `tier_three`, so no plan_tier comparison is possible.

Caveat: equal weighting is my choice; the file carries no user counts, so a user-weighted total cannot be computed.
rewards-ytd-section0.67
Plain-English question: Produce the redemption section of the weekly rewards report through August.
Prompt — what the model saw
Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0107 · 56s · in 56,946 / out 2,561 tok
✓ count✗ spend✗ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

Plain-English answer: 378 redemptions, $27,846 spend, 235 unique redeemers at 1.61 each, custom rewards at 39% of spend, US leading all countries.
The answer in one line: redemption_count = 378; unique_redeemers = 235; redemptions_per_redeemer = 1.61; total_spend_usd = 27846.00
redemption_count378
redemptions_per_redeemer1.61
unique_redeemers235
provider_mix_pct_of_spendTangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_countcountry=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
total_spend_usd27846.00
graded checkscount · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
REDEMPTION SECTION — WEEKLY REWARDS REPORT
Source: redemptions_ytd.csv (378 rows, 2026-01-01 through 2026-08-31)

LAST COMPLETED MONTH
August 2026 (2026-08). It is the last month with a full set of rows; the latest timestamp in the file is 2026-08-31T11:53. Note explicitly: today is 2026-10-19, so the calendar's last completed month is September 2026 — redemptions_ytd.csv contains zero September rows, so no September figures can be produced from the data provided.

HEADLINE (2026-08)
Redemption count: 48
Spend: $3,536.00
Unique redeemers (distinct user_key): 46
Redemptions per redeemer: 1.04 (48 / 46 = 1.0435)

PROVIDER MIX, % OF SPEND (2026-08)
Provider      Redemptions   Spend        % of spend
Tremendous    22            $1,145.00    32.38%   (1,145.00 / 3,536.00)
custom         4            $1,057.00    29.89%   (1,057.00 / 3,536.00)
Snappy        11            $834.00      23.59%   (  834.00 / 3,536.00)
TangoCard     11            $500.00      14.14%   (  500.00 / 3,536.00)
Total         48            $3,536.00    100.00%

Provider shares sum to 100.00% (32.38 + 29.89 + 23.59 + 14.14). Count check: 22 + 4 + 11 + 11 = 48.

TOP 5 COUNTRIES BY REDEMPTIONS (2026-08)
1. US  31
2. CA   7
3-6. GB 2, SG 2, AU 2, NL 2 (four-way tie)
7-8. CH 1, IN 1

A strict top 5 cannot be stated without an arbitrary tiebreak: ranks 3 through 6 are tied at 2 redemptions each (GB, SG, AU, NL). Any three of those four would fill ranks 3-5. Counts reconcile: 31 + 7 + (4 x 2) + (2 x 1) = 48 = total redemptions.

DATA NOTES / GAPS
- No September 2026 rows exist; the September redemption figures are missing from this file.
- Two redeemers had 2 redemptions in August (U-0D807F, U-4A0FE6); all other August redeemers had 1 — this is why 48 redemptions map to 46 redeemers.
- No company-level redemption breakdown was requested; company_alias values (e.g. C-0CB2C1B4, C-0ACCDBC3) are present in the file if that cut is needed.
customer-success 5 tests
churn-save-eligibilityPASS 1.00
Plain-English question: Which at-risk accounts actually qualify for a churn-save offer, and which play fits each?
Prompt — what the model saw
Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0091 · 62s · in 21,727 / out 9,613 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

Plain-English answer: Only accounts meeting all three rules qualify — e.g. C-0F6C0F34 at $49,707 (executive touch) and C-0B827671 at $25,365 (usage revival) — with amount and play per account.
The answer in one line: n_accounts = 30; snapshot_date = 2026-09-05
n_accounts30
eligibleaccount_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
non_eligible_at_riskaccount_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
ruleshealth_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
snapshot_date2026-09-05
total_amount_at_stake224601.00
graded checkseligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Eligibility applied: R1 health_score <60; R2 churn_save_eligible_amount >0; R3 renewal_date on or before 2027-01-03 (2026-09-05 + 120 days).

Eight accounts qualify. “At stake” is the churn-save eligible amount; ARR is shown for context.

Account      | Health | Renewal    | Days | ARR      | At stake | Play                 | Signal justifying play
C-0B0F1BAB   | 38     | 2026-09-23 | 18   | 15,391   | 5,494    | Executive touch      | champion_active=false; health 38; 18 days to renewal
C-0E9C27D1   | 39     | 2026-09-24 | 19   | 75,093   | 41,235   | Commercial concession| health 39; 19 days to renewal; usage flat; 134/157 seats used; champion active
C-0F6C0F34   | 51     | 2026-10-03 | 28   | 86,741   | 49,707   | Executive touch      | champion_active=false; 28 days to renewal; usage growing
C-0B360C78   | 57     | 2026-10-28 | 53   | 60,427   | 35,748   | Commercial concession| health 57; 53 days to renewal; usage growing; 246/327 seats used; champion active
C-0D3278C7   | 54     | 2026-11-12 | 68   | 33,815   | 17,602   | Usage revival        | usage_trend_3m=declining; 126/380 seats used (33.2%)
C-0B827671   | 56     | 2026-11-14 | 70   | 72,088   | 25,365   | Usage revival        | usage_trend_3m=declining; 113/202 seats used (55.9%)
C-0CEF69FD   | 53     | 2026-11-21 | 77   | 79,324   | 32,621   | Executive touch      | champion_active=false; 77 days to renewal; usage growing
C-0CA21961   | 58     | 2026-12-28 | 114  | 31,501   | 16,829   | Usage revival        | usage flat; only 84/325 seats used (25.8%)

Total at stake:
49,707 + 25,365 + 35,748 + 5,494 + 16,829 + 41,235 + 32,621 + 17,602 = $224,601

Full ARR represented: 86,741 + 72,088 + 60,427 + 15,391 + 31,501 + 75,093 + 79,324 + 33,815 = $454,380.

Play routing is not defined in eligibility_rules.csv. I used this explicit precedence: champion_active=false → executive touch; otherwise declining usage or seat utilization below 50% → usage revival; otherwise → commercial concession.

At-risk accounts that do not qualify:

Account      | Health | Renewal    | Days | At stake | Rule failed | Reason
C-0BC71BDD   | 55     | 2026-10-27 | 52   | 0        | R2          | churn_save_eligible_amount is $0.00 despite 29.9% seat utilization and inactive champion
C-0BE96399   | 54     | 2026-10-29 | 54   | 0        | R2          | $0.00 eligible amount; usage declining and only 43/154 seats used
C-10A56B0F   | 54     | 2026-12-12 | 98   | 0        | R2          | $0.00 eligible amount; usage declining and 85/176 seats used
C-0F876796   | 47     | 2027-02-06 | 154  | 19,958   | R3          | Renewal is 154 days out, outside the 120-day window
C-0F6694C3   | 43     | 2027-03-21 | 197  | 0        | R2, R3      | $0.00 eligible amount and renewal is 197 days out
C-0BA71F12   | 52     | 2027-04-11 | 218  | 6,824    | R3          | Renewal is 218 days out, outside the window
C-0FCCD2DF   | 43     | 2027-04-23 | 230  | 0        | R2, R3      | $0.00 eligible amount and renewal is 230 days out

The remaining 15 accounts have health_score ≥60, so they are not at risk under R1; all also have churn_save_eligible_amount of $0.00.
expansion-kitPASS 1.00
Plain-English question: Build the expansion kit for the customer who just replied to our outreach.
Prompt — what the model saw
A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0043 · 29s · in 21,040 / out 1,722 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

Plain-English answer: 37.5% seat coverage, 250 headroom seats worth $15,000 ARR at $60 a seat, usage up 43.2% in six months — and the replier cannot buy; Dana R. (VP People) is the right buyer.
The answer in one line: headroom_seats = 250; coverage_pct = 37.5; company_alias = C-0DDFC9A7 (+3 more below)
headroom_seats250
coverage_pct37.5
company_aliasC-0DDFC9A7
headroom_arr15000.00
per_seat_annual_rate60.00
replierMaria S.
replier_can_buy0
right_buyername=Dana R.; title=VP People
usage_factsactive_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84
usage_health_linesMonthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checkscoverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
EXPANSION KIT — C-0DDFC9A7

SEAT COVERAGE
- Licensed: 150 seats vs. headcount 400 → 37.5% of the company is licensed (150 ÷ 400).
- Active users 2026-08: 126 → 84.0% of licensed seats in use (126 ÷ 150), 31.5% of headcount (126 ÷ 400).
- Implied per-seat rate: $60.00/seat/yr ($9,000 ÷ 150) = $5.00/user/mo. Assumption: flat per-seat pricing; the file gives only aggregate ARR, so rate is derived, not quoted.

USAGE HEALTH (two lines)
- MAU climbed 88 → 126 from March to August 2026: +38 users, +43.2%, averaging +7.4% month over month (95→102→110→118→126), with growth decelerating only slightly (7.95% → 6.78%).
- At 84% of licensed seats already active and steady ~7% monthly growth, the 150-seat cap is the binding constraint: continuing the trailing average implies ~135 MAU by September, i.e. the license ceiling arrives within roughly one to two months.

HEADROOM
- Unused licensed seats: 24 (150 − 126) = $1,440/yr of paid capacity not yet activated.
- Real expansion: 250 seats to cover full headcount (400 − 150).
  - 250 × $60 = $15,000 incremental ARR.
  - $9,000 + $15,000 = $24,000 ARR at 400 seats, a 2.7x uplift.

WHO REPLIED / CAN THEY BUY
- Maria S., People Operations Coordinator, replied 2026-09-02. She cannot buy: she states directly that budget and seat expansion sit with Dana R., VP People, and offers to make an introduction. She is a strong internal advocate — confirms the team "genuinely loves Bonusly" and that MAU keeps climbing. Note her language matches the data; no contradiction to reconcile.

RIGHT BUYER
- Dana R., VP People — the only contact with stated budget authority over seat expansion. Caveat from the data: Dana's last_engaged is 2026-05-18, 107 days before Maria's reply, so she is cold in our records. Maria says Dana "has been asking about our usage numbers," which is a live signal but her own words, not our CRM record. Sam K. (Office Manager, last engaged 2025-11-03) is not a path to budget.

REPLY EMAIL (to Maria, cc none; 128 words)

Subject: RE: Growing your team's recognition program

Hi Maria,

Thanks for the quick reply — and for the kind words about the program. That's good to hear.

An introduction to Dana would be genuinely useful. Our numbers back up what you're seeing: monthly active users went from 88 in March to 126 in August, about 43% growth, and that's 84% of your 150 licensed seats. At that pace you'll be nudging the seat ceiling within a month or two, which is really a capacity conversation, not a pitch.

If it's easier, I'm happy to send Dana a short note on the usage trend first so she has context before we talk. Whatever works best on your side — and thank you for pointing me in the right direction.

Best,
Cole

DATA GAPS
- No contact emails for Dana R. or Maria S. in the provided files; the email is written for Maria's inbox, not addressed with an address.
- No contract term, renewal date, or seat-tier pricing supplied — the $60/seat rate and $15,000 headroom are derived from the single ARR figure and should be confirmed before quoting.
- No meeting notes or activity history for Dana R. beyond a last-engaged date.
onboarding-statusPASS 1.00
Plain-English question: What is done and what is missing in this customer's onboarding before the mid-onboarding call?
Prompt — what the model saw
Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0035 · 26s · in 17,011 / out 1,534 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

Plain-English answer: 25 days live: Slack connected, allowance set, admins added — but the HRIS integration is not connected, with early engagement signals and three call priorities.
The answer in one line: days_live = 25; company_alias = C-0D284E42
days_live25
call_itemsConnect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
checklist{"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
company_aliasC-0D284E42
engagement_signalsavg_active_givers_week1=4.3; avg_active_givers_last_week=13; latest_active_givers=15; days_with_activity=25
graded checkscomplete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
C-0D284E42 — mid-onboarding state as of last data point 2026-09-04 (signup 2026-08-11, 25 days elapsed).

COMPLETE (data-backed)
- Slack integration connected: integration_slack = 2026-08-12 (day 2 after signup, 1 day after signup date).
- Allowance set: allowance_set = 2026-08-13 (day 3).
- Admins added: admins_added = 2.
- First recognition given: first_recognition_at = 2026-08-15 14:22 (day 5).

NOT COMPLETE (no field shows it)
- HRIS integration: integration_hris is blank — not connected.
- First redemption: first_redemption_at is blank — no redemption recorded in the file.
- Allowance amount/currency/recurrence: no field in the file. "Set" is shown; the value is not.
- Admin coverage: no employee-count or seat field, so 2 admins cannot be sized against headcount.
- Usage file carries no company_alias column; mapping onboarding_usage.csv rows to C-0D284E42 assumes it is this account's usage (only account in the account file).

EARLY ENGAGEMENT SIGNALS (active_givers, 2026-08-11 → 2026-09-04, 25 days)
- Week 1 (8/11–8/17): 30 giver-days ÷ 7 = 4.29 avg.
- Week 2 (8/18–8/24): 53 ÷ 7 = 7.57 avg.
- Week 3 (8/25–8/31): 75 ÷ 7 = 10.71 avg.
- Week 4 (9/1–9/04): 56 ÷ 4 = 14.00 avg.
- Total 214 giver-days ÷ 25 days = 8.56 avg.
- First day 3 → last day 15 = +12 givers (+400.0%). First-5-day avg 3.8 → last-5-day avg 13.4 (3.5x).
- Trend is monotonic across all four weeks, no plateau or drop-off week. Peak of 15 reached on 9/3 and held 9/4.
- Recognition is spreading but the file shows zero redemption against rising giving — no points have left the account per data available.

THREE THINGS TO COVER ON THE CALL
1. Redemption gap: first recognition on 2026-08-15 14:22 and no first_redemption_at 20 days later. Confirm employees know how to redeem and that the rewards catalog/currency is visible.
2. HRIS connection: integration_hris is empty. Decide whether to connect it now and at what cost/effort, since user provisioning is the usual blocker to widening giver coverage beyond 15 active givers.
3. Admin coverage and allowance value: only 2 admins and no recorded allowance amount. Confirm who else needs admin rights and verify the allowance figure is correct, then set the next milestone (first redemption, plus giver count as a share of total employees — headcount is missing from the data, so ask for it).
renewal-risk-conflicting-datesPASS 1.00
Plain-English question: Two systems disagree on renewal dates — which do you trust per account, and who is actually renewing inside 90 days?
Prompt — what the model saw
Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0098 · 48s · in 25,719 / out 9,692 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

Plain-English answer: Chargebee wins per account (multi-year contracts are known wrong in ChurnZero), with each trusted date, the reason, and a 90-day-window flag.
The answer in one line: n_accounts = 20; n_disagreements = 5; snapshot_date = 2026-09-05 (+2 more below)
n_accounts20
n_disagreements5
accounts20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
arr_at_risk_90d359409.00
disagreementsaccount_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
snapshot_date2026-09-05
total_arr_renewing_90d1048715.00
trust_rulemulti-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
window2026-09-05 to 2026-12-04
graded checkstotal_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-DAY RENEWAL RISK BRIEF
As-of date: 2026-10-19. Window: 2026-10-19 through 2027-01-17.
Sources: churnzero_renewals.csv, chargebee_terms.csv, usage_12m.csv. No other data used.

METHOD AND ARITHMETIC
- Renewal date rule: Chargebee is the billing system of record. Where ChurnZero and Chargebee agree (all 15 twelve-month accounts), no choice is needed. Where they disagree (all 5 multi-year accounts), I use Chargebee, because ChurnZero renewal dates are known to be wrong on multi-year contracts — every one of those 5 mismatches is a multi-year term (24 or 36 months), which is exactly the failure mode described.
- Seat utilization = seats_used / seats (both from churnzero_renewals.csv).
- 3-month usage trend = active_users 2026-08 vs 2026-06 (last three months of available data), percent change. 12-month change shown as supplemental context (2026-08 vs 2025-09).
- Risk thresholds: High = 3-month decline of 10% or worse, OR seat utilization under 40%. Medium = 3-month decline 3% to 10%, OR utilization 40% to 70%. Low = flat/growing usage AND utilization 70% or better.
- Data limits: usage_12m.csv ends 2026-08. No September or October usage exists in the files, so 16 to 34 days of post-renewal behavior on the already-due accounts is unobserved. Also, seats_used and active_users are not the same measure and do not reconcile (e.g. C-0B20DB64 shows 294 active users against 214 seats_used; C-0B7D2C30 shows 84 active against 274 seats_used). I have not adjusted either field.

HEADLINE
Only 10 of the 20 renewals fall inside the forward 90-day window. The other 10 already crossed their renewal date before 2026-10-19 — and all 6 High-risk accounts are in that already-due group. Every renewal still ahead of us is Low risk except one.

RENEWALS ALREADY DUE (10 accounts, $532,598 ARR)

1. C-0B7D2C30 — CSM Dana Mercer — ARR $65,901.00 — date used 2026-09-15 (Chargebee, 36-month multi-year) — utilization 274/476 = 57.6% — trend 97 → 84 = -13.4% (12-mo -45.8%) — HIGH. Usage has fallen 13.4% in three months and 45.8% over twelve, with 202 seats idle at renewal.
   Disagreement: ChurnZero said 2026-09-10, Chargebee 2026-09-15 (5 days apart). Trust Chargebee, multi-year.

2. C-0BCDB8C2 — CSM Cole Ingram — ARR $54,427.00 — date used 2026-09-18 (Chargebee, 36-month multi-year) — utilization 232/424 = 54.7% — trend 127 → 110 = -13.4% (12-mo -45.0%) — HIGH. Same pattern as above: 13.4% three-month decay, 45% annual, 192 seats unused.
   Disagreement: ChurnZero said 2027-09-18, Chargebee 2026-09-18 — a full year apart. This is the most dangerous mismatch in the set: ChurnZero would have hidden this renewal until 2027. Trust Chargebee, multi-year.

3. C-0D2AB865 — CSM Elena Sinclair — ARR $38,022.00 — date used 2026-09-22 (Chargebee, 24-month multi-year) — utilization 250/407 = 61.4% — trend 125 → 109 = -12.8% (12-mo -45.2%) — HIGH. Three-month decline of 12.8% and a 45.2% annual drop put this in outright contraction.
   Disagreement: ChurnZero said 2026-09-10, Chargebee 2026-09-22 (12 days apart). Trust Chargebee, multi-year.

4. C-0BBE3E60 — CSM Dana Mercer — ARR $30,993.00 — date used 2026-09-26 (Chargebee, 24-month multi-year) — utilization 74/114 = 64.9% — trend 39 → 33 = -15.4% (12-mo -47.6%) — HIGH. Largest three-month decline in the book (-15.4%) on a 24-month contract that has already renewed.
   Disagreement: ChurnZero said 2027-09-26, Chargebee 2026-09-26 — one year apart, again hidden by ChurnZero. Trust Chargebee, multi-year.

5. C-0F5D2323 — CSM Cole Ingram — ARR $90,647.00 — date used 2026-09-29 (Chargebee, 24-month multi-year) — utilization 111/390 = 28.5% — trend 20 → 18 = -10.0% (12-mo -14.3%) — HIGH. Largest single ARR in the brief at 28.5% seat utilization — 279 of 390 seats unused.
   Disagreement: ChurnZero said 2026-09-10, Chargebee 2026-09-29 (19 days apart). Trust Chargebee, multi-year.

6. C-0EC6999D — CSM Elena Sinclair — ARR $79,419.00 — date used 2026-10-03 (both systems agree, 12-month) — utilization 31/112 = 27.7% — trend 17 → 15 = -11.8% (12-mo 0.0%) — HIGH. Second-lowest utilization in the book plus an 11.8% three-month decline on $79,419 of ARR.
   No disagreement: ChurnZero and Chargebee both say 2026-10-03.

7. C-0B20DB64 — CSM Dana Mercer — ARR $21,770.00 — date used 2026-10-07 (both agree, 12-month) — utilization 214/378 = 56.6% — trend 294 → 294 = 0.0% (12-mo +0.3%) — MEDIUM. Stable usage but only 56.6% of seats active, so the renewal is flat and exposed to right-sizing.
   No disagreement.

8. C-0BBC4E7A — CSM Cole Ingram — ARR $56,374.00 — date used 2026-10-10 (both agree, 12-month) — utilization 228/337 = 67.7% — trend 142 → 139 = -2.1% (12-mo -2.1%) — MEDIUM. Flat usage holds this in Medium, but 109 idle seats on $56,374 is the exposure.
   No disagreement.

9. C-0FD551AB — CSM Elena Sinclair — ARR $48,815.00 — date used 2026-10-14 (both agree, 12-month) — utilization 210/376 = 55.9% — trend 123 → 126 = +2.4% (12-mo +1.6%) — MEDIUM. Usage is ticking up, but 44.1% of seats are unused at renewal.
   No disagreement.

10. C-0F9F8F13 — CSM Dana Mercer — ARR $46,230.00 — date used 2026-10-18 (both agree, 12-month) — utilization 199/352 = 56.5% — trend 185 → 182 = -1.6% (12-mo 0.0%) — MEDIUM. Flat, fully-renewing account whose only weakness is 153 unused seats.
   No disagreement.

RENEWALS INSIDE THE 90-DAY WINDOW (10 accounts, $516,117 ARR)

11. C-0BC34584 — CSM Cole Ingram — ARR $16,740.00 — date used 2026-10-22 (both agree, 12-month; 3 days out) — utilization 327/494 = 66.2% — trend 104 → 106 = +1.9% (12-mo +2.9%) — MEDIUM. Smallest ARR in the brief and usage is positive, but utilization sits just under the 70% line with 167 idle seats.
   No disagreement.

12. C-0B7A7546 — CSM Elena Sinclair — ARR $35,062.00 — date used 2026-10-25 (both agree, 12-month; 6 days out) — utilization 182/205 = 88.8% — trend 64 → 63 = -1.6% (12-mo +8.6%) — LOW. Highest seat utilization in the book at 88.8% with usage effectively flat.
   No disagreement.

13. C-0B369871 — CSM Dana Mercer — ARR $85,128.00 — date used 2026-10-29 (both agree, 12-month; 10 days out) — utilization 317/422 = 75.1% — trend 326 → 333 = +2.1% (12-mo +15.2%) — LOW. Growing usage (+15.2% annually) and three-quarters of seats active 10 days before renewal.
   No disagreement.

14. C-0B144C78 — CSM Cole Ingram — ARR $30,899.00 — date used 2026-11-02 (both agree, 12-month) — utilization 169/224 = 75.4% — trend 101 → 106 = +5.0% (12-mo +17.8%) — LOW. Fastest growing account in the book over three months at +5.0%, with utilization above 75%.
   No disagreement.

15. C-0FC4DBB8 — CSM Elena Sinclair — ARR $94,732.00 — date used 2026-11-05 (both agree, 12-month) — utilization 356/464 = 76.7% — trend 189 → 193 = +2.1% (12-mo +14.9%) — LOW. Largest in-window renewal at $94,732, growing 14.9% annually on healthy seat use.
   No disagreement.

16. C-0D5BBE3A — CSM Dana Mercer — ARR $39,740.00 — date used 2026-11-09 (both agree, 12-month) — utilization 85/102 = 83.3% — trend 88 → 91 = +3.4% (12-mo +19.7%) — LOW. Strongest annual growth in the book (+19.7%) on 83.3% seat utilization.
   No disagreement.

17. C-0FB9D5AF — CSM Cole Ingram — ARR $63,158.00 — date used 2026-11-13 (both agree, 12-month) — utilization 144/199 = 72.4% — trend 173 → 176 = +1.7% (12-mo +14.3%) — LOW. Steady growth, 72.4% utilization, no deterioration signal.
   No disagreement.

18. C-0B344485 — CSM Elena Sinclair — ARR $64,384.00 — date used 2026-11-16 (both agree, 12-month) — utilization 224/287 = 78.0% — trend 238 → 244 = +2.5% (12-mo +15.6%) — LOW. Consistent month-over-month growth with 78% of seats in use.
   No disagreement.

19. C-0CB2C1B4 — CSM Dana Mercer — ARR $40,628.00 — date used 2026-11-20 (both agree, 12-month) — utilization 386/473 = 81.6% — trend 47 → 49 = +4.3% (12-mo +14.0%) — LOW. Growing usage at 81.6% utilization; note active users (49) are far below seats_used (386), which is an unresolved measurement gap, not a risk signal I can rate.
   No disagreement.

20. C-22170CA1 — CSM Cole Ingram — ARR $45,646.00 — date used 2026-11-24 (both agree, 12-month) — utilization 251/294 = 85.4% — trend 143 → 146 = +2.1% (12-mo +12.3%) — LOW. High utilization, steady growth, last renewal in the window.
   No disagreement.

DISAGREEMENTS FLAGGED (5 of 20; all multi-year, all resolved to Chargebee)
- C-0B7D2C30: 2026-09-10 vs 2026-09-15 — 5 days. 36-month term.
- C-0BCDB8C2: 2027-09-18 vs 2026-09-18 — 365 days. 36-month term.
- C-0D2AB865: 2026-09-10 vs 2026-09-22 — 12 days. 24-month term.
- C-0BBE3E60: 2027-09-26 vs 2026-09-26 — 365 days. 24-month term.
- C-0F5D2323: 2026-09-10 vs 2026-09-29 — 19 days. 24-month term.
The two $54,427 and $30,993 accounts would have appeared to renew a year from now if ChurnZero were trusted, and both are High risk. No disagreements exist among the 15 twelve-month accounts.

TOTALS
- Total ARR renewing across all 20 accounts: $1,048,715.00
  (65,901 + 54,427 + 38,022 + 30,993 + 90,647 + 79,419 + 21,770 + 56,374 + 48,815 + 46,230 + 16,740 + 35,062 + 85,128 + 30,899 + 94,732 + 39,740 + 63,158 + 64,384 + 40,628 + 45,646)
- Of which already past its renewal date as of 2026-10-19: $532,598.00 (10 accounts)
- Of which renews inside the 90-day window: $516,117.00 (10 accounts)

ARR AT RISK
- High risk: $359,409.00 across 6 accounts (65,901 + 54,427 + 38,022 + 30,993 + 90,647 + 79,419) — all six already crossed their renewal date.
- Medium risk: $189,929.00 across 5 accounts (21,770 + 56,374 + 48,815 + 46,230 + 16,740)
- ARR at risk, defined as High + Medium: $549,338.00 — 52.4% of total ARR renewing.
- ARR at risk that is still ahead of us (High + Medium inside the window): $16,740.00 — C-0BC34584 only.

The forward-looking 90 days are largely clean. The exposure is behind us, in six accounts worth $359,409.00 that went into renewal with 10% to 15% three-month usage decay and, in four of the six, more than 40% of seats idle. Four of those six were understated or pushed a year out by ChurnZero's multi-year dates, so whether they were worked as renewals depends entirely on which system the CSM was reading.
ticket-theme-synthesisPASS 1.00
Plain-English question: What are the real themes in this quarter's support tickets, ignoring the existing tags?
Prompt — what the model saw
Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0051 · 56s · in 9,173 / out 5,322 tok
✓ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

Plain-English answer: 80 tickets in 4-7 themes — points-not-posting leads (20 tickets, 25%, $31,100 ARR affected) — each with counts, distinct accounts, ARR, and a recommendation.
The answer in one line: total_tickets = 80; quarter_window = 2026-06-01 through 2026-08-31
total_tickets80
noteexisting tags are noisy/unreliable; classify by text
quarter_window2026-06-01 through 2026-08-31
ranking_by_arr_exposurehris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
sample_ticket_ids80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
single_account_noise_themebilling_dispute
themes{"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
graded checkstheme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Quarter totals: 80 tickets, 24 accounts, $284,800 ARR in book. All 80 tickets map to 5 text-derived themes (12+18+16+20+14 = 80; shares 15.0+22.5+20.0+25.0+17.5 = 100%). ARR affected per theme = sum of the distinct accounts' ARR, each account counted once — theme sums are $114,000 + $68,800 + $52,000 + $31,100 + $18,900 = $284,800, i.e. no account files in two themes and no double counting.

Ranked by ARR exposure:

1. HRIS provisioning / new-hire sync broken
   12 tickets | 15.0% of volume | 3 accounts | $114,000 ARR (40.0% of book)
   Dates 2026-06-16 → 2026-08-08. Every account in this theme repeats (C-0B2213A9 7 tickets/36,000; C-0DDFC9A7 3/48,000; C-0F6C0F34 2/30,000), and IC-460062 reports sync skipping 12 hires with a clean provisioning log — silent failure, not user error.
   Tickets: IC-460059, IC-460060
   Recommendation: treat as a platform incident, not a ticket queue — instrument provisioning logs to alarm on skipped hires and confirm backfill for the 3 affected accounts before quarter close.

2. Redemption / checkout failures and gift-card delivery
   18 tickets | 22.5% | 7 accounts | $68,800 ARR (24.2%)
   Dates 2026-06-09 → 2026-08-28, no gap, still open at period end. Highest account spread of any failure theme (avg 2.6 tickets/account).
   Tickets: IC-460025, IC-460032
   Recommendation: one checkout-path fix (spin → failed redemption) plus an auto-refund/auto-resend job, since tickets name points deducted without delivery.

3. Billing & invoicing errors (seat count, tier price)
   16 tickets | 20.0% | 1 account | $52,000 ARR (18.3%)
   SINGLE-ACCOUNT: all 16 are C-0E9C27D1, 4 distinct complaint types (tier-price renewal 6, 200-vs-150 seats 5, "third invoice in a row" 3, unapproved seat count 2), 2026-06-16 → 2026-08-29. Repeated identical wording over 10 weeks means prior replies did not resolve it.
   Tickets: IC-460071, IC-460078
   Recommendation: escalate as one named-account billing escalation with a manual invoice audit and credit — volume here is one logo, not a book-wide defect.

4. Points not posting / recognition credits lost
   20 tickets | 25.0% (highest volume) | 9 accounts | $31,100 ARR (10.9%)
   Broadest pattern by account count (avg 2.2 tickets/account), Jun 1 → Aug 25. Text splits into sender-side "delivered but points never arrived" (8) and balance-side "not posting / balance stale" (12) — same ledger symptom reported from both ends.
   Tickets: IC-460004, IC-460016
   Recommendation: reconcile the recognition-write-to-ledger path and publish a self-serve balance-reconciliation check; highest-frequency theme but lowest ARR per account, so fix for deflection, not escalation.

5. Slack integration broken (sync, auth, slash commands)
   14 tickets | 17.5% | 4 accounts | $18,900 ARR (6.6%)
   Most repeat-heavy theme (3.5 tickets/account; C-0BA71F12 alone 6), Jun 4 → Aug 21. Symptoms cluster into token/auth persistence ("re-auth does not stick," "toggle resets itself") and slash-command errors for whole teams.
   Tickets: IC-460041, IC-460046
   Recommendation: stop per-ticket re-auth advice and fix token refresh/persistence — recurring re-auth tickets prove the workaround isn't holding.

Broad vs single-account: themes 1, 2, 4, 5 are multi-account patterns (3–9 accounts each). Theme 3 is single-account concentration and should be handled as an escalation, not a product theme — but it still carries $52,000, so it cannot be dismissed as noise.

On tags: they don't hold. Only 3 of the 15 "billing"-tagged tickets are actually billing (12 are points, redemption, Slack or HRIS), 13 of the 16 billing tickets carry another tag, and 11 failures are tagged "how-to." Ticket counts above come from body text only.
marketing 5 tests
case-study-match0.50
Plain-English question: Which three existing customers are the best case-study matches for this prospect?
Prompt — what the model saw
Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0008 · 24s · in 892 / out 976 tok
✓ top3✗ decoys_excluded✓ cites_fields✗ no_billing

✓ Correct Answer (ground truth)

Plain-English answer: Three ranked matches on industry, size, use case, and region (e.g. C-11C31562), with the higher-scoring decoy explicitly excluded for a mismatched field.
The answer in one line: prospect_alias = C-82AF3719
constraintsno_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
decoys_excludedcompany_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
prospect_aliasC-82AF3719
prospect_fields_usedindustry, size_band, use_case, region
top3company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
graded checkstop3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Ranking (prospect C-82AF3719: Technology / Mid-Market / employee_recognition / NA-West)

Scoring: one point per matching field out of 4; ties broken by whether the use case matches (social proof hinges on the same program type), then industry over region.

1. C-64171065 — 3/4
   Matches: industry=Technology, size_band=Mid-Market, use_case=employee_recognition
   Misses: region (NA-East vs NA-West)
   Arithmetic: 3 of 4 fields matched, including all three non-geo fields. Only eligible record with an exact industry + size + use-case triple.

2. C-11C31562 — 3/4
   Matches: size_band=Mid-Market, use_case=employee_recognition, region=NA-West
   Misses: industry (Manufacturing vs Technology)
   Arithmetic: 3 of 4 fields matched. Same size, same program, same region — the closest geographic/size peer, but a cross-industry reference.

3. C-A13C193D — 3/4
   Matches: industry=Technology, size_band=Mid-Market, region=NA-West
   Misses: use_case (retention vs employee_recognition)
   Arithmetic: 3 of 4 fields matched. Placed third of the ties because the use case differs, which weakens the analogy most for an employee_recognition prospect.

Tie note: three customers scored 3/4. Order follows use-case match first (C-64171065, C-11C31562) over industry/region match without use case (C-A13C193D).

Excluded despite high similarity — no case study (has_case_study=false): C-D6217CAA and C-C153868F (both Technology/Mid-Market/employee_recognition/NA-West, 4/4 on fields) and C-D7CDFD70, C-0BAA9D3E, C-EB10E4AA, C-05B45886, C-37C484FD (lower scores). Not eligible as social proof per your instruction.

Gap: no eligible customer matches all four fields. Billing data and contact names were not used.
channel-attribution-mathPASS 1.00
Plain-English question: Compute cost per SQM, cost per SQO, and pipeline per dollar for every channel over six months.
Prompt — what the model saw
Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0067 · 46s · in 23,290 / out 5,265 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

Plain-English answer: LinkedIn Ads at $960/SQM and $3,000/SQO, organic search at $90,000 pipeline on zero spend, paid social flagged at $18,000 with zero SQMs, and a channel winner named.
The answer in one line: best_pipeline_per_dollar_channel = paid_search
best_pipeline_per_dollar_channelpaid_search
confidence_notewebinars and referral have small SQM samples (<20); treat rates as directional.
flagged_rows_sqo_before_sqmCT-000041, CT-000044
months2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel{"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
reallocation_recommendationfrom=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
undefined_channelspaid_social
graded checksundefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
TRAILING 6 MONTHS: 2026-03 through 2026-08 (all 122 contact rows fall inside the window; no rows excluded for date).

CHANNEL PERFORMANCE

1) paid_search
   Spend: 6,000 x 6 = 36,000
   SQMs: 40 | SQOs: 18
   Cost/SQM: 36,000 / 40 = 900.00
   Cost/SQO: 36,000 / 18 = 2,000.00
   SQM->SQO rate: 18 / 40 = 45.0%
   Pipeline: 720,000 (18 SQOs x 40,000)
   Pipeline per $: 720,000 / 36,000 = 20.00

2) linkedin_ads
   Spend: 4,000 x 6 = 24,000
   SQMs: 25 | SQOs: 8
   Cost/SQM: 24,000 / 25 = 960.00
   Cost/SQO: 24,000 / 8 = 3,000.00
   SQM->SQO rate: 8 / 25 = 32.0%
   Pipeline: 96,000 (8 SQOs x 12,000)
   Pipeline per $: 96,000 / 24,000 = 4.00

3) paid_social
   Spend: 3,000 x 6 = 18,000
   SQMs: 0 | SQOs: 0 | Pipeline: 0
   Cost/SQM, cost/SQO, SQM->SQO rate, pipeline per $: UNDEFINED (spend with zero SQMs; not zero, no denominator)
   Note: no contacts in the first-touch file carry channel = paid_social at all.

4) webinars
   Spend: 1,500 x 6 = 9,000
   SQMs: 12 | SQOs: 5
   Cost/SQM: 9,000 / 12 = 750.00
   Cost/SQO: 9,000 / 5 = 1,800.00
   SQM->SQO rate: 5 / 12 = 41.7%
   Pipeline: 60,000 (5 SQOs x 12,000)
   Pipeline per $: 60,000 / 9,000 = 6.67

ORGANIC (unpaid, no rows in the spend file)

5) organic_search
   Volume (SQMs): 30
   SQOs: 10 | SQO rate: 10 / 30 = 33.3%
   Pipeline: 90,000 (10 SQOs x 9,000)
   Pipeline per SQM: 90,000 / 30 = 3,000.00

6) referral
   Volume (SQMs): 15
   SQOs: 6 | SQO rate: 6 / 15 = 40.0%
   Pipeline: 48,000 (6 SQOs x 8,000)
   Pipeline per SQM: 48,000 / 15 = 3,200.00

Paid total spend 87,000 (36,000 + 24,000 + 18,000 + 9,000). Total first-touch SQMs 122, SQOs 47, pipeline 1,014,000 (720,000 + 96,000 + 60,000 + 90,000 + 48,000).

DATA INTEGRITY FLAGS: SQO date precedes SQM date
   CT-000044 | linkedin_ads | SQM 2026-07-23 | SQO 2026-07-18 | 12,000
   CT-000041 | linkedin_ads | SQM 2026-06-14 | SQO 2026-06-09 | 12,000
   Both are linkedin_ads SQOs (2 of 8, 24,000 of 96,000). If these two are invalid and excluded, linkedin becomes 6 SQOs / 72,000 pipeline: cost/SQO 4,000.00, SQM->SQO 24.0%, pipeline per $ 3.00. Treat linkedin's headline numbers as unconfirmed until attribution is corrected.
   Also flagged as an assumption, not a data error: CT-000007 (paid_search) has SQO = SQM date (2026-03-28), treated as valid (not "precedes").

REALLOCATION RECOMMENDATION
   Shift budget out of linkedin_ads into paid_search and webinars.
   - paid_search is the only channel with both high volume and strong economics: 45.0% SQM->SQO, 2,000 cost/SQO, 20.00 pipeline per dollar — 5x linkedin's 4.00 (and 6.7x if linkedin's flagged rows are excluded).
   - webinars is the cheapest cost/SQO (1,800) at 41.7%, but at 9,000 total spend and 5 SQOs it is spend-constrained, not demand-constrained. It can absorb more before diminishing returns show up.
   - linkedin_ads: 12,000 of its 24,000 spend produced 0 SQOs; the two SQOs it did produce are date-inconsistent. Cut roughly half and hold the rest against a fixed test while attribution is fixed.
   - paid_social: 18,000 spent, zero SQMs across 6 months, no contact-level records. Pause pending tracking, not a performance verdict — the data cannot distinguish "no demand" from "not tagged."
   Concrete shape: move ~6,000 from paid_social and ~8,000 from linkedin_ads per 6-month period into paid_search (a ~39% increase on 36,000) and webinars (a ~156% increase on 9,000, so stage it — take webinars up in increments and watch webinars cost/SQO rather than doubling blind).

CONFIDENCE: LOW-TO-MODERATE.
   SQO counts are thin: paid_search 18, webinars 5, linkedin 8 (6 if flagged rows excluded), organic 10, referral 6. A single SQO moves webinars' cost/SQO by ~300 and its rate by ~8 points, so the paid_search-vs-webinars gap (2,000 vs 1,800 cost/SQO) is not statistically meaningful; the paid_search-vs-linkedin gap (2,000 vs 3,000-4,000) and the paid_social blank are the findings that survive the sample size. Directionally solid, not a basis for large single-step reallocation — move in increments and re-measure after 60-90 days.
competitor-battlecard-updatePASS 1.00
Plain-English question: Update the Rivally battlecard with sourced pricing and where each side wins.
Prompt — what the model saw
Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0027 · 54s · in 2,497 / out 3,599 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

Plain-English answer: Rivally at $7 per user per month (the August source wins over the stale $5, conflict noted), a 13-7 win/loss record, and two unverified old claims flagged.
The answer in one line: competitor = Rivally
citation_ids_available25 items — e.g. S01, S02, S03, S04
competitorRivally
pricingvalue_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
rep_opinion_snippet_ids_not_factsS09, S21
unverified_old_claimsRivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
win_loss_12mowins=13; losses=7
graded checkscompetitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
BATTLECARD: RIVALLY
Prepared 2026-10-19. Sources: competitor_snippets.csv (S01–S25), deals_with_competitor.csv (20 deals). Every claim carries a snippet id; anything unsourced from the old card is flagged.

1. ONE-LINE POSITIONING
Points-based recognition feed that is easy to stand up and is now actively buying EU enterprise credibility; increasingly priced and staffed like an enterprise vendor rather than a mid-market one.
- Points-based recognition feed with engagement strength: S02, S16
- Fast setup / mid-market fit: S04
- EU enterprise push funded and staffed: S11, S15

2. PRICING (newer source wins)
Current list price: $7 per user/month, annual billing required — pricing page, 2026-08-12 (S17).
Superseded: $5 per user/month, annual billing — pricing page, 2026-01-20 (S03), reconfirmed 2026-04-01 (S08).
CONFLICT: S03/S08 vs S17 differ by $2/user/mo. Both are first-party pricing pages; S17 is the newest and wins. The $5 figure is a stale reading, not a current quote.
Observed quoted prices (deal mentions, not list):
- $6.50/user/mo to a 500-seat prospect, annual term — 2026-06-02 (S13). That is 7.1% below today's $7 list ((7 − 6.50) / 7 = 7.1%), and it predates the $7 page update.
- $7/user/mo list with 15% off for a 3-year term — 2026-08-14 (S18). 15% off $7 = $5.95/user/mo effective.
Caveat: S13 ($6.50) is below the arithmetic of S18's discount ($5.95 is lower still), so published list, quoted list, and discount depth do not reconcile across the period. Treat any specific number as quote-specific.
Add-on: Rivally Pulse priced as a separate add-on, not bundled — 2026-09-01 (S23).
Delta since old card: old card's "$5, as of 2026-01" was correct when written (S03) and is now 40% below current list ($5 → $7).

3. WHERE THEY WIN
- EU distributed teams / multi-language: EU enterprise reviewer calls them strong for distributed EU teams and praises multi-language support (S12). Reinforced by EU data residency GA (S15).
- Speed to value: mid-market reviewer — setup under a week, Slack working out of the box (S04).
- Engagement/end-user pull: points-based feed praised as engaging (S02, S16).
- Support responsiveness: under 4-hour response praised (S22).
- Commercial aggression in specific cycles: a prospect was shown list plus a 15% 3-year discount (S18).

4. WHERE WE WIN
- Analytics depth / reporting: 800-seat prospect chose Bonusly over Rivally citing analytics depth (S25). Rivally dashboards described as basic vs enterprise tools (S07).
- Admin and IT scale: no SCIM provisioning, manual user management painful (S10); admin tooling lags peers (S16); no bulk recognition editing (S24).
- Migration leverage and data portability: analytics exports CSV-only, migration off Rivally described as hard (S20).
- EMEA rewards breadth: EMEA catalog thinner than US catalog (S14).
- Win rate: see section 7 — 65% over the last 12 months of recorded competitive deals.

5. OBJECTIONS AND RESPONSES
Objection: "Rivally is cheaper." → Current list is $7/user/mo annual (S17); the $5 figure is a stale page from January/April (S03, S08). Where they have discounted, effective rates have run near $5.95–$6.50 (S13, S18) — price on total cost of admin time and reporting, not seat rate.
Objection: "Rivally is live in a week and works in Slack." → Conceded for basic setup (S04). Shift to what happens after rollout: no SCIM (S10), no bulk recognition editing (S24), admin tooling behind peers (S16).
Objection: "Rivally owns EU enterprise." → Partly true: strong for distributed EU teams with multi-language (S12), EU residency now GA (S15), ex-Workday EMEA VP hired (S11). Counter on analytics and export depth (S07, S20, S25).
Objection: "Their recognition feed is more engaging." → Conceded at the feed level (S02, S16). Pivot to reporting and program measurement, where reviewers say they are basic (S07) and exports are CSV-only (S20).
Objection: "They support us fast." → Not disputed (S22). Do not attack support; compete on admin scale and analytics.

6. RECENT CHANGES (chronological)
- 2026-03-05: launches Rivally Pulse, a lightweight engagement survey add-on (S06).
- 2026-05-09: hires ex-Workday VP EMEA to lead European expansion (S11).
- 2026-07-01: opens Dublin office; EU data residency generally available (S15).
- 2026-08-12: list price raised to $7/user/mo, annual billing required (S17).
- 2026-08-14: prospect quoted $7 list with 15% off for a 3-year term (S18).
- 2026-08-20: Microsoft Teams app v2 in public preview (S19).
- 2026-09-01: Pulse exits beta; sold as add-on, not bundled (S23).
- Funding context: $40M Series C led by Northgate Ventures, announced 2025-11-04 (S01) — outside the recent window but relevant to their expansion capacity.
Not usable as competitor change: AE claims that UI is clunky (S09, explicitly unverified against the product) and that Rivally is discounting aggressively (S21, rep opinion, not confirmed). S21 is directionally consistent with S18 but S18 alone is one quoted offer, not a pricing policy.

7. OUR 12-MONTH WIN/LOSS RECORD
Window: 2025-09 through 2026-08 (the 12 months covered by the deal data). 20 deals, all vs Rivally.
Wins 13, losses 7 → win rate 13 / 20 = 65.0%; loss rate 35.0%.
By month (wins-losses):
  2025-09  1-1 (Deal-7767F5 L, Deal-072E31 W)
  2025-10  2-0 (Deal-A9FD43 W, Deal-F65C8F W)
  2025-11  1-1 (Deal-7AA785 W, Deal-D263E0 L)
  2025-12  1-1 (Deal-44C524 W, Deal-935746 L)
  2026-01  2-0 (Deal-0D0CD6 W, Deal-E46EAB W)
  2026-02  2-0 (Deal-D5B790 W, Deal-1D2392 W)
  2026-03  1-1 (Deal-5C636E W, Deal-9066A6 L)
  2026-04  0-2 (Deal-5645A5 L, Deal-72A02F L)
  2026-05  0-1 (Deal-C6FFAA L)
  2026-06  1-0 (Deal-67BE14 W)
  2026-07  1-0 (Deal-1B6969 W)
  2026-08  1-0 (Deal-F03E7B W)
Shape: 10-3 through 2026-03 (10/13 = 76.9%), then 0-3 in 2026-04/2026-05, then 3-0 in the last three recorded months (2026-06 to 2026-08). Last six months are 4-4 (50.0%). The April–May losing streak coincides with Rivally's EU build-out (S11, S15) and precedes the August price increase (S17) — sequence only; the data does not establish cause.
Win/loss reasons are recorded for one deal only (Deal-… alias not given; 800-seat prospect chose Bonusly on analytics depth, S25). No loss reasons are documented for any of the 7 losses.

8. OLD CARD DISPOSITION
- Positioning "points-based recognition for mid-market": re-sourced (S02, S04) but incomplete — understates the EU enterprise motion (S11, S12, S15).
- Pricing "$5/user/mo, as of 2026-01": was accurate (S03) and is now superseded by $7 (S17).
- "Rivally lacks a Slack integration": FALSE / strike. Slack integration worked out of the box (S04). No snippet supports the old claim.
- "Rivally was acquired by WorkHuman in 2025": UNVERIFIED. No snippet in the provided data mentions any acquisition. Do not use until sourced. The only funding/ownership event in the data is a $40M Series C (S01).
- "Strong in EU enterprise with multi-language support": re-sourced (S12), strengthened by S15.
nurture-sequence-diagnosisPASS 1.00
Plain-English question: What is broken in each email sequence — rates, overlaps, tracking errors?
Prompt — what the model saw
Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0142 · 58s · in 32,677 / out 15,043 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

Plain-English answer: Per-sequence open, reply, and meeting rates with the weakest step (all three weakest at step 3), an opened-above-sent tracking error, and audience overlap flagged.
The answer in one line: fix_priority_first = Cold Outbound - HR Leaders
all_pairwise_overlaps['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
audience_overlap_pairCold Outbound - HR Leaders, Cold Outbound - People Ops
failure_modeCold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_firstCold Outbound - HR Leaders
per_sequence{"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_errorsequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
under_2pct_replyCold Outbound - HR Leaders
graded checkstracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
SEQUENCE METRICS (rates on sent; meeting rate = meetings/sent)

New Logo Nurture — 1,386 sent (500+458+428); open 490/1386 = 35.4%; reply 90/1386 = 6.49%; meetings 27/1386 = 1.95%. Weakest step: 3 (reply 18/428 = 4.21%, vs 8.40% and 6.55%).

Expansion Nurture — 875 sent; open 565/875 = 64.6% (invalid, see below); reply 59/875 = 6.74%; meetings 12/875 = 1.37%. Weakest step: 3 (12/275 = 4.36%, below step 2's 8.33%).

Cold Outbound - HR Leaders — 1,785 sent (600+595+590); open 545/1785 = 30.5%; reply 8/1785 = 0.45%; meetings 0/1785 = 0.00%. Weakest step: 3 (1/590 = 0.17%); step 1 already only 0.83%.

Cold Outbound - People Ops — 1,163 sent (400+386+377); open 340/1163 = 29.2%; reply 29/1163 = 2.49%; meetings 6/1163 = 0.52%. Weakest step: 3 (6/377 = 1.59%).

TRACKING ERROR
Expansion Nurture step 2: opened 340 > sent 300 = 113.3%. Impossible; opens are double-counted or dates misaligned. The sequence-level 64.6% open rate is unusable. Excluding step 2, opens are 225/575 = 39.1%. Meeting/reply for Expansion (20.3%) is depressed by step 2's inflated reply base (25/340 open).

AUDIENCE OVERLAP
23 contacts sit in two sequences (940 unique contacts, 963 rows): 21 shared between Cold Outbound - HR Leaders and Cold Outbound - People Ops (CT-001217, CT-001159, CT-001153, CT-001227, CT-000884, CT-000849, CT-001285, CT-001101, CT-001255, CT-001105, CT-001097, CT-001258, CT-000908, CT-001130, CT-001103, CT-001033, CT-001345, CT-001311, CT-000890, CT-001277, CT-001236); 2 between New Logo Nurture and Expansion Nurture (CT-000301, CT-000624). No duplicate rows within a sequence.

FAILURE MODE UNDER 2% REPLY
Only Cold Outbound - HR Leaders (0.45%). Opens are healthy (30.5%, matching People Ops' 29.2%), so the list and subject lines work; the offer does not. Reply-per-open is 1.5% for HR Leaders vs 8.5% for People Ops, and 0 meetings from 8 replies. Failure is message-target mismatch with the HR-leader persona, not deliverability or list quality.

FIXES AND ORDER
1. Cold Outbound - HR Leaders first: 1,785 sends (34% of volume), 0.45% reply, zero meetings. Change: replace the generic step 1 offer with one HR-leader-specific proof point (compliance/benefits-budget outcome) and a single low-friction ask.
2. People Ops: cut step 3 (1.59% reply, 1 meeting from 377 sends).
3. New Logo: replace step 3's ask with one low-friction CTA (4.21% reply vs 8.40% at step 1).
4. Expansion: repair step 2 open tracking, then refresh step 3's value prop.

Excluded: meeting data beyond counts is absent; no send windows or copy supplied.
weekly-marketing-goals-rollupPASS 1.00
Plain-English question: Are we on pace on the quarter's marketing goals?
Prompt — what the model saw
Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0014 · 39s · in 1,391 / out 1,862 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

Plain-English answer: SQMs ahead of pace, SQOs on pace, DS2s behind (35 short of expected at day 66 of 92), with deltas and same-quarter closes.
The answer in one line: days_in_quarter = 92; days_elapsed = 66; coverage_ratio_active_pipeline = 0.75; elapsed_fraction = 0.7174 (+1 more below)
days_in_quarter92
coverage_ratio_active_pipeline0.75
days_elapsed66
elapsed_fraction0.7174
metrics{"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
pace_ruleexpected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required1
graded checksmetrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 pace basis: 66 of 92 days elapsed = 0.7174 of the quarter. Pace line = target × 0.7174. 26 days remain.

METRIC                 QTD ACTUAL   TARGET    DELTA         PACE LINE   VS PACE      STATUS
SQMs                   230          300       -70 (76.7%)   215.2       +14.8 (+6.9%)  AHEAD
SQOs                   84           120       -36 (70.0%)   86.1        -2.1 (-2.4%)   BEHIND (marginal)
DS2s                   40           75        -35 (53.3%)   53.8        -13.8 (-25.7%) BEHIND
closed_lost_mia rate   20.0% (5/25) 10.0%     +10.0 pp      n/a         n/a            BEHIND (2.0x ceiling)
same_quarter_closes    10           20        -10 (50.0%)   14.3        -4.3 (-30.3%)  BEHIND
active_pipeline        3,000,000    4,000,000 -1,000,000 (75.0%) 2,869,565 +130,435 (+4.5%) AHEAD

Arithmetic
- Pace fraction: 66 / 92 = 0.7174.
- SQMs: 300 × 0.7174 = 215.2; 230 − 215.2 = +14.8.
- SQOs: 120 × 0.7174 = 86.1; 84 − 86.1 = −2.1.
- DS2s: 75 × 0.7174 = 53.8; 40 − 53.8 = −13.8.
- closed_lost_mia rate: 5 / 25 = 0.200; 0.200 − 0.100 = +0.100 (lower_better, so 10 pp over ceiling).
- same_quarter_closes: 20 × 0.7174 = 14.3; 10 − 14.3 = −4.3.
- active_pipeline: 4,000,000 × 0.7174 = 2,869,565; 3,000,000 − 2,869,565 = +130,435.

Run-rate check for the 26 days left (remaining gap ÷ 26 vs. actual per-day to date):
- SQMs: 70 / 26 = 2.69/day needed vs 3.48/day achieved (230/66). On track for 300.
- SQOs: 36 / 26 = 1.38/day vs 1.27/day achieved. Needs ~9% acceleration.
- DS2s: 35 / 26 = 1.35/day vs 0.61/day achieved. Needs ~2.2x.
- same_quarter_closes: 10 / 26 = 0.38/day vs 0.15/day achieved. Needs ~2.5x.
- active_pipeline: 1,000,000 / 26 = 38,462/day vs 45,455/day added (3,000,000/66). On track for 4,000,000.
- closed_lost_mia rate: to land at 10.0% with 25 closed-lost deals already booked, no further closed-lost deal may be MIA; each additional MIA deal raises the rate (26 deals → 19.2%, 30 → 16.7%), so the ceiling is only recoverable by closing more deals without new MIA losses.

What moved this week: not available. The only file provided is MTD/QTD cumulative (marketing_qtd.csv); no weekly, daily, or prior-week snapshot, no deal-level rows, and no channel/source breakdown. Any statement about this week's movement would be invented, so I'm not making one. To produce it, send a prior-week snapshot of the same six metrics (for a week-over-week delta) or a dated activity extract.

Data caveats: targets.csv supplies no target for a same-quarter close rate or pipeline coverage ratio, so coverage is reported in dollars only (3,000,000 / 4,000,000 = 0.75x); DS2 is not defined in the provided files, so it is reported as given with no conversion.
communication 5 tests
ceo-slack-compressionPASS 1.00
Plain-English question: Compress the forecast analysis into one Slack message the CEO can read in ten seconds.
Prompt — what the model saw
Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0036 · 22s · in 20,358 / out 854 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

Plain-English answer: Three sentences, under 80 words, decision first, every distinct number kept plus the one caveat — all checked by deterministic text rules.
The answer in one line: weighted_number = 115976; must_keep = data-quality caveat — deleting the bad news fails
weighted_number115,976
limitmax 3 sentences, max 80 words, no em-dashes
must_keepdata-quality caveat — deleting the bad news fails
graded checksmax_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Don't sign off on the $115,977 Q3 forecast ($44,729 COMMIT plus 35% of $203,565 = $71,248): owners are blank on 85 of 86 deals and no why-buy exists on any of the 7 COMMIT deals, so the committed $44,729 is undocumented. 32 deals worth $227,575 are excluded for post-9/30 close dates, including COMMIT Deal-D348E1 at $13,770 dated 2026-10-15. Fix owner and why-buy fields before forecasting again.
stalled-deal-followup-emailPASS 1.00
Plain-English question: Write a follow-up email for this stalled deal — under 120 words, one prior touch, one ask.
Prompt — what the model saw
Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0037 · 26s · in 18,984 / out 1,262 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

Plain-English answer: An email that references the July 28 milestone-awards demo, makes one clear ask, and invents nothing — Deal-0D2F7A, 31 days since last engagement.
The answer in one line: amount = 5100; word_cap = 120; days_since_last_engagement = 31; deal_alias = Deal-0D2F7A (+4 more below)
amount5,100
days_since_last_engagement31
word_cap120
allowed_facts14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_newENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
deal_aliasDeal-0D2F7A
must_referenceengagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
ownerAlex Franklin
single_cta1
stageDS3
tonewarm, concise, professional (matches prior thread)
graded checksword_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Following up on 150-seat pricing

Hi,

I sent a recap of our July 28 demo with 150-seat pricing on August 5 and haven't heard back since. I don't want to assume where this sits internally.

What stood out was the People team's reaction to automated milestone awards and the points catalog.

One ask: do you have 20 minutes in the next two weeks to confirm whether 150 seats is still the right scope, or should I close the loop for now?

Thanks,
Alex Franklin
funnel-digest-paragraphPASS 1.00
Plain-English question: Write the weekly GTM digest — one paragraph per team, every number traceable.
Prompt — what the model saw
Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0013 · 27s · in 319 / out 1,927 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

Plain-English answer: Four paragraphs (Marketing, Sales, CS, PLG) using only the allowed numbers, no contact names, no performance commentary, under 400 words.
The answer in one line: word_cap = 400
word_cap400
allowed_numbers16 items — e.g. 2, 3, 9, 08
bannedcustomer contact names, employee performance commentary, compensation
required_paragraphsMarketing, Sales, CS, PLG
graded checksfour_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing: SQMs landed at 46 against a 52 target — 6 short and 88% of plan (46 ÷ 52 = 0.885). Webinar carried 18 of the 46, or 39% of the week's total (18 ÷ 46 = 0.391). Cost per SQM on paid search held flat at $150. Read on that combination: the miss is volume, not efficiency — the same dollar bought SQMs at the same rate, there were just fewer of them.

Sales: 14 SQOs converted, 9 DS2 meetings set, $310,000 in new pipeline created, 3 same-quarter closes booked. Average pipeline per SQO is about $22,100 ($310,000 ÷ 14 = $22,143), but the extract does not confirm that all $310,000 traces to this week's 14 SQOs, so treat that as a rough ratio rather than a clean per-SQO figure. No pipeline target or coverage ratio was provided, so attainment on the pipeline number cannot be assessed. The 3 closes are the only booked outcome in the extract.

CS: 2 renewals saved this week, team NPS at 61, and 3 open red-flag accounts heading into next week. No prior-week NPS was provided, so 61 is a level, not a trend — direction is unknown. Renewal dollars were not given either, so the 2 saves cannot be sized; they are a count only.

PLG: 412 new signups, with activation at 31% — that implies roughly 128 activated users (412 × 0.31 = 127.7). Separately, 38 companies reached the aha moment of 10 recognition gives. Those two figures are not the same unit: signups are users, aha is companies, so no company-level aha rate can be computed from this extract. Prior-week signups were not provided, so week-over-week movement is unknown.

Missing across the set: prior-week comparisons for SQMs, NPS, and signups, plus any pipeline target, renewal value, or company-level denominator. Nothing above is trended because the extracts give single-week levels only.
partner-digest-sectionPASS 1.00
Plain-English question: Write the partner digest so every partner in the roster appears exactly once.
Prompt — what the model saw
Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0006 · 24s · in 770 / out 671 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

Plain-English answer: Four active partners with status and numbers, four quiet partners with one line each, and partner-sourced pipeline that reconciles to the deal data.
The answer in one line: rule = every roster partner named exactly once; quiet partners get one line; no partners outside roster.
active_no_dealsWorkWell Group
partner_sourced_pipelinecount=5; amount=550000
per_partner_pipeline{"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
quiet_partnersRecogniq, KudosWave, PeopleFirst Advisors, TotalPerk
rosterpartner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
ruleevery roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checksall_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
PARTNER DIGEST — 2026-08-24 to 2026-09-04

ACTIVE PARTNERS

Apex Rewards Co — Active. 2 partner-sourced deals, $275,000 total (Deal-DDAAF2 $180,000 + Deal-2CF33E $95,000), both UTM Source = Partner, both in DS1. Co-webinar locked for 09-15.

HRCloud Partners — Active. 1 partner-sourced deal, $140,000 (Deal-F1CDA5), UTM Source = Partner. Sourced opp cleared security review and moved to DS2.

CultureBridge — Active. 2 partner-sourced deals, $135,000 total (Deal-096E1D $60,000 + Deal-067213 $75,000), UTM Source = Partner. Both early stage, from the lunch-and-learn.

WorkWell Group — Active, no sourced deals this period. Wants to restart the joint playbook in Q4; planning call booked 09-09.

QUIET PARTNERS

Recogniq — Quiet, no activity reported this period.
KudosWave — Quiet, no activity reported this period.
PeopleFirst Advisors — Quiet, no activity reported this period.
TotalPerk — Quiet, no activity reported this period.

PARTNER-SOURCED PIPELINE (period)

Count: 5 deals
Amount: $550,000

Arithmetic: Apex $180,000 + $95,000 = $275,000; HRCloud $140,000; CultureBridge $60,000 + $75,000 = $135,000. $275,000 + $140,000 = $415,000; $415,000 + $135,000 = $550,000. Deal rows in partner_deals.csv: 5, all UTM Source = Partner, all matched to a roster partner.

Nothing omitted: 8 roster partners listed (4 active, 4 quiet); 5 deal rows accounted for. Nothing added: no deal, partner, or figure outside the provided files.
rfp-answer-from-policyPASS 1.00
Plain-English question: Answer the 10 RFP items from the policy excerpts only — no guessing.
Prompt — what the model saw
Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0007 · 28s · in 768 / out 934 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

Plain-English answer: Seven answered with excerpt citations (Q1-Q7), three flagged unanswerable with the document that would answer them (Q8-Q10).
The answer in one line: answered_count = 7; rule = unanswerable items must be flagged and the document that would answer them named; do not guess.
answered_count7
answer_map{"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
ruleunanswerable items must be flagged and the document that would answer them named; do not guess.
unanswerableQ8, Q9, Q10
valid_citation_idsP1, P2, P3, P4, P5, P6, P7, P8
graded checksanswered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
**Q1 — Encryption at rest**
Yes. All customer data is encrypted at rest using AES-256. [P1]

**Q2 — Encryption in transit**
Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

**Q3 — SOC 2 report**
Yes. Bonusly maintains a SOC 2 Type II report, renewed annually, and it is available to prospects under NDA. [P2]

**Q4 — Data retention after termination**
Customer data is retained for the duration of the contract plus 90 days; after that point it is deleted from production systems. Total retention cannot be stated in calendar terms here because the contract term itself is not specified in the excerpts — the formula is contract term + 90 days. [P3]
Separately, encrypted backups are retained 35 days per [P6]; the excerpts do not state whether backup expiry is aligned to the P3 deletion timeline.

**Q5 — Subprocessor list and change notification**
The current list is maintained at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

**Q6 — SAML single sign-on**
Yes. SAML 2.0 SSO — along with SCIM user provisioning — is available on the Enterprise plan. [P5] Both are plan-gated; the excerpts do not state availability on other tiers.

**Q7 — Backup schedule and retention**
Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

**Q8 — ISO 27001**
Unanswerable from these excerpts — no ISO 27001 certification or attestation is referenced anywhere in P1–P8. The document that would answer this is the compliance certifications section of the trust portal (trust.bonusly.com) or a current ISO 27001 certificate issued to Bonusly by an accredited registrar.

**Q9 — Contractual uptime SLA**
Unanswerable from these excerpts — no uptime commitment or SLA percentage appears in P1–P8. The document that would answer this is the Master Services Agreement or the SLA addendum/Order Form executed with the customer.

**Q10 — HIPAA BAA**
Unanswerable from these excerpts — no statement about HIPAA or Business Associate Agreements appears in P1–P8. The document that would answer this is Bonusly's standard BAA template, or a legal/compliance written confirmation of willingness to sign; that is a contractual commitment and cannot be inferred from the security policy excerpts.

Summary of coverage: 7 of 10 answered (Q1–Q7), 3 unanswerable (Q8, Q9, Q10).
ops-maintenance 5 tests
skill-file-reconciliationPASS 1.00
Plain-English question: Audit a skill set against its manifest and flag every conflict with a severity and an action.
Prompt — what the model saw
Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.0194 · 50s · in 85,513 / out 11,017 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

Plain-English answer: A severity-ranked list of TRIM_DESC / UPDATE_BODY / DELETE_SKILL / MERGE / REVIEW actions covering every rule violation found.
The answer in one line: files_audited = 14; rank = CRITICAL / WARNING / INFO with one fix action per issue
files_audited14
findingsduplicate trigger, circular delegation chain, dangling file reference, oversized description, hardcoded values, manifest drift
rankCRITICAL / WARNING / INFO with one fix action per issue
graded checksduplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
RECONCILIATION — 14 manifest rows, 14 SKILL.md files supplied, all 14 names matched.

────────────────────────────────────────
1) OVERLAPPING / DUPLICATED ALWAYS-TRIGGER PHRASES
────────────────────────────────────────

FINDING 1.1 — CRITICAL — MERGE
Skill pair: comms-drafter ↔ email-drafter.
Both descriptions claim the same trigger surface in near-identical wording. Verbatim duplicates across the two descriptions: "write me an email", "draft a follow-up", "what should I say", "bump email", "contract nudge", plus "help me reply" / "help me reply to this". Both also enumerate the same body types in the same order (outbound prospecting, follow-ups, post-demo recaps, pricing/contract follow-up, end-of-quarter push, renewal/expansion, QBR follow-up, onboarding), and comms-drafter's scope ("any external communication… Intercom and support ticket responses, partner outreach") is a strict superset of email-drafter's ("customer-facing email of any kind"). Two skills fire on one utterance, so routing is non-deterministic.
Proposal: collapse email-drafter into comms-drafter as the single external-comms entry point, retaining email-drafter only as a signature-retrieval sub-routine invoked by comms-drafter.

FINDING 1.2 — WARNING — TRIM_DESC
Skill trio: pipeline-intelligence-report ↔ weekly-pipeline-report ↔ next-to-close.
pipeline-intelligence-report triggers on "pipeline update", "pipeline review", "what's the pipeline look like"; weekly-pipeline-report triggers on "run the pipeline update", "update the pipeline", "what does pipeline look like". That is a one-word difference on the same utterance, and both assert exclusivity ("Master pipeline scoring skill — never answer pipeline questions inline"; "ALWAYS trigger when the user says 'run the pipeline update'"). next-to-close adds "what's closing this week" adjacent to weekly's "give me this week's numbers".
Proposal: give each skill a disjoint trigger token (e.g. scored/tiered → pipeline-intelligence-report, weekly cadence → weekly-pipeline-report, shortlist → next-to-close) and delete the shared "pipeline update"/"pipeline report" phrases from all but one.

FINDING 1.3 — INFO — TRIM_DESC
signalforge-claim-compressor claims "ALWAYS trigger for: … pipeline updates …" — the same phrase the two pipeline report skills claim as their entry point, on a skill that is a post-processing pass, not a report.
Proposal: replace its report-name trigger list with a positional trigger ("runs after analysis-validator on any SignalForge analytical output").

────────────────────────────────────────
2) CIRCULAR DELEGATION CHAIN
────────────────────────────────────────

FINDING 2.1 — WARNING — UPDATE_BODY
Cycle: email-drafter → deal-strategy-coach → email-drafter.
Evidence: email-drafter carries a lane marker sending strategic asks to deal-strategy-coach; deal-strategy-coach's "Manager-to-prospect email frameworks" section sends manager-email drafting to email-drafter ("use the email-drafter skill which automatically retrieves your Gmail signature"). Each skill treats the other as the terminal step for the same request, so a "draft a manager email for a stalled deal" ask can ping-pong indefinitely. comms-drafter is an inbound feeder into the same loop (comms-drafter → deal-strategy-coach → email-drafter → deal-strategy-coach).
Proposal: declare deal-strategy-coach terminal for anything it diagnoses (it already produces "Rep-ready output"), and change email-drafter's lane marker to a one-way pointer that does not re-enter.

────────────────────────────────────────
3) DANGLING DELEGATION TARGETS (no manifest row, no file)
────────────────────────────────────────

FINDING 3.1 — CRITICAL — REVIEW
bonusly-brand. Declared mandatory gate by three skills: comms-drafter ("Before drafting any communication, apply the bonusly-brand skill"), email-drafter ("All emails must reflect Bonusly's brand voice… apply the bonusly-brand org skill"), sales-forecast ("Always reference bonusly-brand skill for full voice, color, and typography guidance"). Not in the manifest.
Proposal: either supply the skill file and add the manifest row, or inline the brand pillars/tone anchors into comms-drafter.

FINDING 3.2 — CRITICAL — REVIEW
prospect-research-multithreading. Blocks drafting when recipient is unresolved: comms-drafter ("If any of those are missing… invoke prospect-research-multithreading in Contact Lookup mode first"), email-drafter (same, plus "surface the second person to CC"), deal-strategy-coach (cross-skill handoff section, five trigger conditions). Not in the manifest.
Proposal: register it in the manifest with a declared_name, or downgrade all three references to "ask the user for the contact" until it exists.

FINDING 3.3 — WARNING — REVIEW
Analysis-validator's specialist delegation table (§12.4) names eight skills with no manifest row and no file: bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, bonusly-datadog-questions. The validator instructs "always delegate to specialist skill" for product/rewards/legal/flag claims — that path currently terminates in nothing.
Proposal: add all eight as manifest rows, or mark the table as external-to-bundle so the validator falls back to HOLD instead of delegating.

FINDING 3.4 — WARNING — REVIEW
signalforge-reports. Required read in pipeline-intelligence-report ("MANDATORY PRE-BUILD STEPS: 1. Read /mnt/skills/organization/signalforge-reports/SKILL.md…") and weekly-pipeline-report ("Read the SignalForge design system (/mnt/skills/organization/signalforge-reports/SKILL.md and DESIGN-SYSTEM.md)"). No manifest row, and the /mnt/skills/… path does not exist in this bundle.
Proposal: add signalforge-reports plus its DESIGN-SYSTEM.md and signalforge.css as bundle assets, or repoint both skills at an inline style block.

FINDING 3.5 — WARNING — REVIEW
skill-orchestrator. Referenced as a cascading file in analysis-validator §11 and as the registration authority in signalforge-feedback's activation checklist ("Skill registered in skill-orchestrator as a terminal step"). No manifest row.
Proposal: add the manifest row, or rewrite both references as optional.

FINDING 3.6 — INFO — REVIEW
Non-skill file dependencies also absent: CUSTOMER_DATA_REFERENCE, HUBSPOT_CONNECTOR_REFERENCE, SIGNALFORGE_PRODUCT_INSIGHT_SKILL (analysis-validator §11); references/data-sources.md, references/report-structure.md, references/cadence.md, references/report-template.html, references/TEMPLATE_README.md (sales-forecast); references/report-spec.md, references/queries.md (weekly-pipeline-report); references/html-spec.md (pipeline-intelligence-report, self-declared deprecated).
Proposal: list these as bundle assets in the manifest's file column so absence is detectable at load time.

────────────────────────────────────────
4) VERSION CONFLICTS
────────────────────────────────────────

FINDING 4.1 — WARNING — UPDATE_BODY
analysis-validator is self-inconsistent at v3.6. Frontmatter says "**Version:** 3.6", footer says "analysis-validator v3.6", changelog top row is 3.6 — but the Validation Trail template in §7 still renders "Validator: analysis-validator v3.2". Downstream pipeline-intelligence-report cites "Analysis Validator v3.6", confirming 3.6 as the live version. Survivor: v3.6. Loser: the v3.2 string in the §7 trail block.
Proposal: change the trail template's version string to v3.6 (single-line edit; no logic change).

FINDING 4.2 — WARNING — UPDATE_BODY
analysis-validator gate ranges are stale in three places. §2 correctly defines G1-A…G1-L and G2-A…G2-F, but §1 Full Mode says "Gate 1: All Data Correctness checks (G1-A through G1-H)" and "Gate 2: All Output Quality checks (G2-A through G2-E)", and §6's decision tree says "Run Gate 2 (G2-A through G2-E)" while the §7 trail renders G2-F. G1-I through G1-L and G2-F were added in 3.1/3.5/3.6 without updating the executive summaries. Survivor: the §2 enumeration (the operative definition).
Proposal: normalise all three references to G1-A–G1-L and G2-A–G2-F.

FINDING 4.3 — WARNING — UPDATE_BODY
sales-forecast declares itself quarter-agnostic in changelog 1.1 ("Quarter-agnostic (Q2 → current quarter throughout)") but the body still hardcodes the abandoned generation: Step 1A heading "HubSpot: Open Q2 Deals", Step 2E/Step 3 Tab 6 "Q2 Narrative". Survivor: v1.1's stated intent (current-quarter computed at STEP 0).
Proposal: replace the two remaining "Q2" literals with the STEP 0 quarter variable.

FINDING 4.4 — WARNING — UPDATE_BODY
model-selection's registry is version-locked to 2026-05-19 and enforces its own rule: "If today's date is more than 14 days past the `last_checked` date below, run the self-update procedure before making any recommendations." Today is 2026-10-19, i.e. 153 days past last_checked, so by its own rule every recommendation it makes now is invalid. Survivor: the self-update rule; the yaml block is the stale artifact.
Proposal: add a hard precondition line — if last_checked is >14 days old, output "registry stale, self-update required" and refuse to annotate — so the conflict cannot pass silently.

────────────────────────────────────────
5) MANIFEST DESCRIPTIONS EXCEEDING 1,024 CHARACTERS
────────────────────────────────────────

Arithmetic from the manifest's description_chars column (14 rows):
656, 897, 996, 792, 965, 676, 945, 1004, 1006, 962, 1006, 708, 762, 656
max = 1006 (pipeline-intelligence-report, signalforge-claim-compressor)
1006 − 1024 = −18
rows > 1024 = 0 of 14 (0%)

FINDING 5.1 — INFO — REVIEW
Reported count: 0. Caveat: the declared counts are not credible against the frontmatter supplied. comms-drafter's description alone (single paragraph, "Communications writer for everyone at Bonusly… this skill diagnoses.") visibly runs well past 996, and pipeline-intelligence-report's description names 8 signals, 6 tiers, 10 tabs and a master-skill claim in ~1,400 characters of prose, against a declared 1006. The manifest column appears stale or truncated, so "0" is the manifest's answer, not a safe one.
Proposal: regenerate description_chars by measuring the frontmatter block of each file, then re-run this check before trusting it.

────────────────────────────────────────
6) HARDCODED PAGE IDS, DATES, PERSON NAMES IN BODIES
────────────────────────────────────────

FINDING 6.1 — CRITICAL — UPDATE_BODY
Person names inside gate logic. analysis-validator G1-K HOLD output hardcodes "Escalate to Finance (Manish or Amani)." A validator that forbids hardcoded benchmarks makes a routing decision on two individuals; it breaks on any role change. §10 repeats "Flag for Finance (Manish or Amani)". Survivor: role-based escalation.
Proposal: replace both with a role reference (Finance owner of record) resolved at run time.

FINDING 6.2 — CRITICAL — UPDATE_BODY
Hardcoded GTM roster with owner IDs inside a validator. analysis-validator §12.3 is a dated roster ("Updated May 4, 2026") of 16 people with HubSpot owner IDs, used by G2-F ID resolution as ground truth; pipeline-intelligence-report separately hardcodes five AE names and owner IDs ("verified May 2026": Bryce Harmon 119337721, Dana Mercer 83155923, Cole Ingram 83155924, Alex Franklin 84342457, Gavin Porter 1520255671). Two copies of the same roster, both dated, both disagreeing in scope (6 Core AEs vs 5 AEs).
Proposal: move the roster to one lookup (owners connector / §12.3) and have pipeline-intelligence-report reference it rather than restating it.

FINDING 6.3 — WARNING — UPDATE_BODY
Person names as routing rules in deal-strategy-coach ICP: ".edu domains are not auto-disqualified — routed to Farid for manual qualification" and "India (routed to Perseus)".
Proposal: replace with a role/queue reference.

FINDING 6.4 — WARNING — UPDATE_BODY
Person name in a skill title. weekly-pipeline-report H1 reads "# Weekly Pipeline Report — Ben Lavin · Demand Generation · Bonusly", and the body refers to "Ben's review"/"deliver the HTML file to Ben" five times. A skill bound to one named reviewer cannot be run by anyone else.
Proposal: parameterise the reviewer at run time and remove the name from the title.

FINDING 6.5 — WARNING — UPDATE_BODY
Hardcoded Confluence/Slack/Drive identifiers (page or routing breakage on any space change):
  partner-digest — cloudId 73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f, spaceId 1958248479, folder ID 2286616609, reference page 2286321666, Slack user U03QLMBL7AR, plus named contacts "Kelli", "Jen Lee", "Hani", "Bryce", "Sara" and a dated canonical issue "May 16, 2026".
  signalforge-feedback — page ID 2295136266, space 2232811524, parent 2234417154, Build Log 2247295002, cloudId as above.
  sales-forecast — space ID 2232811524, cloudId, parent page 2232582148.
  stale-pipeline-report — Slack channel C0561C1JCPJ, support owner ID 55483190, HubSpot org 1973303.
  weekly-pipeline-report — Google Sheet IDs 1CLZeOsElVDF_LF0ZG_t2nfwvhnZ6bpwqM_nX3WEYzcw and 1ENuaEcCuLjdKhMvp8FK3Ys1ek5Aw9ZuOZhsHJJFoB_k.
  deal-strategy-coach — playbook page 2257879045.
Proposal: hoist all destination IDs into a single config block per skill, leaving the prose to reference the block.

FINDING 6.6 — WARNING — UPDATE_BODY
Dated snapshots that will silently rot:
  weekly-pipeline-report — "Business days complete in Q2 (April 1 – June 30, 2026; total ≈ 64–65)" and "Q1 2026 context (static): Sales Bookings Actual $365,152 vs $475,000 plan (77%); Pipeline Addition Actual $2,490,532 vs $3,288,000 forecast (76%)". Today is 2026-10-19 (Q4), so the window is two quarters stale and the static comparators are frozen.
  analysis-validator — "Expected ranges (as of May 2026)", "Population anchors… ~452,000 / ~110,097", §12.3 "Updated May 4, 2026", §13.3 "confirmed current as of May 4, 2026".
  closed-lost-analysis — "30-deal AI-field sample from May 2026", "field confirmed May 2026", "MinIO: rep vacation May 4–12".
  pipeline-intelligence-report — frontmatter "v6 · May 2026", "AE owner IDs (verified May 2026)".
  model-selection — "last_checked: 2026-05-19".
  partner-digest / stale-pipeline-report / signalforge-claim-compressor / sales-forecast — changelog dates 2026-05-09 to 2026-06-10.
Proposal: replace every dated constant with a run-time query or a `last_verified` field the skill checks and refuses to run on once expired.

FINDING 6.7 — INFO — UPDATE_BODY
Company names used as evidence inside taxonomy examples (not a defect, but they are customer names in a policy file): closed-lost-analysis cites Softheon, Estee Lauder, LIFTOFF, Nestlé, Ozinga, MinIO, Aurora Innovation, GCash, Ethos Cannabis, StickerYou, Panopto, Schneider Downs; deal-strategy-coach pricing section cites no customers but claim-compressor's examples cite "Felix Construction" ($15K TCV).
Proposal: confirm these are cleared for internal tooling, or replace with anonymised aliases.

────────────────────────────────────────
7) MANIFEST DRIFT — BOTH DIRECTIONS
────────────────────────────────────────

FINDING 7.1 — INFO — REVIEW (files with no manifest row)
Count: 0. All 14 supplied files appear in the manifest and every frontmatter `name` equals its declared_name and its filename:
analysis-validator, closed-lost-analysis, comms-drafter, deal-strategy-coach, email-drafter, model-selection, next-to-close, partner-digest, pipeline-intelligence-report, sales-forecast, signalforge-claim-compressor, signalforge-feedback, stale-pipeline-report, weekly-pipeline-report — 14/14 matched, 0 orphans.
Proposal: none.

FINDING 7.2 — INFO — REVIEW (manifest rows with no file)
Count: 0 of 14. Every row resolved to a supplied file.
Proposal: none.

FINDING 7.3 — WARNING — REVIEW (drift by omission)
The manifest is complete against the files supplied but incomplete as an inventory of the bundle's actual dependency surface. 12 skills are invoked by supplied bodies with neither a manifest row nor a file: bonusly-brand, prospect-research-multithreading, signalforge-reports, skill-orchestrator, bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, bonusly-datadog-questions (detail in findings 3.1–3.5). Against a 14-row manifest that is 12 unresolved edges.
Proposal: extend the manifest schema with a `depends_on` column and add the 12 missing rows so drift is measurable on both axes, not just filename presence.

FINDING 7.4 — INFO — REVIEW
body_lines cannot be verified. The manifest carries a body_lines value per row (106 to 1429), but the supplied artifacts are rendered skill bodies with no line-count ground truth, and pipeline-intelligence-report's 536 is implausible for the body text shown. Name-level drift is confirmed clean; size drift is unverified.
Proposal: regenerate body_lines from the files at manifest build time and treat any delta >10% as drift.
runbook-from-slack-threadPASS 1.00
Plain-English question: Turn the resolved incident thread into a runbook.
Prompt — what the model saw
Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0018 · 46s · in 948 / out 2,615 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

Plain-English answer: Ordered steps from the thread, each with the exact command, who ran it, how success was verified, and rollback where state changed.
The answer in one line: rule = no steps beyond what the thread contains; every step traces to a message id.
ambiguous_steporder=4; msg=M05; mark=needs-confirmation
ruleno steps beyond what the thread contains; every step traces to a message id.
stepsorder=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
graded checksordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
RUNBOOK — reward-worker queue backlog (#incident-2026-08-29-reward-queue)
Derived only from the incident thread, M01–M10. Each step cites its message.

STEP 1 — Acknowledge alert, assume IC
  Who: Bryce Harmon [M01], 14:02:10Z
  Action taken: acknowledged PagerDuty alert (reward-worker queue depth > 10k), took Incident Commander role.
  Exact command: none recorded in thread.
  Success verified: not stated in thread — needs confirmation (no ack confirmation or PagerDuty state cited).
  Rollback: n/a, no infrastructure state changed.

STEP 2 — Measure queue depth (baseline diagnostic, read-only)
  Who: Farid Osman [M02], 14:04:33Z
  Command: bundle exec rake sidekiq:queue_depth
  Result / success verified: output showed reward queue at 48,213 pending jobs, against a normal of under 500. 48,213 / 500 ≈ 96x the stated normal ceiling.
  Rollback: n/a, read-only.

STEP 3 — Inspect dead set (read-only)
  Who: Farid Osman [M03], 14:06:02Z
  Reported finding: dead set holds 112 jobs, all Redis::TimeoutError, originating from around 13:58.
  Exact command: not recorded in thread — needs confirmation (M02's rake task covered queue depth only).
  Success verified: n/a, inspection step; the 112-job count is the only stated evidence.
  Rollback: n/a, read-only.

STEP 4 — Pause recognition enqueue to stop the bleed (STATE CHANGE)
  Who: Farid Osman [M04], 14:08:45Z
  Command: bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'
  (Command is recorded as bin/rails runner, not bundle exec — reproduced exactly as written in M04.)
  Success verified: no confirmation in thread that the flag actually registered as disabled — needs confirmation. The only later supporting evidence is the falling depth at [M07].
  Rollback (as given verbatim in M04): bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'

STEP 5 — Clear the dead set (STATE CHANGE — under-specified)
  Who: Elena Sinclair [M05], 14:15:20Z
  Stated action: "While I was in the console I cleared out the dead set."
  Exact command: NOT recorded — needs confirmation. The thread does not say what "cleared" means operationally (discarded vs. requeued, which tool/console, which queue).
  Success verified: NOT recorded — needs confirmation. No post-clear dead-set count appears (the 112 figure is pre-clear, from [M03]; [M08] checks queue depth, not the dead set).
  Rollback: NOT recorded — needs confirmation. No restore or requeue path is stated for the 112 discarded jobs.
  Note: this is the weakest step in the thread and the only state change with no command, no verification and no rollback.

STEP 6 — Scale reward-worker up (STATE CHANGE)
  Who: Bryce Harmon [M06], 14:21:07Z
  Command: kubectl scale deployment/reward-worker --replicas=6 (prior value 3)
  Success verified: not stated at this step — needs confirmation. No pod-ready or replica-count confirmation is cited; the nearest evidence is queue depth at [M07] (9,400) and zero at [M08].
  Rollback: kubectl scale deployment/reward-worker --replicas=3

STEP 7 — Track drain rate (read-only)
  Who: Farid Osman [M07], 14:33:41Z
  Reported: queue depth down to 9,400 and falling ~1,200/min.
  Exact command: not restated in M07 — needs confirmation (presumably the [M02] rake task; thread does not say).

STEP 8 — Verify backlog cleared and errors normal (read-only verification gate)
  Who: Cole Ingram [M08], 14:47:55Z
  Command: bundle exec rake sidekiq:queue_depth
  Success verified: returned 0; Datadog error rate back to baseline.
  Rollback: n/a, read-only.

STEP 9 — Re-enable recognition enqueue (STATE CHANGE)
  Who: Bryce Harmon [M09], 14:49:10Z
  Command: bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'
  Success verified: 40 new jobs processed cleanly in the following 3 minutes.
  Rollback: inverse command given in [M04] — bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'. M09 does not restate a rollback; this is carried over from M04, not asserted at M09.

STEP 10 — Scale reward-worker back down (STATE CHANGE)
  Who: Bryce Harmon [M10], 14:55:00Z
  Command: kubectl scale deployment/reward-worker --replicas=3
  Success verified: queue stable at 0.
  Rollback: kubectl scale deployment/reward-worker --replicas=6 (inverse of [M06]; not restated at M10).

ORDER OF OPERATIONS AS ACTUALLY RUN
  M01 ack → M02 measure → M03 inspect dead set → M04 disable enqueue → M05 clear dead set → M06 scale up → M07 track drain → M08 verify zero → M09 enable enqueue → M10 scale down.
  Elapsed: 14:02:10Z to 14:55:00Z = 52m50s.

NEEDS CONFIRMATION (thread gaps — nothing added)
  - [M01] No alert-acknowledgement or PagerDuty-state evidence.
  - [M03] Command used to read the dead set is not recorded.
  - [M04] No explicit verification that the feature flag went to disabled.
  - [M05] No command, no verification, no rollback for clearing the dead set; "cleared" is ambiguous (discard vs. requeue) and the fate of the 112 Redis::TimeoutError jobs is unstated.
  - [M06] No explicit confirmation that 6 replicas came up healthy.
  - [M07] Command for the drain measurement is not restated.
  - [M09], [M10] Rollbacks are inferred from earlier messages; the thread does not state them at those steps.
  - Root cause: the thread contains no root-cause step. Redis::TimeoutError from ~13:58 [M03] is the only causal signal offered and is not established as cause.
  - No follow-up/action items, owner for root cause, or monitoring change are recorded before resolution [M10].
datadog-incident-timelinePASS 1.00
Plain-English question: Sequence the failure from the logs — what broke first, and what cascaded after it?
Prompt — what the model saw
From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0049 · 30s · in 20,182 / out 3,064 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

Plain-English answer: A reward-service Redis timeout at 14:01:12Z cascades to sidekiq at 14:01:40 and api-gateway at 14:03:05, with the Datadog query that confirms the first error.
The answer in one line: job_class_involved = RewardGiveJob
cascade_orderservice=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
clean_servicespostgres
confirming_queryservice:reward-service status:error "@message:Redis::TimeoutError"
first_errortimestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
job_class_involvedRewardGiveJob
logs_do_not_showroot cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
graded checksfirst_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
FAILURE SEQUENCE (all times 2026-09-03 UTC, from datadog_logs.csv unless noted)

FIRST ERROR
  14:01:12Z  service: reward-service  level: ERROR
  "Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s"

  Preceding healthy state: last reward-service INFO before it is 13:59:30Z "job enqueued".
  Blind gap: 13:59:30 -> 14:01:12 = 1m42s (102s) with zero reward-service log lines.
  So the true onset is bounded, not pinpointed: somewhere in 13:59:30–14:01:12.

CASCADE, IN ORDER

  1. 14:01:12Z  reward-service  ERROR  root timeout to redis-primary:6379 (5s threshold breached)
  2. 14:01:20Z  reward-service  ERROR  "retry exhausted for RewardGiveJob"   (+8s)
  3. 14:01:30Z  reward-service  ERROR  "retry exhausted for RewardGiveJob"   (+10s)
  4. 14:01:40Z  reward-service  ERROR  "retry exhausted for RewardGiveJob"   (+10s)
  5. 14:01:40Z  sidekiq        ERROR  "RewardGiveJob failed: Redis::TimeoutError; retrying in 60s"
     sidekiq_jobs.csv first failure: J-00005 at 14:01:46Z (+34s after first error)
     J-00001..J-00006 fail 14:01:46–14:01:57Z, all Redis::TimeoutError
  6. 14:02:28Z  sidekiq        ERROR  "RewardGiveJob failed: Redis::TimeoutError; retrying"  (+48s)
  7. 14:02:30Z  sidekiq        WARN   "Queue reward depth above 10,000"  (+78s)
     J-00007..J-00012 fail 14:02:51–14:02:58Z
  8. 14:02:36Z  sidekiq_jobs   J-00013 RecognitionDigestJob fails — second job class hit, same Redis error
     J-00013..J-00016 fail 14:02:36–14:05:50Z
  9. 14:03:05Z  api-gateway    ERROR  "502 upstream timeout calling reward-service /gives"
     latency to edge: 14:03:05 - 14:01:12 = 1m53s (113s)
 10. 14:03:30Z  web-app        ERROR  "Give form submission failed: upstream 502 from api-gateway"  (+25s)
 11. 14:03:31Z  sidekiq        ERROR  RewardGiveJob retry fails
 12. 14:03:48Z  api-gateway    502 upstream timeout calling reward-service
 13. 14:04:13Z  api-gateway    502
 14. 14:04:22Z  sidekiq        ERROR  RewardGiveJob retry fails
 15. 14:04:45Z  web-app        Give form submission failed: upstream 502
 16. 14:05:16Z  api-gateway    502
 17. 14:05:26Z  sidekiq        ERROR  RewardGiveJob retry fails
 18. 14:05:42Z  web-app        Give form submission failed: upstream 502
 19. 14:06:47Z  sidekiq        ERROR  RewardGiveJob retry fails
 20. 14:06:49Z  web-app        Give form submission failed: upstream 502
 21. 14:06:52Z  api-gateway    502
     (then 4m04s silent: 14:06:52 -> 14:10:56)
 22. 14:10:56–14:20:59Z  postgres INFO "checkpoint complete" x6 — routine, no error state
 23. 14:22:10Z  reward-service INFO "Redis connection restored; resuming job processing"
 24. 14:24:45Z  sidekiq        INFO  "Queue reward depth below 500"

SERVICE AND JOB
  Originating service: reward-service (only service emitting the Redis error).
  Failing dependency: Redis at redis-primary:6379 (no logs from that host in the slice).
  Jobs: RewardGiveJob (primary, 12 failures in sidekiq_jobs.csv: J-00001–J-00012)
        RecognitionDigestJob (secondary, 4 failures: J-00013–J-00016)
  Edge/consumer services affected: api-gateway, web-app.

TOTALS (counted from the slice)
  Error lines by service, 14:01:12–14:06:52:
    reward-service 4 (1 timeout + 3 retry-exhausted)
    sidekiq 6 retry failures
    api-gateway 5 x 502
    web-app 4 x Give form failure
    total 19 ERROR lines
  sidekiq_jobs.csv rows: 16 (12 RewardGiveJob + 4 RecognitionDigestJob = 16)
  Impact window to Redis recovery: 14:22:10 - 14:01:12 = 20m58s (1,258s)
  Impact window to queue drain: 14:24:45 - 14:01:12 = 23m33s (1,413s)

DATADOG QUERY TO CONFIRM THE FIRST ERROR

  service:reward-service status:error "Redis::TimeoutError" @timestamp:[2026-09-03T13:55:00Z TO 2026-09-03T14:05:00Z]
  — sort ascending by timestamp, top hit should be 2026-09-03T14:01:12Z.

  Narrowing query for the connection-level failure itself:
  service:reward-service status:error "redis-primary:6379"

  Both use only true values present in the slice (service name, log status, exact message substring). I am not asserting any attribute name such as @redis.host — that field is not shown in the data, so I have not written it into the query.

WHAT THE LOGS DO NOT SHOW
  - No Redis server-side telemetry at all: redis-primary emits zero lines. No cause for the 5s timeout (CPU, memory, maxclients exhaustion, fork/RDB save, network path, GC, latency spike).
  - No supporting metrics: no Redis connection counts, command latency, or host/pod resource stats; no trace IDs or span data linking the 14:01:12 timeout to specific failed gives.
  - No deploy, config change, or scale event near 13:59–14:01, so a trigger cannot be attributed.
  - No infrastructure identity: no pod, node, region, or instance IDs, so blast radius per host is unknown.
  - No customer/user impact numbers: count of gives actually lost versus eventually retried successfully (to completion) is absent. "Queue depth below 500" at 14:24:45 shows drain, not success.
  - RecognitionDigestJob failures appear ONLY in sidekiq_jobs.csv (J-00013–J-00016). The datadog slice contains no log line naming that job class.
  - Accounting mismatch: 3 "retry exhausted for RewardGiveJob" log lines at 14:01:20/30/40 versus 6 RewardGiveJob rows (J-00001–J-00006) at 14:01:46–14:01:57. The three earliest retry-exhausted log entries have no matching rows in sidekiq_jobs.csv, and the earliest job row (14:01:46) postdates the first log error by 34s. The two sources do not reconcile on job identity or on first-failure time.
  - 4m04s log gap 14:06:52–14:10:56: state of the system in that window is unknown.
  - No api-gateway or web-app recovery confirmation lines; their return to normal is inferred from silence, not stated.
  - Postgres "checkpoint complete" lines (6, 14:10:56–14:20:59) are INFO and routine; the slice establishes no causal link between postgres and the Redis outage.
feature-flag-state-summaryPASS 1.00
Plain-English question: Summarize what each feature flag controls and who has it on.
Prompt — what the model saw
Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0041 · 26s · in 20,826 / out 1,563 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: Recognition streaks on for 42 beta companies, redeem-flow redesign off with a 12-company target list, plus any flag with no code reference or no targets.
The answer in one line: rule = company counts must equal the export; do not describe flags absent from the export.
code_referencedrecognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
flags{"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
flags_with_no_targetspaused_offboard_cleanup
orphan_flags_no_code_referencelegacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
rulecompany counts must equal the export; do not describe flags absent from the export.
graded checksorphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
FLAG-BY-FLAG (9 flags in export, 6 referenced in code)

1. recognition_streaks_v2 — on, segment:beta_companies, 42 companies
   Code: app/models/recognition.rb. Gates StreakTracker.record(give) behind a per-company flag check; when enabled, streak tracking is recorded on each give.

2. points_budget_guardrails — on, all_companies, 220 companies
   Code: app/services/budget_service.rb. Gates BudgetService#enforce!(giver, points); when enabled, point budgets are enforced at give time.
   Note: 220 is simultaneously the largest count and the "all_companies" rule — the export treats the whole book as 220 companies. No separate universe count is given to cross-check against.

3. slack_dm_nudges — on, segment:region_na, 87 companies
   Code: app/jobs/nudge_job.rb. Early-returns unless enabled, so the SlackDm.send_nudge(user) call only runs for enabled companies. Segment is region-based, not plan- or tier-based.

4. redeem_flow_redesign — off, targeted_list, 12 companies
   Code: app/controllers/redeem_controller.rb. Binary branch: RedeemV2Component when enabled, RedeemV1Component otherwise. Because state is off, all 12 targeted companies currently render V1; the target list is staged but inactive.

5. analytics_dashboard_v3 — on, segment:tier_three, 65 companies
   Code: app/controllers/analytics_controller.rb. When enabled, @dashboard is built by AnalyticsV3; the excerpt shows no else-branch, so for the other companies the assignment simply does not happen in this code path.

6. ms_teams_app_v2 — off, targeted_list, 9 companies
   Code: app/services/teams_installer.rb. Gates TeamsAppV2.install(company). With state off, the install call is inert for all 9 targeted companies — zero effective exposure despite having targets.

7. legacy_give_modal — off, segment:legacy_plan, 14 companies
   No code reference in the excerpt. Target and count exist in the export, but nothing in app/ or lib/ as provided reads this key. Cannot state what it controls.

8. survey_boosters_q3 — on, segment:legacy_plan, 7 companies
   No code reference in the excerpt. State is on and 7 companies are targeted, so it is live against a segment with no visible consumer in the supplied code. Cannot state what it controls.

9. paused_offboard_cleanup — off, (empty targeting_rules), 0 companies
   No code reference and no targets. Empty targeting rule plus a zero count: not reachable by any company and not referenced in code.

SEGMENTS / TARGETING SUMMARY
Targeting rule types in the export: one all_companies (220), four named segments (beta_companies 42, region_na 87, tier_three 65, legacy_plan 14+7=21 across two flags), two targeted_list (redeem_flow_redesign 12, ms_teams_app_v2 9), one empty (0).
legacy_plan appears on two flags carrying 21 company-row counts combined; no company-level data is provided, so overlap between those 14 and 7 cannot be determined.

ARITHMETIC CHECK
Per-flag counts: 42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0
  42 + 220 = 262; +87 = 349; +12 = 361; +65 = 426; +9 = 435; +14 = 449; +7 = 456; +0 = 456.
On-state: 42 + 220 + 87 + 65 + 7 = 421.
Off-state: 12 + 9 + 14 + 0 = 35.
421 + 35 = 456 = row total. Consistent.

Flags with a code reference: 6 of 9. Flags without: 3 (legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup).
Flags with zero companies targeted: 1 (paused_offboard_cleanup).
Flags on but unreachable in code: 1 (survey_boosters_q3).
Flags with targets but inert due to state off: 2 (redeem_flow_redesign, ms_teams_app_v2).

MISSING DATA
No company names or IDs are in the export — only aggregate counts — so I cannot name which companies sit in any segment or targeted list. Segment definitions (what beta_companies, region_na, tier_three, legacy_plan actually contain) are not provided. The excerpt is partial (app/ and lib/ only, and likely not exhaustive), so absence of a code reference is evidence only for these supplied files, not proof of an orphaned flag.
nda-triagePASS 1.00
Plain-English question: Triage each NDA green, yellow, or red — and quote the clause that decides it.
Prompt — what the model saw
Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0032 · 23s · in 18,890 / out 552 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

Plain-English answer: NDA-1 green (standard mutual terms), NDA-2 yellow for an embedded 18-month non-solicit, with the driving clause quoted for every classification.
The answer in one line: non_solicit_in = NDA-2
checksclassification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
classifications{"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_inNDA-2
graded checksgreen · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 [PARTY A] — GREEN
Clause 3: "information that (a) is or becomes publicly available through no breach, (b) was known prior to disclosure, (c) is received from a third party without duty of confidence, (d) is independently developed, or (e) must be disclosed by law or court order."
Reasoning: fully mutual, complete five-part carve-outs, Delaware law, and Clause 5 expressly disclaims license/obligation/exclusivity — nothing to negotiate.

NDA-2 [PARTY B] — YELLOW
Clause 4: "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."
Reasoning: an embedded non-solicit (3-year term + 18 months = 4.5 years of restriction) with no standard carve-out for general job postings or unsolicited applicants, so counsel should narrow it rather than legal needing a full review.

NDA-3 [PARTY C] — RED
Clause 2: "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business."
Reasoning: a three-year, unbounded non-compete stacked on one-way confidentiality, a five-year term, zero confidentiality carve-outs ("No confidentiality carve-outs are stated in this Agreement"), and exclusive Irish jurisdiction — reject or escalate in full.

Data missing: NDA-1 and NDA-2 state no standstill, no residuals, and no definition of Confidential Information beyond the implied one — not flagged, but unstated rather than confirmed standard. NDA-3's non-compete has no geographic or scope limit stated, which is itself part of why it is RED.