← back to all models ← RevenueBench home
moonshotai

moonshotai/kimi-k3

Mean 0.976 · 36/40 perfect tests · $7.67 total · modalities in: text, image, video · out: text · each card: the prompt → the correct answer (gold) → this model's actual answer

What do the modalities mean?

Modalities in means what you can send this model: text, images, files, video, or audio. Modalities out means what it can send back. Bench tasks feed text extracts, so modalities do not change the score. They are here so you know what this model can handle in real work.

data-crm 5 tests
stage-hygiene-auditPASS 1.00
Plain-English question: How much of the open DS2+ pipeline is missing a substantive why-buys, and is anything missing DS1 timestamps or out of order?
Prompt — what the model saw
Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.3913 · 380s · in 25,552 / out 20,944 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

Plain-English answer: 81.5% of the 124 DS2+ deals are missing a substantive why-buys — Alex Franklin 48, Cole Ingram 19, Dana Mercer 18, Bryce Harmon 15, Farid Osman 1 — and every deal has a DS1 timestamp with none regressed.
The answer in one line: n_deals = 156; ds2plus = 124; noncomp_pct = 81.5; bryce_noncomp = 15; no_ds1 = 0; regressed = 0 (+1 more below)
n_deals156
bryce_noncomp15
ds2plus124
no_ds10
noncomp_pct81.5
regressed0
per_owner_noncompAlex Franklin=48 of 61 (78.7%); Cole Ingram=19 of 22 (86.4%); Dana Mercer=18 of 22 (81.8%); Bryce Harmon=15 of 18 (83.3%); Farid Osman=1 of 1 (100%)
graded checksnoncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
DEFINITIONS
- DS2+ = stage DS2, DS3, DS4, or DS5. DS1 deals are excluded from the why-buys audit.
- Why-buys non-compliant = why_buys empty, under 15 chars, or a placeholder (tbd, n/a, see notes). LONG(n) counts as present and substantive per the brief.
- Missing DS1 = t_ds1 blank.
- Regressed = any earlier-stage entry timestamp later than a later-stage entry timestamp across t_ds1 -> t_ds5 (populated fields only).

SCOPE
156 open deals total: 124 DS2+, 32 DS1.

WHY-BUYS COMPLIANCE (DS2+ ONLY)
Overall: 101 of 124 non-compliant = 81.5% (101/124 = 0.8145).
All 23 compliant deals hold LONG(n) values; every failure is an empty field — no placeholder or short-text values exist in the data.

By owner (non-compliant / DS2+ deals = %):
- Bryce Harmon:        15/18  = 83.3%
- Dana Mercer:        18/22  = 81.8%
- Alex Franklin:     48/61  = 78.7%
- Cole Ingram: 19/22 = 86.4%
- Farid Osman:        1/1   = 100.0%
- Elena Sinclair: n/a — 0 DS2+ deals (her only open deal, Deal-57FF13, is DS1)

Cross-check: 15+18+48+19+1 = 101; 18+22+61+22+1 = 124. ✓

OTHER HYGIENE CHECKS
- Deals with no DS1 timestamp: 0 (all 156 rows have t_ds1 populated).
- Deals that regressed: 0 (every populated t_ds1 -> t_ds5 sequence is non-decreasing).

THREE PROCESS FIXES
1. Hard-gate why-buys at the DS2 transition. Failure is systemic (81.5%, every owner at 78.7%+), so make a 15+ character substantive why-buys a CRM validation requirement to enter DS2 — with a placeholder blocklist — instead of relying on retroactive cleanup.
2. Publish a per-owner hygiene scorecard in weekly pipeline reviews and require fixes before forecast commit. Prioritize by absolute backlog: Alex Franklin (48 non-compliant) first, then Cole Ingram (19) and Dana Mercer (18); Farid Osman's 100% rate is n=1 but should still be flagged.
3. Fix stage-entry timestamp capture. t_ds5 is blank on all 10 DS5 deals (their DS5 entry exists only in entered_current_stage), and Deal-0D0211 and Deal-7E2131 carry a t_ds4 while sitting at DS3, while Deal-C7F9BF carries a t_ds3 while sitting at DS2 — timestamps for stages beyond the current stage suggest unlogged back-and-forth movement. Automate write-on-stage-entry and backfill, or regression and time-in-stage analytics stay uncomputable.
owner-resolution0.83
Plain-English question: Every open deal has a blank owner name — resolve each one to a real person using the owners table.
Prompt — what the model saw
The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.3581 · 84s · in 40,164 / out 14,500 tok
✓ resolved_count✗ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: All 156 deals resolve to six named owners; jake_amount maps to Bryce Harmon ($1,054,144) and justin_amount to Alex Franklin ($624,310).
The answer in one line: jake_amount = 1,054,144; justin_amount = 624,310; resolved = 156
jake_amount1,054,144
justin_amount624,310
resolved156
deactivated
per_ownerBryce Harmon=1,054,144; Alex Franklin=624,310; Dana Mercer=341,195; Cole Ingram=288,161.43; Farid Osman=4,134; Elena Sinclair=2,100
graded checksresolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
All 156 rows joined cleanly against the owners table. Results:

1) Deals resolved: 156 of 156.

2) Deals with no owner match: none. Every owner_id present in deals_open.csv (119337721, 701163055, 716654662, 83155923, 83155924, 84342457) exists in owners.csv. No names had to be guessed.

3) Owner ids mapping to archived/deactivated owners: none. All six matched owners have archived=false. The two archived owners in the table — Gavin Porter (1520255671) and Hugo Lindqvist (77260721) — do not appear on any open deal in this extract.

4) Total pipeline amount per resolved owner (sum of amount over their deals):

  Alex Franklin (84342457)      624,310.00   (67 deals)
  Bryce Harmon (119337721)      1,054,144.00   (35 deals)
  Dana Mercer (83155923)         341,195.00   (24 deals)
  Cole Ingram (83155924) 288,161.43   (22 deals)
  Farid Osman (716654662)         4,134.00   ( 7 deals)
  Elena Sinclair (701163055)       2,100.00   ( 1 deal)

  Arithmetic check: 624,310.00 + 1,054,144.00 + 341,195.00 + 288,161.43 + 4,134.00 + 2,100.00 = 2,314,044.43 total resolved pipeline across 156 deals. Cole Ingram's total is the only non-integer: seven of his deals carry cent amounts (7,225.40; 7,781.20; 6,947.50; 2,480.40; 4,779.88; 3,334.80; 58,529.25).

Note: pipeline figures are raw sums of the amount field as provided — no forecast-category weighting or stage filtering was applied, since none was requested.
gong-hubspot-join-integrityPASS 1.00
Plain-English question: How many call records point at deal ids that do not exist in the open pipeline?
Prompt — what the model saw
Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.5054 · 142s · in 52,059 / out 21,766 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

Plain-English answer: 30 Gong call rows are orphans keyed to dead deal ids, none are duplicates, and DS3 call coverage is 29.4%.
The answer in one line: orphans = 30; ds3_cov_pct = 29.4; dup = 0
orphans30
ds3_cov_pct29.4
dup0
orphan_aliasesDeal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
graded checksorphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
Integrity check results (computed by exact join on hs_deal_id → deal_id; gong rows = 67, deals_open = 156).

(a) ORPHANS — gong rows with no matching deal in deals_open: 30 of 67 (44.8%)
    These 30 orphan rows account for 304 logged calls (46+24+21+21+17+17+14+12+11+11+9+8+7+6+6+6+6+5x10+4x3 = 304). 37 of 67 gong rows do match an open deal.
    Sample orphan aliases (all from the gong table):
      Deal-8FA85D (id 60251290957, 46 calls, last_call 2026-11-05)
      Deal-8FC3F9 (id 60251649055, 24 calls, last_call 2026-10-19)
      Deal-3B7945 (id 60251639682, 21 calls, last_call 2026-10-20)
      Deal-42B265 (id 61227242540, 21 calls, last_call 2026-11-25)
      Deal-9CCC42 (id 61430316324, 17 calls, last_call 2026-11-27)
      Deal-36EA09 (id 60251082126, 17 calls, last_call 2026-11-19)
    Two structural anomalies inside the orphan set, both taken straight from the data:
      - The highest-volume orphans are future-dated: 8 orphan rows have last_call after 2026-09-05 (up to 2026-11-27), which is impossible for a "calls in the last 90 days" extract as of today. Every matched row's last_call falls on/before 2026-09-16.
      - The entire orphan block with 46/24/21/21/17/17/14/12/11/11/9/8/7/6/5... calls has calls_with_ds2_date=0 AND calls_with_sqo_date=0, and its gong_owner_id values (e.g. 373679877710534937, 6440417017135074779, 7971335727414442543, 4654480998631630424, 7441100256678694562, 247470326654037859) never appear on any matched row. This looks like a different Gong source/segment (or a different pipeline whose deals aren't in the default-pipeline deals_open extract), not random key breakage. Per the note, gong_owner_id can't be cross-walked to HubSpot owner_id to verify.

(b) DUPLICATE CONVERSATION KEYS: 0 rows.
    Every one of the 67 gong rows has calls_90d == distinct_conversation_keys; no row has calls_90d exceeding distinct_conversation_keys (and none below it). Conversation keys are unique per row across the file.

(c) DS3+ open deals with at least one logged call: 25 of 85 = 29.4%
    Arithmetic: open deals at stage DS3/DS4/DS5 = 61 + 14 + 10 = 85. Of these, 25 have their deal_id present in the gong table (12 of 61 DS3, 6 of 14 DS4, 7 of 10 DS5). 25/85 = 0.2941 → 29.4%. 60 DS3+ deals have no logged call in the 90d window, including COMMIT-stage Deal-403845 (DS5), Deal-2465CE (DS5), Deal-FD9F4E (DS5) and BEST_CASE Deal-7BBDFA (DS3, amount 37440), Deal-530B50 (DS3, 31200), Deal-B25F40 (DS3, 40000).
    Coverage rises sharply with stage: DS3 12/61 = 19.7%, DS4 6/14 = 42.9%, DS5 7/10 = 70.0%.

Caveat: if the orphan rows belong to deals outside the default pipeline (deals_open is default-pipeline only), the 29.4% figure is understated for DS3+; the data provided does not include other pipelines, so that cannot be confirmed here.
snowflake-sql-trapped-catalogPASS 1.00
Plain-English question: Write one Snowflake query for first-month usage per customer using only the provided, trap-laden catalog.
Prompt — what the model saw
Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.1747 · 157s · in 15,475 / out 8,549 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

Plain-English answer: A single correct SQL query that uses the right catalog tables and avoids the trap columns — graded on the exact tables and columns.
The answer in one line: use_table = PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
avoid_stalePRODUCTION.HUBSPOT.DEALS (stale)
avoid_unpopulatedV2_LIVE.OBJECTS_DEALS (unpopulated)
forbidden_filterdeleted-giver exclusion must NOT be applied
outputone fenced SELECT with per-FROM/JOIN table-choice comment
required_filtersucceeded redemptions only
use_tablePRODUCTION.PLG.COMPANY_COHORT_SUMMARY
graded checksavoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Per customer company, first calendar month as a customer:
-- unique givers, recognition count, successful redemption count.
--
-- Catalog-driven decisions:
--   * The catalog contains NO raw giving/recognition event table, so historical giving counts
--     can only come from pre-aggregated M1_* columns. Consequently the deleted-giver exclusion
--     (which per business rules must NOT be applied to historical giving counts) is not applied
--     anywhere -- there is no giver-level table in this catalog to apply it to.
--   * PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2 is the documented raw source for
--     redemption counts (STATE = 'succeeded' only), but the catalog documents no columns for it
--     (no company key, no event timestamp), so joining it for a per-company first-month count
--     would require inventing schema. M1_REDEMPTIONS, the documented month-1 redemption count,
--     is used instead.
--   * Stale/unpopulated/retired tables avoided: HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS
--     (unpopulated), PRODUCTION.HUBSPOT.DEALS (stale since 2023-03),
--     PRODUCTION.HUBSPOT.GONG_HUBSPOT_MAP_FAST (retired).
--   * Coverage caveat: this table covers self-serve companies only; the catalog offers no
--     giving/redemption source for sales-led customers, so they cannot be reported here.
SELECT
    COMPANY_ID,                                                -- catalog states one row per self-serve company but does not name the key column; COMPANY_ID is the assumed identifier
    DATE_TRUNC('MONTH', FIRST_SUB_PAYMENT_DATE)
        AS FIRST_CUSTOMER_CALENDAR_MONTH,                      -- first subscription payment = became a customer; truncated to its calendar month
    M1_USERS       AS UNIQUE_GIVERS,                           -- PROXY ONLY: catalog documents no unique-givers metric; M1_USERS (month-1 users) is the only month-1 people count available
    M1_GIVES       AS RECOGNITION_COUNT,                       -- month-1 gives = recognitions sent
    M1_REDEMPTIONS AS SUCCESSFUL_REDEMPTION_COUNT              -- documented month-1 redemption count (see note above on the raw events table)
-- FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY: the only cataloged table holding per-company
-- giving and redemption metrics; one row per self-serve company with M1_* = first-month values,
-- so first-calendar-month metrics require no event scan and no other table.
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
WHERE FIRST_SUB_PAYMENT_DATE IS NOT NULL                       -- restrict to companies that actually became customers (made a first subscription payment); M1 anchor is not documented and is assumed to align with this first month
;
```
crm-completeness-enrichmentPASS 1.00
Plain-English question: Audit the CRM for missing and invalid fields, and propose the fixes.
Prompt — what the model saw
Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.2608 · 135s · in 8,000 / out 15,157 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

Plain-English answer: 9 companies missing employee count, 6 missing HQ country, 4 invalid emails, 1 domain mismatch, 2 duplicate clusters, and 25 enrichment matches.
The answer in one line: n_contacts = 52; n_companies = 34; enrichment_matches = 25; invalid_email_count = 4; n_dup_clusters = 2; domain_mismatch_count = 1 (+4 more below)
n_contacts52
domain_mismatch_count1
enrichment_matches25
invalid_email_count4
n_companies34
n_dup_clusters2
company_missingindustry=0; employee_count=9; hq_country=6
domain_mismatchesCT-0011
duplicate_clusters['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
invalid_emailsCT-0010, CT-0080, CT-0081, CT-0192
graded checksdup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
CRM AUDIT — COMPLETENESS & FIXES
(3 files received: companies.csv [34 rows], contacts.csv [52 rows], zoominfo_enrichment.csv [25 rows]. NO deals file was provided — see §1.)

════════════════════════════════════════
1. COMPLETENESS PER FIELD
════════════════════════════════════════

DEALS — NOT AUDITABLE. No deals.csv was included in the extract. Completeness for owner, stage, amount, close_date, and why_buys cannot be computed. Fix 1 below is to supply the file; until then pipeline amount at stake is unquantifiable for every fix.

COMPANIES (n = 34)
  industry        34/34 = 100.0%  (but non-normalized values: 'tech', 'Tech ' [trailing space], 'health care' — see §6)
  employee_count  25/34 = 73.5%   missing (9): C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386, C-93C8BF
  hq_country      28/34 = 82.4%   missing (6): C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB
                                  (also non-normalized: 'US', 'USA', 'United States', 'UK', 'Canada' all in use)

CONTACTS (n = 52)
  email     52/52 = 100.0% populated, but only 48/52 = 92.3% are valid-format (4 malformed, §3)
  title     39/52 = 75.0%   missing (13): CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170
  persona   37/52 = 71.2%   missing (15): CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181

COVERAGE GAP (not a field, but a completeness issue): 14 of 34 companies (41%) have ZERO contacts: C-2C60E5, C-B97B4E, C-2D7423, C-7BBDFA, C-50D386, C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20, C-0A092931, C-0A092932, C-0A092933, C-0A092934. No orphan contacts (every contact's company_alias resolves).

════════════════════════════════════════
2. DUPLICATE COMPANY CLUSTERS (shared domain)
════════════════════════════════════════

Cluster A — domain acme-corp.com
  C-0A092931  industry='Technology', employees=500, hq='US'
  C-0A092932  industry='tech',       employees=510, hq='USA'
  SURVIVOR: C-0A092931 (fully populated, canonical-style industry label, lower ordinal).
  Unresolved conflict: employee_count 500 vs 510 — ZoomInfo has NO row for acme-corp.com, so the data cannot arbitrate. Do not guess; verify manually, then drop C-0A092932.
  No contacts or deals reference either alias, so the merge loses nothing.

Cluster B — domain globex.io
  C-0A092933  industry='SaaS',       employees=200, hq='US'
  C-0A092934  industry='Technology', employees=200, hq='US'
  SURVIVOR: C-0A092934 (matches the 'Technology' taxonomy used by the rest of the file).
  Unresolved conflict: industry 'SaaS' vs 'Technology' — no ZoomInfo row for globex.io; confirm with the business, then drop C-0A092933. Employees/country agree.

No name-variant clusters beyond these two (all other domains unique).

════════════════════════════════════════
3. INVALID EMAILS & DOMAIN MISMATCHES
════════════════════════════════════════

Malformed emails (4) — all are truncated 'user@' with no domain:
  CT-0010 (C-66D1FC)  'user0@'
  CT-0080 (C-92D97D)  'user0@'
  CT-0081 (C-92D97D)  'user1@'
  CT-0192 (C-425E2A)  'user2@'
  Fix: the local part + company domain strongly suggest the intended address (e.g. user0@66d1fc.com), but that is a reconstruction, not data — verify with the source system before writing. Do not auto-fill.

Domain mismatch (1):
  CT-0011 (C-66D1FC)  email 'user1@other-domain.com' vs company domain '66d1fc.com'
  Fix: flag for rep review — either the contact is mis-associated to C-66D1FC, or the email belongs to a different employer. Data cannot resolve which.

════════════════════════════════════════
4. ENRICHMENT FILLS (ZoomInfo) — only where a matching domain row exists
════════════════════════════════════════

FILLABLE (9 fills across 7 companies — ZI row exists and ZI value is non-blank):
  C-EC3025 (ec3025.com)   employee_count = 400
  C-96039F (96039f.com)   employee_count = 400
  C-44EA29 (44ea29.com)   employee_count = 400
  C-D04904 (d04904.com)   employee_count = 400
  C-B23205 (b23205.com)   employee_count = 400
  C-60C75F (60c75f.com)   employee_count = 400
  C-7BBDFA (7bbdfa.com)   employee_count = 400
  C-50D386 (50d386.com)   employee_count = 400
  (arithmetic: 8 company fills listed, 9 fields — C-EC3025 employee_count only; its industry is already populated)

Correction — recount: the 9 employee_count-blank companies; 8 have a ZI value, C-93C8BF does not. So 8 fills, not 9.

NOT FILLABLE — ZI row exists but ZI value also blank (do not invent):
  hq_country: C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5

NOT FILLABLE — no ZI row at all:
  C-93C8BF (employee_count), C-EE9FFB (hq_country)

Post-fill projected completeness: employee_count 33/34 = 97.1%; hq_country unchanged at 82.4%.

════════════════════════════════════════
5. CRM vs ZOOMINFO DISAGREEMENTS & RECOMMENDED SOURCE
════════════════════════════════════════

VALUE CONFLICTS — industry, 8 companies. CRM says Technology/tech/Tech; ZI says 'Computer Software':
  C-66D1FC, C-EC3025, C-44EA29, C-92D97D, C-D04904, C-77A95A, C-AA8DDA, C-B25F40, C-60C75F, C-425E2A
  (recount from output: 10 companies, not 8 — C-66D1FC, C-EC3025, C-44EA29, C-92D97D, C-D04904, C-77A95A, C-AA8DDA, C-B25F40, C-60C75F, C-425E2A)
  These are the same segment under two labels, not a factual dispute. RECOMMENDATION: keep CRM as system of record, adopt ONE canonical taxonomy ('Technology'), and map ZI 'Computer Software' -> 'Technology' on every future import. Also normalize within CRM: 'tech' -> 'Technology', 'Tech ' -> 'Technology' (trim whitespace), 'health care' -> 'Healthcare' (C-7BBDFA, C-50D386), 'SaaS' resolved via Cluster B.

FORMATTING-ONLY — hq_country, 10 companies. CRM 'US'/'USA' vs ZI 'United States':
  C-66D1FC, C-950043, C-EC3025, C-96039F, C-77A95A, C-B23205, C-E51FB7, C-D0662E, C-425E2A, C-2D7423
  Same country, different strings. RECOMMENDATION: neither source is wrong — standardize on one code list (ISO-2 'US' is the cheapest to enforce at entry) and apply it to both CRM and the ZI import mapping. Within the dup Cluster A, C-0A092931 'US' vs C-0A092932 'USA' is the same non-issue; the real conflict there is 500 vs 510 employees (see §2, unresolved — no ZI row).

Employee_count: no CRM-vs-ZI numeric conflicts outside the dup cluster (wherever both sides have a number, they agree: 900, 1500, 120, 340, 50 all match).

════════════════════════════════════════
6. TOP 10 FIXES BY PIPELINE AMOUNT AT STAKE
════════════════════════════════════════

No deals file exists, so no amount can be attached to any fix. Pipeline at stake = UNKNOWN ($0 quantifiable) for every item; ranked instead by revenue risk severity. If a deal is later found tied to any alias below, that fix jumps the queue.

 1. Supply the deals extract. 0 of 5 required deal fields (owner, stage, amount, close_date, why_buys) are auditable. Amount at stake: the entire pipeline — unquantifiable until provided.
 2. Merge dup Cluster A (acme-corp.com): survivor C-0A092931, resolve 500 vs 510 manually. Amount: unknown — no deals data. Risk: any future deal double-counted across two records.
 3. Merge dup Cluster B (globex.io): survivor C-0A092934, confirm industry. Amount: unknown. Same double-count risk.
 4. Repair 4 malformed emails (CT-0010, CT-0080, CT-0081, CT-0192) via source system. Amount: unknown — three of the four are on companies (C-66D1FC, C-92D97D, C-425E2A) whose other contacts are champions/EBs; broken emails kill sequenced outreach.
 5. Resolve CT-0011 domain mismatch (user1@other-domain.com at C-66D1FC). Amount: unknown. Risk: champion contact pointed at the wrong company.
 6. Apply the 8 ZoomInfo employee_count fills (C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386). Amount: unknown. Unblocks territory/segment scoring.
 7. Fill the 15 missing personas and 13 missing titles (keys listed in §1), prioritizing companies with open pipeline once deals exist. Amount: unknown. Persona-based routing is blind on 29% of contacts.
 8. Create contact coverage for the 14 zero-contact companies (§1) — 41% of the account base is unreachable. Amount: unknown; this is the single largest structural gap after the missing deals file.
 9. Normalize taxonomies: industry -> {Technology, Manufacturing, Retail, Healthcare, Finance, ...}; country -> ISO-2. Covers 'tech', 'Tech ', 'health care', 'US'/'USA'/'United States'. Amount: none directly, but every report/filter built on these fields is currently wrong.
 10. Re-pull ZoomInfo (or alternate source) for the 7 unfillable fields: hq_country on C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB; employee_count on C-93C8BF. Amount: unknown. ZI returned blanks/no-row; left empty, these stay empty — flagging rather than inventing.

Bottom line: contacts are 100% email-populated but 92.3% email-valid, 75.0% title, 71.2% persona; companies are 100% industry, 73.5% employee_count (97.1% after ZI fills), 82.4% country; deals are 0% auditable. Send the deals file and I can convert this severity ranking into a dollar-ranked one.
deal-intelligence 5 tests
closed-lost-classificationPASS 1.00
Plain-English question: Why did we lose the last six months of deals — one primary category per deal, and whose side is the reason on?
Prompt — what the model saw
Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.5736 · 210s · in 14,210 / out 34,137 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

Plain-English answer: Every closed-lost deal gets exactly one category (pricing, competitor, no decision, timing, product gap, champion left, other) plus a side, per the computed classification.
The answer in one line: total_lost_6mo = 90; top_reason = competitor (26)
total_lost_6mo90
categoriespricing, competitor, no decision, timing, product gap, champion left, other
patternstwo most actionable, tag-vs-text disagreements counted
sideBonusly / buyer / unknown per deal
top_reasoncompetitor (26)
graded checkstotal_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Classification rules applied (only tag + free-text used):
- "MIA" tag + silence text -> no decision. "Doing nothing/Not a priority/Cost" + pause/deprioritize -> no decision, unless text names an incumbent contract (then competitor) or explicit budget failure (then pricing). "Feature Request" -> product gap unless text says they bought something else (then competitor). "Lost DM" is ambiguous and was classified from text only. Timing required a stated delay/reconnect window; "at this time" with no re-engagement path -> no decision. Side = Bonusly when text cites something we control (feature gaps, platform fit); buyer when the cause sits on their side; unknown when text is silent or generic ("went another direction", "MIA").

PER-DEAL CLASSIFICATION (alias | tag -> primary category | side)

1. Deal-DB0AAC | Lost- Timing -> timing | buyer
2. Deal-F7F635 | Competitor -> competitor | unknown
3. Deal-AC944F | MIA -> no decision | unknown
4. Deal-214060 | MIA -> no decision | unknown
5. Deal-91A056 | Lost- Timing -> timing | buyer
6. Deal-29326C | Lost- Timing -> timing | unknown
7. Deal-5DB9B0 | Does not fit ICP -> other | unknown (text: "Spam.")
8. Deal-831B7B | Lost- Timing -> timing | buyer
9. Deal-F97C37 | Competitor -> competitor | buyer
10. Deal-13E9CF | Doing nothing -> no decision | buyer ("Not a budget issue - ... deprioritized")
11. Deal-39E25C | Lost- Timing -> timing | buyer
12. Deal-7ED004 | Lost- Budget/Price -> pricing | buyer
13. Deal-21B045 | MIA -> no decision | unknown
14. Deal-B3ABED | Lost- Timing -> timing | buyer
15. Deal-422BA6 | Competitor -> competitor | buyer
16. Deal-ED9AE7 | Lost DM -> no decision | buyer ("Timing, budget, authroity" — multi-factor stall, no single driver)
17. Deal-988493 | MIA -> no decision | unknown
18. Deal-381C8C | Competitor -> competitor | unknown
19. Deal-F308CA | MIA -> no decision | unknown
20. Deal-F1E8A6 | Competitor -> competitor | unknown
21. Deal-B6AC09 | Lost- Timing -> timing | buyer
22. Deal-70F704 | Lost DM -> no decision | unknown
23. Deal-E6E80A | Lost- Timing -> timing | buyer
24. Deal-B038F0 | Lost- Timing -> timing | buyer
25. Deal-4664E1 | MIA -> no decision | unknown
26. Deal-175756 | Lost- Timing -> timing | buyer
27. Deal-E74A73 | Doing nothing -> no decision | buyer (manual test first)
28. Deal-DDAB52 | Competitor -> competitor | buyer (Rippl)
29. Deal-ACE061 | Competitor -> competitor | buyer (likely HeyTaco)
30. Deal-BB78F3 | Lost- Timing -> timing | buyer
31. Deal-D48E0B | MIA -> no decision | unknown
32. Deal-15DA99 | Lost- Timing -> timing | buyer
33. Deal-F4AF5D | Lost- Timing -> timing | buyer
34. Deal-79B7A1 | Lost- Timing -> timing | unknown
35. Deal-583ADB | MIA -> no decision | unknown
36. Deal-8E27DA | Feature Request -> competitor | buyer (bought a swag provider instead)
37. Deal-2D2F8D | Competitor -> competitor | unknown
38. Deal-E0441F | MIA -> no decision | unknown
39. Deal-7CB44D | MIA -> no decision | unknown
40. Deal-0F96AA | Competitor -> competitor | unknown
41. Deal-1BCA50 | Competitor -> competitor | buyer
42. Deal-7CC678 | Competitor -> competitor | unknown
43. Deal-FAC17C | Lost DM -> no decision | buyer (internal approval blocked)
44. Deal-242273 | Competitor -> product gap | Bonusly (couldn't solution onsite points spending)
45. Deal-50E5D8 | Doing nothing -> no decision | buyer
46. Deal-A2C349 | Competitor -> competitor | buyer (stayed with Awardco)
47. Deal-9F176A | Lost- Timing -> timing | buyer
48. Deal-7B2236 | Doing nothing -> pricing | buyer ("simpler and cheaper")
49. Deal-AFA56C | MIA -> no decision | unknown
50. Deal-C7156E | Competitor -> competitor | unknown
51. Deal-C33D91 | Lost- Budget/Price -> pricing | buyer
52. Deal-9048EB | MIA -> product gap | Bonusly (text: "bad fit ... multiple feature gaps")
53. Deal-5E64CE | Doing nothing -> competitor | buyer (locked into Nectar through Oct 2027)
54. Deal-8A0992 | Competitor -> competitor | buyer (Canadian provider)
55. Deal-D0C698 | Competitor -> competitor | buyer (past Kudos user)
56. Deal-69CF3D | Lost- Timing -> timing | buyer
57. Deal-ECBF89 | Lost- Timing -> timing | buyer
58. Deal-3618CC | Lost DM -> product gap | Bonusly ("Wanted Surveys")
59. Deal-EECC02 | Competitor -> competitor | unknown
60. Deal-5AD03E | Competitor -> product gap | Bonusly ("Wanted more defined budget access")
61. Deal-D1A623 | Lost- Timing -> timing | unknown
62. Deal-413C56 | Doing nothing -> timing | buyer (CEO not ready, school-year priority)
63. Deal-47F1A1 | Competitor -> competitor | buyer (WorkTango)
64. Deal-BF2A98 | Competitor -> competitor | buyer (HiThrive)
65. Deal-2A292B | Doing nothing -> competitor | buyer (build internally = build-vs-buy competitor)
66. Deal-D1AABF | MIA -> no decision | unknown
67. Deal-FEDBCB | Doing nothing -> no decision | unknown
68. Deal-1E7DA9 | Competitor -> competitor | unknown
69. Deal-2BBA21 | MIA -> no decision | unknown
70. Deal-286F9C | Competitor -> competitor | buyer
71. Deal-7FBAC6 | Doing nothing -> no decision | buyer (paused again)
72. Deal-369281 | Competitor -> competitor | buyer (Paylocity native)
73. Deal-386F6E | MIA -> no decision | unknown
74. Deal-9FCD0D | Competitor -> competitor | buyer (Canadian company, CEO preference)
75. Deal-55867E | Lost- Timing -> no decision | buyer ("not moving forward at this time", no reconnect date given)
76. Deal-DAFB82 | Lost- Budget/Price -> pricing | buyer (no budget until 2028)
77. Deal-2FEDDB | Doing nothing -> timing | buyer (unsure when she can move it)
78. Deal-64B19A | Competitor -> competitor | buyer (likely stayed with Motivosity)
79. Deal-3F86A0 | MIA -> no decision | unknown
80. Deal-096750 | MIA -> no decision | unknown
81. Deal-F325A5 | Lost DM -> champion left | buyer (layoffs + change in leadership)
82. Deal-ABD14C | Doing nothing -> no decision | buyer
83. Deal-79E61A | MIA -> no decision | unknown
84. Deal-8A119B | Lost- Budget/Price -> pricing | buyer
85. Deal-AE7C4E | MIA -> no decision | unknown
86. Deal-DAB4F1 | MIA -> no decision | unknown
87. Deal-B4B50F | MIA -> no decision | unknown
88. Deal-981AD4 | Feature Request -> product gap | Bonusly (UI fit, not UK-focused)
89. Deal-DC77FE | Competitor -> competitor | buyer (customization — label points as dollars; price explicitly not a factor)
90. Deal-5885B9 | MIA -> no decision | unknown

SUMMARY

Total deals: 90

Category counts (count / % / $):
- no decision:    31 (34.4%)  $281,844.20
- competitor:     27 (30.0%)  $352,594.96
- timing:         20 (22.2%)  $257,611.00
- pricing:         5 (5.6%)   $172,450.00
- product gap:     5 (5.6%)   $178,245.00
- other:           1 (1.1%)   $10,800.00
- champion left:   1 (1.1%)   $14,400.00
Arithmetic: 31+27+20+5+5+1+1 = 90. Total $ = 281,844.20+352,594.96+257,611.00+172,450.00+178,245.00+10,800.00+14,400.00 = $1,267,945.16

Side split (count / % / $):
- buyer:    49 (54.4%)  $593,874.16
- unknown:  36 (40.0%)  $495,826.00
- Bonusly:   5 (5.6%)   $178,245.00
Arithmetic: 49+36+5 = 90. $593,874.16 + $495,826.00 + $178,245.00 = $1,267,945.16 (ties to total)

Tag-vs-text clear disagreements: 10 deals
- Deal-8E27DA (Feature Request tag; text = chose a swag provider)
- Deal-242273 (Competitor tag; text = we couldn't meet the need)
- Deal-7B2236 (Doing nothing tag; text = wanted simpler/cheaper)
- Deal-9048EB (MIA tag; text = bad fit + multiple feature gaps)
- Deal-5E64CE (Doing nothing tag; text = locked into Nectar contract)
- Deal-5AD03E (Competitor tag; text = wanted defined budget access — a capability ask)
- Deal-413C56 (Doing nothing tag; text = timing/CEO not ready)
- Deal-2A292B (Doing nothing tag; text = building internally)
- Deal-55867E (Timing tag; text = not moving forward, no reconnect date)
- Deal-2FEDDB (Doing nothing tag; text = timing uncertainty)
Note: "Lost DM" was treated as unmapped, so those reclassifications aren't counted in the 10.

Two patterns most worth acting on:

1. We are losing to specific capabilities, not to competitors wholesale. At least 6 competitor/product-gap losses name an exact feature the winner had and Bonusly lacked: surveys (Deal-3618CC, Deal-A2C349), defined budget access (Deal-5AD03E), points-as-dollars customization (Deal-DC77FE), onsite/internal points spending (Deal-242273), diversified offerings beyond R&R (Deal-F97C37), swag (Deal-8E27DA). Add Deal-9048EB ($41,790, "multiple feature gaps") and the 5 product-gap deals alone total $178,245. These are roadmap-addressable — surveys and flexible points/budget configuration recur. Fix the features and a meaningful slice of the 27 competitor losses gets winnable. All Bonusly-side losses are only 5.6% of deals but are the most directly controllable.

2. The pipeline leaks at the top and we aren't learning why. No decision (31 deals, 34.4%) is the largest category, and 21 of those 31 are pure ghosting (MIA/unresponsive/no response). Several died immediately after intro/demo with 1-2 contacts engaged (Deal-F308CA $30,321, Deal-4664E1, Deal-2BBA21, Deal-096750) and one was inherited stale from a departed rep (Deal-E0441F). On top of that, 36 of 90 deals (40%) have unknown side — generic notes like "went another direction" / "Nothing specific provided" (Deal-0F96AA $76,800, Deal-EECC02 $66,690) mean we're flying blind on our biggest losses. Action: multi-thread deals past 1-2 contacts, tighten post-demo follow-up SLAs, and enforce structured competitor/exit fields at closed-lost so $495,826 of unknown-side pipeline isn't unexplainable.
pipeline-tieringPASS 1.00
Plain-English question: Score all 156 open deals into exactly one of LOCK / ACTION / BUILD / REVIVE / WATCH / RISKY.
Prompt — what the model saw
Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.4646 · 511s · in 45,117 / out 21,200 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

Plain-English answer: All 156 open deals tiered exactly once across the six tiers using forecast category, stage, meetings, contacts, and recency.
The answer in one line: total = 156
total156
graded checksjson_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
```json
{
  "tier_counts": {"LOCK": 3, "ACTION": 20, "BUILD": 39, "WATCH": 80, "REVIVE": 7, "RISKY": 7},
  "tier_examples": {
    "LOCK": ["Deal-D348E1", "Deal-C26D20", "Deal-403845"],
    "ACTION": ["Deal-25F752", "Deal-547B2B", "Deal-944310"],
    "BUILD": ["Deal-D73B89", "Deal-523604", "Deal-2B39B0"],
    "WATCH": ["Deal-C6FE92", "Deal-93C8BF", "Deal-036E80"],
    "REVIVE": ["Deal-2D1F1B", "Deal-F0EBBB", "Deal-1BEEBF"],
    "RISKY": ["Deal-B7EBD1", "Deal-0660B4", "Deal-BA571A"]
  },
  "risky_deals": ["Deal-B7EBD1", "Deal-0660B4", "Deal-BA571A", "Deal-7BBDFA", "Deal-4A13AD", "Deal-690476", "Deal-635B8E"],
  "lock_violations": 0,
  "pipeline_shape": "156 open deals, and the shape is bottom-heavy with a thin, healthy top: only 3 LOCKs (all DS5 COMMIT with meetings in the last 30d, fresh engagement, 3+ contacts), backed by 20 ACTION deals clustered in DS4/DS5/COMMIT-BEST_CASE that are actively touched but meeting-starved — the single gap keeping them out of LOCK. 7 RISKY deals carry optimistic forecasts (COMMIT/BEST_CASE) contradicted by zero meetings_30d and stale or near-zero 30-day activity, including DS5 COMMIT Deal-B7EBD1 (nothing since 2026-08-20) and BEST_CASE Deal-7BBDFA (silent since 2026-07-21). The middle is the bulk: 80 WATCH deals, mostly DS2/DS3 running on email-only engagement, and 39 BUILD deals (DS1/DS2) showing real meeting momentum. 7 REVIVE deals are effectively dormant, and hygiene flags include several $1-amount placeholders (Deal-012CB1, Deal-483B2D, Deal-3795AD, the 716654662-owned DS1 batch) and at least one close_date already in the past (Deal-333EBB, 2026-08-28)."
}
```
call-transcript-extractionPASS 1.00
Plain-English question: Extract the CRM write-back fields from each call transcript as JSON.
Prompt — what the model saw
For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.1524 · 83s · in 16,335 / out 6,892 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

Plain-English answer: Per transcript: prospect-only why-buys, pain points, stakeholders, budget and timeline signals, and competitors — e.g. TX-001: '$40k earmarked', live before open enrollment.
The answer in one line: rep = Alex Franklin
allowed_newTX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
per_transcript{"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
repAlex Franklin
rep_sourced_nulls{"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
graded checksjson_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "Automate anniversary and birthday awards — 'The big win for us would be automating anniversary and birthday awards' (VP People)"
    ],
    "pain_points": [
      "HR team of three cannot keep up with awards manually",
      "Everything tracked in a spreadsheet; 'people slip through the cracks' (HR Admin)"
    ],
    "stakeholders": ["VP People", "HR Admin"],
    "budget_signal": "~$40k earmarked for engagement tools this fiscal year (VP People)",
    "timeline_signal": "Ideally live before open enrollment in November",
    "competitor_mentioned": "Achievers — looked at last year; 'too heavy for a team our size'",
    "next_step": "Security review with IT lead on September 12 (explicitly agreed)",
    "objections": [
      "Requires SSO and audit logs for IT to sign off (HR Admin)"
    ],
    "confidence": "high — budget, timeline, competitor, and a dated next step are all prospect-stated"
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "Tie recognition to retention for the hourly workforce (Head of Total Rewards)"
    ],
    "pain_points": [
      "Regretted turnover over 30% in the hourly workforce"
    ],
    "stakeholders": ["Head of Total Rewards", "CFO"],
    "budget_signal": "$25k pilot budget approved by finance for this quarter (CFO)",
    "timeline_signal": "Decision by end of September (CFO)",
    "competitor_mentioned": null,
    "next_step": "Send pilot agreement; prospect will route it to legal this week (explicitly agreed)",
    "objections": [
      "Workday integration 'has to be rock solid' — CFO's one condition"
    ],
    "confidence": "high — approved budget, decision date, and agreed next step with CFO in the room"
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "Make recognition visible across 12 retail locations (People Ops Manager)"
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition today"
    ],
    "stakeholders": ["People Ops Manager"],
    "budget_signal": null,
    "timeline_signal": "'No rush on our side until Q1' (People Ops Manager)",
    "competitor_mentioned": "Bucketlist — CEO used it at her last company and liked it",
    "next_step": "Schedule a call with prospect's CEO; People Ops Manager to send two times (explicitly agreed)",
    "objections": [
      "CEO has to be sold first — 'she decides anything people-related'",
      "CEO already likes Bucketlist (competitor affinity)"
    ],
    "confidence": "medium-low — agreed CEO intro call, but no prospect budget signal, no urgency until Q1, and a gatekeeping CEO who likes the competitor"
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "Consolidate three separate recognition tools into one (VP People)"
    ],
    "pain_points": [
      "Paying for three tools and none of them talk to the HRIS"
    ],
    "stakeholders": ["VP People", "IT Security Lead"],
    "budget_signal": "Under $15k annually, VP People can approve without going to the board (approval threshold, not committed spend)",
    "timeline_signal": "Procurement cycle runs six to eight weeks minimum (IT Security Lead)",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "Security review took three months for their last vendor — IT Security Lead's stated hesitation",
      "Anything at or above $15k annually requires board approval"
    ],
    "confidence": "medium — clear pain and a stated approval threshold, but long procurement, security hesitation, and no agreed next step ('no promises')"
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "Automate service milestones (HR Director)",
      "Analytics on recognition equity across departments (HR Director)"
    ],
    "pain_points": [
      "Night-shift teams feel invisible; their engagement scores run 20 points lower (People Ops Coordinator)"
    ],
    "stakeholders": ["HR Director", "People Ops Coordinator"],
    "budget_signal": "$12k approved under the engagement line (HR Director)",
    "timeline_signal": "Needs to be running before the January all-hands",
    "competitor_mentioned": "Nectar — prospect is mid-pilot with Nectar; 'you'd need to beat that experience'",
    "next_step": "Present to prospect's exec team on October 2 (explicitly agreed)",
    "objections": [
      "Exec team is skeptical after a failed rollout two years ago",
      "Must beat the active Nectar pilot experience"
    ],
    "confidence": "high — approved budget, hard deadline, dated exec presentation; active incumbent pilot is the main risk"
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "Cut the admin time on service awards (HR Manager)"
    ],
    "pain_points": [
      "HR Manager personally spends five hours a month ordering and shipping plaques"
    ],
    "stakeholders": ["HR Manager"],
    "budget_signal": "No figure stated; prospect: 'Budget isn't the issue — time is.'",
    "timeline_signal": "Fiscal year resets in February; a Q1 start is realistic",
    "competitor_mentioned": null,
    "next_step": "Send the one-page overview; HR Manager will forward it to the COO this week (explicitly agreed)",
    "objections": [
      "COO usually prefers building things in-house (prospect's 'fair warning')"
    ],
    "confidence": "medium — champion with agreed next step, but no budget figure, Q1 timeline, and COO build-vs-buy risk"
  }
]
```

Explicit gaps and exclusions per your rules:

- TX-003 budget_signal is null: the only pricing in the transcript ($8/employee/month) came from the rep, not the prospect, so it was excluded.
- TX-002, TX-004, TX-006 competitor_mentioned are null: no competitor was named by the prospect. TX-002's "you're the first vendor we've had a real demo with" and TX-006's in-house alternative are not competitor mentions; TX-004's three incumbent tools were never named.
- TX-004 next_step is null: the CFO follow-up was rep-proposed and the VP People said "Maybe — I need to check her calendar, no promises," which is not explicit agreement.
- Stakeholders are limited to prospect speakers as instructed. People mentioned but not on the speaker list (IT lead in TX-001, CEO in TX-003, CFO in TX-004, COO in TX-006) appear only in next_step/objections where the prospect referenced them.
- No arithmetic was required — all dollar figures and dates are direct prospect quotes, restated as given.
next-to-close0.75
Plain-English question: Which 3 deals are closest to signature right now?
Prompt — what the model saw
Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0853 · 50s · in 15,661 / out 2,554 tok
✗ exactly_three✓ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

Plain-English answer: Deal-547B2B ($11,200), Deal-B7EBD1 ($9,000), and Deal-403845 ($9,000) — all DS5 COMMIT deals closing September 10-11, with what is left on each.
blocked_excludeddeal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
exactly_three1
ranking_ruleSlack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
top3rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
graded checksexactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
As of 2026-09-05 (Slack export dated 2026-09-04):

1. Deal-547B2B — $11,200 — Alex Franklin, DS5/COMMIT, CRM close 2026-09-11 (6 days out).
Why close: redlines came back clean, signing page is out, and their VP People said they are signing "tomorrow" = 2026-09-05. Signature-imminent.
Left: the signature itself.

2. Deal-403845 — $9,000 — DS5/COMMIT, CRM close 2026-09-11 (6 days).
Why close: order form is with their finance team and "moving fine."
Left: finance team action on the order form; no signature date given. (Note: Slack update came from Dana Mercer; CRM owner is Alex Franklin.)

3. Deal-A2B47C — $6,360 — Alex Franklin, DS5/COMMIT, CRM close 2026-09-11 (6 days).
Why close: Slack confirms it is "still warm," normal legal-review pace.
Left: legal review to complete, then signature.

Not ranked: Deal-2465CE ($5,400) — CRM still shows DS5/COMMIT for 2026-09-10 (5 days), but Slack supersedes: champion left, procurement froze new vendors, Dana Mercer is pulling it from commit; realistically Q4. Deal-D348E1 is warm but CRM-dated 2026-10-15 (40 days). Deal-B7EBD1 (DS5/COMMIT, 2026-09-10) has no Slack status, so I ranked the Slack-confirmed movers ahead of it.
product-gap-detectionPASS 1.00
Plain-English question: Which prospect complaints are true product gaps versus rollout or enablement issues?
Prompt — what the model saw
Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0256 · 13s · in 1,083 / out 1,202 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

Plain-English answer: Two real gaps (SCIM provisioning, ADP Workforce Now integration) separated from plan gates and enablement issues, each with the prospect quote and deal.
allowed_newTG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
enablement_issuestranscript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voicetranscript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
plan_gatestranscript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
real_gapstranscript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
graded checksreal_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
CANDIDATE REVIEW

TG-001 | Deal-EC3025 | amount: not provided in data
Prospect (IT Security Lead): "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
Classification: REAL GAP. Docs state SCIM user provisioning is "NOT currently listed as supported capabilities." No tier offers it.

TG-002 | Deal-D0D6B5 | amount: not provided in data
Prospect (HRIS Manager): "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
Classification: REAL GAP. Docs state ADP Workforce Now is "NOT currently listed as supported capabilities." Supported HRIS list is Workday, BambooHR, Gusto, Rippling only.

TG-003 | Deal-CFE7F4 | amount: not provided in data
Prospect (VP People): "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?"
Classification: PLAN GATE. Custom report builder exists in the product (Enterprise tier per the docs table). The capability is not missing; the prospect's plan doesn't include it. Excluded from summary.

TG-004 | Deal-84DBA6 | amount: not provided in data
Prospect (People Ops Manager): "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it."
Classification: ROLLOUT/ENABLEMENT ISSUE. Slack integration is supported on all plans; the problem is adoption/training, not product capability. Excluded from summary.

TG-005 | Deal-36C33F | excluded
The only gap statement (no native mobile app) comes from Alex Franklin (rep), not the prospect. Rep language does not count as prospect voice. The prospect's own line ("The web version should be fine for our office staff for now") raises no gap.

SUMMARY — TRUE PRODUCT GAPS (prospect-voiced)

1. SCIM user provisioning — Deal-EC3025 — amount: not provided in data
   "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management." (IT Security Lead)
2. ADP Workforce Now integration — Deal-D0D6B5 — amount: not provided in data
   "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us." (HRIS Manager)

Missing data, stated explicitly:
- Deal amounts: no amount field exists in transcripts_gaps.csv or product_docs.md, so amounts cannot be named.
- Deal status: no field indicates which deals are "active," so all five listed deals were evaluated as candidates.
rep-performance 5 tests
stale-pipeline-by-rep0.67
Plain-English question: Which open deals have had no email, call, or meeting in the last 7 days?
Prompt — what the model saw
The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.5273 · 113s · in 53,127 / out 23,892 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✗ jenn_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

Plain-English answer: 65 open deals are stale; the largest is Deal-2D1F1B at $240,000, with Bryce Harmon carrying the most stale dollars at $626,243.
The answer in one line: jake_amt = 626,243; jenn_amt = 261,645; kevin_amt = 252,905.03; top_amount = 240,000; stale_count = 65; top_alias = Deal-2D1F1B
jake_amt626,243
jenn_amt261,645
kevin_amt252,905.03
stale_count65
top_amount240,000
top_aliasDeal-2D1F1B
graded checksstale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Snapshot 2026-09-05. "Last 7 days" window = 2026-08-29 through 2026-09-05 inclusive. Recency = max(last_email, last_call, last_meeting) from engagements table, ignoring last_meeting values dated AFTER the snapshot (scheduled meetings are not logged activity). Days since last contact = 2026-09-05 minus that date. Two deals (Deal-3EED2C, Deal-57FF13) have no engagement row at all, so days-since-contact cannot be computed — flagged as "no engagement data."

**Bryce Harmon**

| Deal | Stage | Amount | Days since last contact |
|---|---|---|---|
| Deal-2D1F1B | DS1 | $240,000 | 81 (last: 2026-06-16) |
| Deal-66D1FC | DS1 | $99,000 | 16 (last: 2026-08-20) |
| Deal-950043 | DS1 | $70,000 | 19 (last: 2026-08-17) |
| Deal-B23205 | DS1 | $45,000 | 16 (last: 2026-08-20) |
| Deal-7BBDFA | DS3 | $37,440 | 46 (last: 2026-07-21) |
| Deal-332637 | DS2 | $36,000 | 9 (last: 2026-08-27) |
| Deal-1BEEBF | DS1 | $31,500 | 19 (last: 2026-08-17) |
| Deal-A414F6 | DS1 | $25,200 | 19 (last: 2026-08-17) |
| Deal-C5658B | DS1 | $23,400 | 16 (last: 2026-08-20) |
| Deal-40522D | DS3 | $21,000 | 19 (last: 2026-08-17) |
| Deal-C1FA6D | DS1 | $18,000 | 16 (last: 2026-08-20) |
| Deal-01E193 | DS1 | $12,600 | 8 (last: 2026-08-28) |
| Deal-F0EBBB | DS3 | $11,400 | 24 (last: 2026-08-12) |
| Deal-927338 | DS1 | $10,920 | 18 (last: 2026-08-18) |
| Deal-E25A09 | DS1 | $6,000 | 9 (last: 2026-08-27) |
| Deal-C9C286 | DS2 | $5,502 | 9 (last: 2026-08-27) |
| Deal-012CB1 | DS1 | $1 | 23 (last: 2026-08-13) |
| Deal-3795AD | DS2 | $1 | 8 (last: 2026-08-28) |

Bryce Harmon: 18 stale deals, $692,964 total.

**Dana Mercer**

| Deal | Stage | Amount | Days since last contact |
|---|---|---|---|
| Deal-44EA29 | DS2 | $60,000 | 10 (last: 2026-08-26) |
| Deal-E51FB7 | DS2 | $43,875 | 12 (last: 2026-08-24) |
| Deal-B42F46 | DS1 | $27,000 | 19 (last: 2026-08-17) |
| Deal-BA3DDC | DS3 | $23,400 | 15 (last: 2026-08-21) |
| Deal-9DDE86 | DS2 | $20,000 | 15 (last: 2026-08-21) |
| Deal-215CCA | DS3 | $18,900 | 17 (last: 2026-08-19) |
| Deal-5EED42 | DS3 | $16,250 | 11 (last: 2026-08-25) |
| Deal-57887A | DS2 | $15,000 | 8 (last: 2026-08-28) |
| Deal-944310 | DS4 | $10,500 | 33 (last: 2026-08-03) |
| Deal-B7EBD1 | DS5 | $9,000 | 16 (last: 2026-08-20) |
| Deal-3974EB | DS4 | $9,000 | 8 (last: 2026-08-28) |
| Deal-F40F04 | DS2 | $8,100 | 15 (last: 2026-08-21) |
| Deal-7599B8 | DS3 | $7,350 | 18 (last: 2026-08-18) |
| Deal-87DDD1 | DS1 | $5,000 | 19 (last: 2026-08-17) |
| Deal-F336B6 | DS3 | $4,200 | 15 (last: 2026-08-21) |
| Deal-0660B4 | DS4 | $1,920 | 16 (last: 2026-08-20) |

Dana Mercer: 16 stale deals, $279,495 total.

**Cole Ingram**

| Deal | Stage | Amount | Days since last contact |
|---|---|---|---|
| Deal-D04904 | DS2 | $58,529.25 | 11 (last: 2026-08-25) |
| Deal-B25F40 | DS3 | $40,000 | 8 (last: 2026-08-28) |
| Deal-813836 | DS2 | $32,175 | 11 (last: 2026-08-25) |
| Deal-1BA595 | DS2 | $31,750 | 11 (last: 2026-08-25) |
| Deal-CFE1E8 | DS3 | $18,000 | 11 (last: 2026-08-25) |
| Deal-CD47A6 | DS2 | $12,168 | 11 (last: 2026-08-25) |
| Deal-627646 | DS3 | $11,193 | 11 (last: 2026-08-25) |
| Deal-FF809F | DS2 | $7,781.20 | 11 (last: 2026-08-25) |
| Deal-AF932D | DS2 | $7,225.40 | 11 (last: 2026-08-25) |
| Deal-A71728 | DS2 | $6,947.50 | 11 (last: 2026-08-25) |
| Deal-8BC9F5 | DS2 | $5,616 | 10 (last: 2026-08-26) |
| Deal-175395 | DS3 | $4,779.88 | 11 (last: 2026-08-25) |
| Deal-481E24 | DS3 | $4,140 | 10 (last: 2026-08-26) |
| Deal-C7F9BF | DS2 | $3,360 | 11 (last: 2026-08-25) |
| Deal-2F3A66 | DS3 | $3,334.80 | 11 (last: 2026-08-25) |
| Deal-342E96 | DS2 | $2,700 | 24 (last: 2026-08-12) |
| Deal-E568D5 | DS3 | $1,875 | 11 (last: 2026-08-25) |
| Deal-FD9F4E | DS5 | $1,330 | 10 (last: 2026-08-26) |

Cole Ingram: 18 stale deals, $252,905.03 total.

**Alex Franklin**

| Deal | Stage | Amount | Days since last contact |
|---|---|---|---|
| Deal-CC08D1 | DS1 | $24,000 | 16 (last: 2026-08-20) |
| Deal-E73427 | DS3 | $18,000 | 10 (last: 2026-08-26) |
| Deal-885F45 | DS2 | $9,300 | 12 (last: 2026-08-24) |
| Deal-C2FF3C | DS1 | $8,316 | 10 (last: 2026-08-26) |
| Deal-3EED2C | DS2 | $7,200 | no engagement data |
| Deal-0D2F7A | DS3 | $5,100 | 12 (last: 2026-08-24) |
| Deal-6C60D4 | DS3 | $4,800 | 12 (last: 2026-08-24) |
| Deal-13FEBD | DS2 | $4,680 | 12 (last: 2026-08-24) |
| Deal-819506 | DS1 | $4,400 | 8 (last: 2026-08-28) |
| Deal-9D0060 | DS3 | $3,840 | 12 (last: 2026-08-24) |
| Deal-690476 | DS2 | $3,600 | 18 (last: 2026-08-18) |
| Deal-C6D97A | DS4 | $3,240 | 8 (last: 2026-08-28) |
| Deal-EE195F | DS3 | $3,120 | 8 (last: 2026-08-28) |
| Deal-278DEC | DS3 | $2,700 | 8 (last: 2026-08-28) |
| Deal-635B8E | DS3 | $2,600 | 18 (last: 2026-08-18) |
| Deal-6883F3 | DS1 | $2,400 | 16 (last: 2026-08-20) |
| Deal-4A13AD | DS3 | $2,160 | 26 (last: 2026-08-10) |
| Deal-F67D31 | DS2 | $1,800 | 8 (last: 2026-08-28) |
| Deal-5FDCE4 | DS3 | $1,600 | 12 (last: 2026-08-24) |
| Deal-BA571A | DS4 | $1,080 | 18 (last: 2026-08-18) |

Alex Franklin: 20 stale deals, $113,936 total.

**Farid Osman**

| Deal | Stage | Amount | Days since last contact |
|---|---|---|---|
| Deal-8BA24E | DS1 | $2,880 | 8 (last: 2026-08-28) |
| Deal-8FDCD2 | DS1 | $1 | 15 (last: 2026-08-21) |

Farid Osman: 2 stale deals, $2,881 total.

**Elena Sinclair**

| Deal | Stage | Amount | Days since last contact |
|---|---|---|---|
| Deal-57FF13 | DS1 | $2,100 | no engagement data |

Elena Sinclair: 1 stale deal, $2,100 total.

Total: 75 stale open deals, $1,344,281.03 across 6 owners.
activity-mix-vs-outcomePASS 1.00
Plain-English question: Per rep, what did the last 30 days of emails, calls, and meetings look like next to the deals that advanced?
Prompt — what the model saw
The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.3393 · 344s · in 30,419 / out 16,505 tok
✓ alex_ds2_30d✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

Plain-English answer: Alex Franklin is the standout signal with 18 deals entering DS2 in 30 days (next closest: Bryce Harmon with 4), alongside per-rep activity totals.
The answer in one line: alex_ds2_30d = 18; bryce_ds2_30d = 4; note = legacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
alex_ds2_30d18
bryce_ds2_30d4
notelegacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
graded checksalex_ds2_30d · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
SNAPSHOT: 2026-09-05 | 30-day window: 2026-08-06 through 2026-09-05 (inclusive) | Activities = emails_30d + calls_30d + meetings_30d from engagements table, aggregated per owner across their deals.

PER-REP TOTALS (last 30 days)
==============================

Alex Franklin (84342457)
  Emails 307, Calls 36, Meetings 41 -> Total 384
  Mix: Emails 307/384 = 79.9% | Calls 36/384 = 9.4% | Meetings 41/384 = 10.7%
  DS2 entries in window: 18
    Deal-403845 (09-02), Deal-1FC049 (09-03), Deal-3EED2C (09-03), Deal-7FA0C3 (08-07), Deal-E531A6 (08-07), Deal-5296C9 (08-28), Deal-36C33F (08-11), Deal-EE195F (08-06), Deal-F436DA (08-19), Deal-317E6F (08-12), Deal-D1E6C2 (08-11), Deal-D9A72E (08-06), Deal-CA5E44 (08-24), Deal-4F775F (08-17), Deal-898FC5 (08-28), Deal-46988D (08-26), Deal-E73427 (08-28), Deal-92D97D (09-02)
  Activities per DS2 entry: 384/18 = 21.3

Bryce Harmon (119337721)
  Emails 162, Calls 0, Meetings 43 -> Total 205
  Mix: Emails 162/205 = 79.0% | Calls 0/205 = 0.0% | Meetings 43/205 = 21.0%
  DS2 entries in window: 4
    Deal-25F752 (08-10), Deal-CA7DC0 (08-12), Deal-1CCE5C (08-06), Deal-D73B89 (09-03)
  Activities per DS2 entry: 205/4 = 51.3

Cole Ingram (83155924)
  Emails 96, Calls 14, Meetings 1 -> Total 111
  Mix: Emails 96/111 = 86.5% | Calls 14/111 = 12.6% | Meetings 1/111 = 0.9%
  DS2 entries in window: 2
    Deal-42326B (08-26), Deal-1BA595 (08-12)
  Activities per DS2 entry: 111/2 = 55.5

Farid Osman (716654662)
  Emails 38, Calls 0, Meetings 34 -> Total 72
  Mix: Emails 38/72 = 52.8% | Calls 0/72 = 0.0% | Meetings 34/72 = 47.2%
  DS2 entries in window: 1
    Deal-499BF6 (08-26)
  Activities per DS2 entry: 72/1 = 72.0

Dana Mercer (83155923)
  Emails 84, Calls 18, Meetings 11 -> Total 113
  Mix: Emails 84/113 = 74.3% | Calls 18/113 = 15.9% | Meetings 11/113 = 9.7%
  DS2 entries in window: 1
    Deal-57887A (08-07)
  Activities per DS2 entry: 113/1 = 113.0

Elena Sinclair (701163055)
  One deal (Deal-57FF13, DS1, no t_ds2). No row in the engagements table -> 0 emails, 0 calls, 0 meetings.
  DS2 entries: 0. Ratio undefined (0/0) — excluded from ranking.

Gavin Porter (1520255671) and Hugo Lindqvist (77260721): archived, no deals in the file — no data to compute.

EFFICIENCY RANKING (lowest activities per DS2 entry = most efficient)
=====================================================================
  1. Alex Franklin        21.3   (384 / 18)
  2. Bryce Harmon           51.3   (205 / 4)
  3. Cole Ingram   55.5   (111 / 2)
  4. Farid Osman          72.0   (72 / 1)
  5. Dana Mercer          113.0   (113 / 1)

ANSWER
======
Most efficient rep: Alex Franklin (21.3 activities per DS2 entry).
Highest-volume rep: Alex Franklin (384 total activities).
They do NOT differ — the same rep is both most efficient and highest volume.

DATA NOTES
==========
- Boundary convention: "within 30 days" treated as t_ds2 >= 2026-08-06. Deals at exactly 2026-08-06 are included (Deal-1CCE5C, Deal-EE195F, Deal-D9A72E); Deal-55164C at 2026-08-05 and Deal-13FEBD at 2026-08-04 are excluded. If the boundary were exclusive, ratios become Bryce 205/3 = 68.3 and Alex 384/16 = 24.0; the ranking is unchanged.
- Deal-3EED2C (Alex Franklin) has no row in the engagements table; counted as 0 activities.
- Bryce Harmon and Farid Osman logged zero calls in the window; Farid's mix is meeting-heavy (47.2% meetings vs ~10% or less for email-led reps).
- No cause attributed to any difference in volume, mix, or efficiency.
ae-qtd-scorecardPASS 1.00
Plain-English question: Build Alex Franklin's QTD scorecard: bookings versus quota, pipeline, and conversion.
Prompt — what the model saw
Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.3805 · 222s · in 20,494 / out 20,519 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

Plain-English answer: 75% attainment ($150,000 on a $200,000 quota), $113,500 new versus $36,500 expansion, pipeline by stage, and the rolling DS2-to-won rate.
The answer in one line: quota = 200000; bookings = 150000; new_bookings = 113500; expansion_bookings = 36500; open_deals_count = 125; attainment_pct = 75; owner = Alex Franklin (+12 more below)
quota200,000
attainment_pct75
bookings150,000
expansion_bookings36,500
expansion_count3
losses_qtd27
new_bookings113,500
new_count5
open_deals_count125
wins_qtd8
activity_30d_totalsemails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
allowed_new18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
ds2_to_wonwindow_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
ownerAlex Franklin
pipeline_by_stage{"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
prior_quarter_excludeddeal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
quarter2026-Q3
snapshot_date2026-09-05
top_loss_reasonreason=Lost- Timing (1 year or more); count=13
graded checksbookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD SCORECARD — Alex Franklin — as of 2026-09-05 (2026-Q3: 07-01 → 09-30)

1. BOOKINGS VS QUOTA
   Quota: $200,000
   QTD bookings: $150,000 across 8 closed-won deals
     Deal-A1C3E5 $40,000 (07-15) + Deal-F2C7D8 $20,000 (07-24) + Deal-B7D2F4 $35,000 (07-31)
     + Deal-C9E1A6 $21,000 (08-12) + Deal-A8B4D6 $12,000 (08-19) + Deal-D4B8C2 $11,000 (08-21)
     + Deal-E6F3A9 $6,500 (09-02) + Deal-C5D9E2 $4,500 (09-03) = $150,000
   Attainment: 150,000 / 200,000 = 75.0%
   Excluded per instructions: Deal-B3E6F1 $24,000, closed-won 2026-06-20 — before Q3 start, not counted.

2. NEW VS EXPANSION SPLIT (of $150,000)
   New:       $113,500 (5 deals: Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2, Deal-E6F3A9) — 75.7% (113,500/150,000)
   Expansion:  $36,500 (3 deals: Deal-F2C7D8, Deal-A8B4D6, Deal-C5D9E2) — 24.3% (36,500/150,000)

3. ACTIVE PIPELINE BY STAGE (open deals, all close dates)
   DS1: 20 deals,  $284,621
   DS2: 28 deals,  $353,760
   DS3: 67 deals,  $552,705
   DS4:  5 deals,   $23,574
   DS5:  5 deals,   $45,730
   Total: 125 open deals, $1,260,390
   Note: only 22 of these ($109,363) carry a close_date on or before 09-30; $1,151,027 is dated in Q4 or later.

4. ROLLING 90-DAY DS2-TO-WON RATE (window 2026-06-07 → 2026-09-05)
   Cohort: deals with entered_ds2 in window that have a closed outcome.
   Won: 8 (the 8 QTD wins) | Lost: 27 | Decided: 35
   Rate: 8 / 35 = 22.9%
   (If measured on all deals decided in the window regardless of DS2 entry date: 9 won / 36 decided = 25.0%.)

5. WIN/LOSS COUNTS AND TOP LOSS REASON
   Wins QTD: 8 | Losses QTD: 27 (all 27 closed-lost fall in Q3; total lost value $329,272 vs $150,000 won)
   Loss reasons: Lost- Timing (1 year or more) 13 deals / $184,681 — TOP REASON (48% of losses: 13/27)
                 Competitor 5 / $49,020 · MIA 5 / $45,831 · Lost DM 2 / $17,940
                 Feature Request 1 / $21,000 · Lost- Does not fit ICP 1 / $10,800

6. ACTIVITY VOLUME, LAST 30 DAYS (161 deal rows in ae_engagements.csv)
   Emails: 807 | Meetings: 128 | Calls: 112 | Notes: 50 | Total: 1,097
   Mix: 73.5% email, 11.7% meetings, 10.2% calls, 4.6% notes.

THREE COACHING OBSERVATIONS

1. Attainment risk is real, and September won't save it. 75.0% attained with $50,000 to go, but late-stage cover is thin: DS4+DS5 totals just $69,304 (10 deals) across ALL close dates, and only $34,204 of it (6 deals) closes by 09-30. Pipeline coverage on the remaining gap is ~1.4x using total late-stage ($69,304/$50,000) and just 0.68x using in-quarter late-stage — well below the ~3x rule of thumb. The four September-closing DS5/DS4 deals with zero 30-day calls or meetings — Deal-547B2B ($11,200, 0 meetings), Deal-A2B47C ($6,360, 0 meetings), Deal-F9A08A ($2,484, 14 emails but 0 calls/meetings) — need direct engagement now, not email sequences.

2. Losses are a qualification and multi-threading problem, not an activity problem. QTD loss count runs 27 against 8 wins (a 23% QTD win rate on closed deals), and the two biggest controllable buckets — MIA (5 deals, $45,831) and Lost DM (2 deals, $17,940) — point to single-threaded deals going dark. The activity data supports this: 73.5% of all touches are emails. Compare the 8 QTD wins, which averaged 11.1 emails + 3.9 calls + 2.9 meetings in 30 days, against late-stage open deals riding on email alone. The dominant reason, Lost- Timing (13 deals, $184,681 — including the $40,001 Deal-B3ABED), suggests deals entering DS2 without validated urgency; tighter exit criteria at DS1→DS2 would protect cycle time.

3. The pipeline is large but stale and back-weighted. $1.26M open looks healthy, but 91% of it ($1,151,027) is dated Q4+, and 29 open DS2+ deals worth $282,960 entered DS2 more than 90 days ago — several from Q1 (Deal-F9A08A entered DS2 2025-11-24 and is still sitting in DS4; Deal-60C2C2 $19,000 in DS3 since 2026-02-23). Given a 22.9% rolling DS2-to-won rate, these aged deals are statistically more likely to be future "Timing" losses than future bookings. Recommend a formal pipeline scrub: re-qualify or close out the 29 aged deals, and rebuild Q4 pipeline with deals created in-quarter, where his win rate is demonstrably stronger.
multithreading-gapPASS 1.00
Plain-English question: Which open deals are single-threaded or under-threaded?
Prompt — what the model saw
Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.1269 · 66s · in 4,606 / out 6,927 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

Plain-English answer: Deals with fewer than 2 active contacts (single-threaded) or fewer than 3, or all in one persona (under-threaded), each flagged with the missing persona.
The answer in one line: active_window_start = 2026-07-07
active_window_start2026-07-07
allowed_new59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
best_persona_ruleDS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flag_ruleflag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
flaggeddeal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flaggedDeal-4B0BEB, Deal-84DBA6, Deal-D348E1
persona_universeeconomic buyer, champion, HR admin, IT security, finance
graded checksflagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Reference date: 2026-09-05 (today). 60-day active window = last_engaged_date on/after 2026-07-07.

Data gaps stated up front: deal amount and deal stage do not exist in either file, so those fields are "not provided" for every deal below. Because stage is unavailable, the persona-priority rule applied is: if the deal has no active economic buyer, that is the most valuable add (no deal closes without budget authority); otherwise IT security (most common late-stage blocker in HR-tech sales). If you can supply stages, I'll re-rank.

10 of 13 open deals flagged. 3 not flagged (Deal-84DBA6, Deal-4B0BEB, Deal-D348E1: each has 3+ active contacts across 3+ personas).

FLAGGED DEALS

1. Deal-EC3025 (C-FDD0C7) — single-threaded
   Amount: not provided. Stage: not provided.
   Active contacts: 1 (CT-047C54, champion, 2026-09-02)
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: CT-6827DB, Chief People Officer, economic buyer — yes, on file.
   Note: CT-F2C1AE (economic buyer) is marked former and does not count.

2. Deal-92D97D (C-E23238) — single-threaded
   Amount: not provided. Stage: not provided.
   Active contacts: 1 (CT-01F5B4, HR admin, 2026-08-28)
   Personas present: HR admin
   Personas missing: economic buyer, champion, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: none on file.
   Note: CT-A902AE (champion) last engaged 2026-06-01, 96 days out — outside the 60-day window.

3. Deal-50D386 (C-EB10E4) — under-threaded (2 active)
   Amount: not provided. Stage: not provided.
   Active contacts: 2 (CT-AA41B2 champion 2026-09-01; CT-B9C35B HR admin 2026-08-25)
   Personas present: champion, HR admin
   Personas missing: economic buyer, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: CT-A1C4B3, Chief People Officer, economic buyer — yes, on file.

4. Deal-D0D6B5 (C-32918E) — under-threaded (3 active, all one persona)
   Amount: not provided. Stage: not provided.
   Active contacts: 3 (CT-87CED4, CT-DE6D7C, CT-FD70B2 — all champion)
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: CT-1FA4DB, Chief People Officer, economic buyer — yes, on file.

5. Deal-5BFE3B (C-535D36) — under-threaded (2 active, one persona)
   Amount: not provided. Stage: not provided.
   Active contacts: 2 (CT-57123B, CT-5CE757 — both champion)
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: none on file.

6. Deal-36C33F (C-077A0E) — single-threaded
   Amount: not provided. Stage: not provided.
   Active contacts: 1 (CT-4FE556, IT security, 2026-08-15)
   Personas present: IT security
   Personas missing: economic buyer, champion, HR admin, finance
   Priority add: economic buyer
   On-file unengaged fit: CT-1DB73E, Chief People Officer, economic buyer — yes, on file.
   Note: CT-405B45 (champion) and CT-86B22F (economic buyer) are both marked former.

7. Deal-885F45 (C-5E8EFB) — under-threaded (2 active)
   Amount: not provided. Stage: not provided.
   Active contacts: 2 (CT-51C81E economic buyer 2026-08-26; CT-D9A0E8 champion 2026-08-11)
   Personas present: economic buyer, champion
   Personas missing: HR admin, IT security, finance
   Priority add: IT security (EB already covered; security review is the typical blocker)
   On-file unengaged fit: CT-B3F25D, IT Security Lead, IT security — yes, on file.

8. Deal-FCBE5B (C-737030) — single-threaded
   Amount: not provided. Stage: not provided.
   Active contacts: 1 (CT-4A5317, champion, 2026-08-29)
   Personas present: champion
   Personas missing: economic buyer, HR admin, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: none on file.

9. Deal-5408B0 (C-2AE3AA) — under-threaded (2 active)
   Amount: not provided. Stage: not provided.
   Active contacts: 2 (CT-D33AE4 champion 2026-09-01; CT-8742FD HR admin 2026-08-18)
   Personas present: champion, HR admin
   Personas missing: economic buyer, IT security, finance
   Priority add: economic buyer
   On-file unengaged fit: CT-07FA76, Chief People Officer, economic buyer — yes, on file.

10. Deal-F9A08A (C-0D15DF) — single-threaded
    Amount: not provided. Stage: not provided.
    Active contacts: 1 (CT-931B10, champion, 2026-09-03)
    Personas present: champion
    Personas missing: economic buyer, HR admin, IT security, finance
    Priority add: economic buyer
    On-file unengaged fit: CT-697541, Chief People Officer, economic buyer — yes, on file.
    Note: CT-913581 (economic buyer) last engaged 2026-06-20, 77 days out — outside the window.

11. Deal-C6D97A (C-5A8FC2) — under-threaded (3 active, all one persona)
    Amount: not provided. Stage: not provided.
    Active contacts: 3 (CT-223DDC, CT-B03555, CT-4E8A2B — all champion)
    Personas present: champion
    Personas missing: economic buyer, HR admin, IT security, finance
    Priority add: economic buyer
    On-file unengaged fit: none on file.

SUMMARY
- Single-threaded (<2 active): Deal-EC3025, Deal-92D97D, Deal-36C33F, Deal-FCBE5B, Deal-F9A08A (5 deals)
- Under-threaded (<3 active or one persona): Deal-50D386, Deal-D0D6B5, Deal-5BFE3B, Deal-885F45, Deal-5408B0, Deal-C6D97A (6 deals)
- 9 of 10 flagged deals lack an active economic buyer — the dominant gap.
- 7 of 10 flagged deals have an on-file unengaged contact matching the priority persona; 3 do not (Deal-92D97D, Deal-5BFE3B, Deal-C6D97A).
- Unused on-file contacts: CT-2A0169 (finance, C-BE9BE0) and CT-8E04F5 (HR admin, C-804F29) — both companies' deals are already multi-threaded, so no action flagged.
talk-track-from-transcriptsPASS 1.00
Plain-English question: From this rep's last 10 calls: how do they open, handle objections, and how often do they lock a next step?
Prompt — what the model saw
From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0681 · 40s · in 3,014 / out 3,649 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

Plain-English answer: 8 of 10 calls open with the same customer-win story; objection handling, next-step agreement rate, and every competitor raised, with quotes.
The answer in one line: n_transcripts = 10; objection_count = 3; rep = Alex Franklin (+3 more below)
n_transcripts10
objection_count3
allowed_new13 items — e.g. TT-001, TT-002, TT-003, TT-004
competitor_quotesAwardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
competitors_prospect_raisedAwardco, Kudos
next_stepagreed=7; total=10; rate=0.7
objectionsobjection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
opener_themetheme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
repAlex Franklin
rep_raised_competitors_excludedWorkhuman
graded checksobjection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
CALL ANALYSIS — Alex Franklin, 10 calls (TT-001 through TT-010, 2026-07-28 to 2026-09-03)

1) WHAT HE LEADS WITH (first five minutes)

Primary opener — 8 of 10 calls (Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-EDC141, Deal-D9A12F, Deal-84DBA6): an identical customer proof point.
Quote: "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."

Exceptions:
- Deal-403845 (TT-004), agenda-led: "I put together a short agenda — security review first, then pricing."
- Deal-1E2498 (TT-009), pricing-first: "You asked for straight pricing last time, so let's start there."
- Note: on Deal-C61CF7 (TT-005) he adds unprompted competitive positioning at minute 2 ("And unlike Workhuman..."), the only call where he names a competitor himself.

2) THREE MOST COMMON OBJECTIONS AND HANDLING

a) Budget locked — raised 4 times (Deal-D348E1, Deal-547B2B, Deal-60C2C2, Deal-84DBA6). Handled identically all 4 times with a turnover-savings funding reframe:
"Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."

b) Timing / revisit next quarter — raised 3 times (Deal-5408B0, Deal-C61CF7, Deal-D9A12F). Handled identically with a 90-day pilot offer:
"Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"

c) Status quo (spreadsheet + gift cards) — raised 3 times (Deal-403845, Deal-EDC141, Deal-1E2498). Handled identically with an automation/analytics differentiation:
"Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

Also present but outside the top 3: committee sign-off (Deal-403845, Deal-84DBA6 — 2x) and no urgency (Deal-EDC141 — 1x). Both were met with passive acceptance and no counter ("Understood — I'll leave it with you." / "Fair enough." / "Understood, thanks for the candor.").

3) CONCRETE NEXT-STEP RATE

He asks to lock a working session in 7 calls (Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-D9A12F, Deal-1E2498) and all 7 accept.
Quote: "Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager."
No next step in 3 calls: Deal-403845, Deal-EDC141, Deal-84DBA6.
Rate: 7 / 10 = 70%.

4) COMPETITORS RAISED BY PROSPECTS

- Awardco — Deal-547B2B (TT-003): "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
- Kudos — Deal-EDC141 (TT-007): "How are you different from Kudos? Our CEO used them at her last company."
- Workhuman is NOT prospect-raised; it appears only in the rep's own line on Deal-C61CF7 (TT-005, minute 2). No other competitors appear in the transcripts.

COACHING NOTES

1. There is no play for committee/no-urgency stalls. All 3 calls without a next step (Deal-403845, Deal-EDC141, Deal-84DBA6) ended with passive acceptance of those exact objections, which accounts for the entire 30% gap in next-step rate. Coach a counter — e.g., offer to arm the champion for the committee or attach a date to "think about it" — instead of closing the call.

2. Don't volunteer competitors. His prospect-raised competitive defenses (Awardco, Kudos) are crisp, but on Deal-C61CF7 he introduced Workhuman himself at minute 2 unprompted, which plants a comparison the prospect never made. Hold differentiation until the prospect names the alternative.
reporting-analytics 5 tests
q3-weighted-forecastPASS 1.00
Plain-English question: Build the Q3 2026 weighted forecast from the open pipeline, and show the math deal by deal.
Prompt — what the model saw
Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.2153 · 115s · in 20,857 / out 9,837 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

Plain-English answer: A stage-weighted Q3 forecast number computed from the open pipeline, with per-stage math that reconciles to the deal data.
The answer in one line: excluded_amount = 227575; best_case_total = 203565; weighted_forecast = 115,976.75; commit_total = 44729; excluded_out_of_quarter = 32; best_case_count = 24; flags = blank owner column + empty why_buys fields must be surfaced (+2 more below)
excluded_amount227,575
best_case_count24
best_case_total203,565
commit_count7
commit_total44,729
excluded_out_of_quarter32
weighted_forecast115,976.75
flagsblank owner column + empty why_buys fields must be surfaced
top5_best_caseDeal-2D7423, Deal-25F752, Deal-E53952, Deal-5EED42, Deal-FA32A0
graded checkscommit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
# Q3 2026 Forecast (close dates 2026-07-01 to 2026-09-30)

## COMMIT (inside quarter): 7 deals, $44,729.00
Arithmetic: 11,200 (Deal-547B2B) + 9,000 (Deal-B7EBD1) + 9,000 (Deal-403845) + 6,360 (Deal-A2B47C) + 5,400 (Deal-2465CE) + 2,520 (Deal-A5E80A) + 1,249 (Deal-499BF6) = **$44,729.00**

## BEST_CASE (inside quarter): 24 deals, $203,565.00
Arithmetic: 38,935 + 24,000 + 19,656 + 16,250 + 11,116 + 10,800 + 10,500 + 9,890 + 9,720 + 9,000 + 7,200 + 5,400* + 3,840 + 3,780 + 3,600 + 3,240 + 3,150 + 3,120 + 3,060 + 2,916 + 2,760 + 2,484 + 1,920 + 528 = **$203,565.00**
(*Deal-2D7423, Deal-25F752, Deal-E53952, Deal-5EED42, Deal-FA32A0, Deal-FC22A3, Deal-944310, Deal-5195DB, Deal-180D02, Deal-3974EB, Deal-5D8CEE, Deal-357C30, Deal-9D0060, Deal-46988D, Deal-C6D97A, Deal-DAF1D9, Deal-EE195F, Deal-55164C, Deal-001FF4, Deal-7B3B0F, Deal-F9A08A, Deal-1FC049, Deal-8952F0, Deal-87412C — the 5,400 is Deal-357C30? No: Deal-357C30 is 3,600; the 5,400-slot is actually absent — see top-5 list for the largest five; the 24 deals above sum to 203,565.00 exactly as listed.)

Correction for clarity — the 24 in-quarter BEST_CASE deals are: Deal-2D7423 (38,935), Deal-25F752 (24,000), Deal-E53952 (19,656), Deal-5EED42 (16,250), Deal-FA32A0 (11,116), Deal-FC22A3 (10,800), Deal-944310 (10,500), Deal-5195DB (9,890), Deal-180D02 (9,720), Deal-3974EB (9,000), Deal-5D8CEE (7,200), Deal-9D0060 (3,840), Deal-46988D (3,780), Deal-357C30 (3,600), Deal-C6D97A (3,240), Deal-DAF1D9 (3,150), Deal-EE195F (3,120), Deal-55164C (3,060), Deal-001FF4 (2,916), Deal-7B3B0F (2,760), Deal-F9A08A (2,484), Deal-1FC049 (1,920), Deal-8952F0 (2,100), Deal-87412C (528). Sum = $203,565.00.

## Weighted forecast
100% x COMMIT + 35% x BEST_CASE = 44,729.00 + 0.35 x 203,565.00
= 44,729.00 + 71,247.75
= **$115,976.75**

## Deal counts inside the quarter
- COMMIT: 7
- BEST_CASE: 24
- PIPELINE: 23 (count $0 in the forecast per the rule)
- Total inside quarter: 54 deals

## Excluded for close date outside the quarter
- Count: 32 deals
- Total amount: **$227,575.00** (includes COMMIT Deal-D348E1, $13,770, close 2026-10-15, and BEST_CASE deals Deal-C61CF7, Deal-48B656, Deal-901332, Deal-47AE31, Deal-15D24F, Deal-ED725A, Deal-8AD4A5, Deal-5FDCE4, Deal-F5A622)

## Top 5 BEST_CASE deals by amount (inside the quarter)
1. Deal-2D7423 — $38,935 (close 2026-09-30)
2. Deal-25F752 — $24,000 (close 2026-09-25)
3. Deal-E53952 — $19,656 (close 2026-09-30)
4. Deal-5EED42 — $16,250 (close 2026-09-30)
5. Deal-FA32A0 — $11,116 (close 2026-09-25)

## Data quality
The owner field is blank on 85 of 86 deals (only Deal-C9C286 has an owner, Bryce Harmon), so there is no way to attribute or chase any deal. 71 of 86 deals have why_buys_chars = 0, meaning no documented buying rationale behind most category calls, including several BEST_CASE deals. Close dates pile up suspiciously on 2026-09-30 (17 deals), a pattern consistent with default/placeholder quarter-end dates rather than real customer commitments. Two COMMIT deals sit in early stages (Deal-A5E80A at DS1, Deal-499BF6 at DS2), which contradicts normal stage/category hygiene, and one amount is fractional (Deal-42326B at $2,480.40), suggesting inconsistent amount handling in the extract.
aha-moment-2x2PASS 1.00
Plain-English question: Do companies that hit 5+ givers and one redemption in month one actually retain better at 24 months?
Prompt — what the model saw
Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.3194 · 273s · in 35,345 / out 13,754 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

Plain-English answer: The four cohort cells are graded against computed 24-month retention; the numbers come from the scorer's cohort table, not opinion.
The answer in one line: both_5plus_givers_and_redemption = 66.0% (31 of 47)
both_5plus_givers_and_redemption66.0% (31 of 47)
exclusionsnone — claim bounded as correlation, not causation
givers_only46.9% (23 of 49)
largest_single_signal_liftunique givers
neither40.0% (38 of 95)
redemption_only31.0% (9 of 29)
graded checksboth_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
ACTIVATION HYPOTHESIS TEST — 2023 PLG company cohort
Definition: retained at 24 months = current_status 'active'. Signals use first-calendar-month fields m1_users >= 5 (5+ unique givers) and m1_redemptions >= 1.

Denominator / exclusions
- 220 companies in the extract; 0 excluded. company_key is unique across all 220 rows, and the four key fields (company_key, signup_month, current_status, m1_users/m1_redemptions) are non-null and parseable on every row.
- 3 companies have current_status = 'non_renewing' (C-0B2078FB, C-0A96134F, C-0BEAF685). Per the stated definition, only 'active' counts as retained, so these are counted in the denominator as NOT retained, not excluded.
- Data quirks kept as-is (fields used exactly as given): 6 rows have m1_redemptions > m1_gives (C-0FD34543, C-0D78BCB2, C-108C5CE4, C-0FEB5AAF, C-0F58EBD7, C-0D1104A2) and 1 row has m1_users = 0 with m1_redemptions = 10 (C-F0A562FF, lands in redemption-only).

2x2 RESULTS (n and 24-month retention rate)

  Both signals (m1_users>=5 AND m1_redemptions>=1):
    n = 47, retained = 31 -> 31/47 = 66.0%
  Givers-only (m1_users>=5, m1_redemptions=0):
    n = 49, retained = 23 -> 23/49 = 46.9%
  Redemption-only (m1_users<5, m1_redemptions>=1):
    n = 29, retained = 9 -> 9/29 = 31.0%
  Neither:
    n = 95, retained = 38 -> 38/95 = 40.0%

  Check: 47 + 49 + 29 + 95 = 220. Retained total 31+23+9+38 = 101 = count of 'active' rows.

Lifts vs the 'neither' baseline (40.0%)
- Givers-only: 46.9% - 40.0% = +6.9 pp
- Redemption-only: 31.0% - 40.0% = -9.0 pp (NEGATIVE — worse than neither)
- Both: 66.0% - 40.0% = +26.0 pp
- Marginal contribution: adding the redemption signal on top of 5+ givers: 66.0% - 46.9% = +19.0 pp; adding the 5+ givers signal on top of a redemption: 66.0% - 31.0% = +34.9 pp

Single signal with the largest retention lift: 5+ unique givers (m1_users >= 5). Alone it lifts retention +6.9 pp over baseline, and it is also the stronger marginal contributor (+34.9 pp when added to a redemption, vs +19.0 pp for the reverse). A redemption without breadth of givers is actually associated with WORSE-than-baseline retention (31.0% vs 40.0%).

What this proves
- In this cohort, the combined activation state is strongly associated with 24-month retention: 66.0% vs 40.0% baseline, a +26.0 pp gap. The hypothesis "both signals retain better than one or neither" is directionally supported, with the caveat that givers-only (46.9%) also beats neither, so the effect is not exclusive to the combined state — but the combined cohort is clearly the best-retained cell, and redemption-only is the worst.

What it does NOT prove
- Causation. This is an observational correlation in one extract. Companies reaching 5+ givers may simply be better-fit, larger, or better-onboarded accounts that would have retained anyway (self-selection).
- It does not isolate which signal drives the outcome — the givers signal and the redemption signal overlap and interact, and with only 29 redemption-only companies (9 retained) that cell is small, so its -9.0 pp could swing materially with a few accounts.
- It ignores confounders visible in the file itself: tier (the extract is mostly tier_three actives vs tier_two cancellations), geography, industry, and integrations (has_slack etc.) are not controlled for.
- The 3 'non_renewing' accounts are treated as not-retained per your definition; treating them as retained would shift rates slightly (redemption-only most, since 2 of the 3 sit there).
- No statistical-significance test was requested or run; at n=220 with cells of 29–95, the both-vs-neither gap is large, but the single-signal cells are too small to draw firm conclusions about the redemption signal alone.
arr-rollforward-reconciliationPASS 1.00
Plain-English question: Does CRM ARR match billing ARR — and if not, exactly where is the variance?
Prompt — what the model saw
Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.2351 · 86s · in 4,986 / out 14,074 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

Plain-English answer: Billing $604,739.28 vs CRM $603,581.76: a $1,157.52 variance decomposed into status-mismatch, rounding, and missing-record buckets.
The answer in one line: n_companies = 40; n_mismatched_accounts = 7; as_of = 2026-09-05 (+4 more below)
n_companies40
n_mismatched_accounts7
as_of2026-09-05
billing_arr_total604739.28
bucket_sum_equals_variance1
buckets{"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
crm_arr_total603581.76
ruleterm_months != 12 requires cf_agreement_end_date
variance1157.52
variance_definitionbilling_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
violationssubscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
graded checksbilling_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
ARR RECONCILIATION — CRM vs Billing as of 2026-09-05

TOTALS
Billing ARR (37 active subs; ARR = MRR x 12):
  Total active MRR = $50,394.94  ->  $50,394.94 x 12 = $604,739.28
CRM ARR (39 company records, sum of hubspot_arr): $603,581.76

VARIANCE = CRM - Billing = $603,581.76 - $604,739.28 = -$1,157.52
(CRM understates billing by $1,157.52; cancelled subs excluded from billing but still carried in CRM.)

VARIANCE DECOMPOSITION (sums exactly to -$1,157.52)
1. Status mismatch (cancelled in billing, still in CRM): +$13,158.48
   - SUB-000E C-0C8323BF: billing $0.00 (cancelled) vs CRM $4,905.24 -> +4,905.24
   - SUB-000F C-0DC4FB8C: billing $0.00 (cancelled) vs CRM $8,253.24 -> +8,253.24
   - 4,905.24 + 8,253.24 = 13,158.48
2. Rounding: +$36.00
   - SUB-0005 C-0D66DF9E: CRM 23,200.00 - billing 23,184.00 = +16.00
   - SUB-0008 C-14D70CE0: CRM 18,200.00 - billing 18,180.00 = +20.00
   - 16.00 + 20.00 = 36.00
3. Missing records: -$11,952.00
   - SUB-0004 C-21629AA4: billing 2,370.77 x 12 = 28,449.24; no CRM record -> -28,449.24
   - C-0D5BBE3A: CRM 16,497.24; no billing subscription -> +16,497.24
   - 16,497.24 - 28,449.24 = -11,952.00
4. Other (ARR/content mismatch, active sub, CRM record exists): -$2,400.00
   - SUB-0006 C-0F7269D7: billing 2,233.00 x 12 = 26,796.00 vs CRM 24,396.00 = -2,400.00
     (exactly $200.00/month for 12 months — likely one component of the sub never written to CRM)

Check: 13,158.48 + 36.00 - 11,952.00 - 2,400.00 = -1,157.52 ✓

MISMATCHED ACCOUNTS AND SUGGESTED OWNER
(Suggested owners are role-based inferences; no owner data was provided in either file.)
- SUB-000E / C-0C8323BF — cancelled sub still in CRM — Sales Ops (remove/refresh CRM record)
- SUB-000F / C-0DC4FB8C — cancelled sub still in CRM — Sales Ops
- SUB-0004 / C-21629AA4 — billing sub with no CRM record — CRM/Data Admin (create company record)
- C-0D5BBE3A — CRM record with no billing subscription — Billing Ops (verify/create subscription)
- SUB-0005 / C-0D66DF9E — $16.00/yr rounding gap — Deal Desk (align quoted vs billed amount)
- SUB-0008 / C-14D70CE0 — $20.00/yr rounding gap — Deal Desk
- SUB-0006 / C-0F7269D7 — $2,400.00/yr ARR mismatch — Finance Controller (validate which side is authoritative)

BUSINESS-RULE VIOLATIONS (term ≠ 12 months requires cf_agreement_end_date)
- SUB-0002 / C-1794A52C — term 24 months, cf_agreement_end_date blank — VIOLATION
- SUB-0019 / C-22170CA1 — term 36 months, cf_agreement_end_date blank — VIOLATION

Compliant non-12-month subs (for completeness): SUB-000C / C-0DB48281 (24 mo, 2027-11-30); SUB-001A / C-0FC4DBB8 (36 mo, 2027-11-30).

Note: per the stated rule "Billing ARR is MRR x 12", multi-year subs are annualized as MRR x 12; no term-length adjustment was applied.
kvm-month-over-monthPASS 1.00
Plain-English question: How did the four core KVMs move in August versus July?
Prompt — what the model saw
Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0840 · 23s · in 6,431 / out 3,691 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

Plain-English answer: Giving rate 60.27% (up 0.07%), redemptions per user 1.7302 (up 0.01%), with 1:1 meetings and pulse engagement each up fractionally — direction reported for all four.
The answer in one line: n_company_rows_per_month = 30; aggregation = simple mean of company-level values per month
n_company_rows_per_month30
aggregationsimple mean of company-level values per month
biggest_moverpulse_engagement
driving_segmentsize_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
kvms{"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
segment_detail{"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
graded checksfour_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
KVM report: 2026-08 vs 2026-07 (n = 30 companies, simple average per month; arithmetic shown in table).

| KVM | 2026-07 | 2026-08 | Abs change | Rel change | Direction |
|---|---|---|---|---|---|
| Giving rate | 0.6023 | 0.6027 | +0.0004 | +0.07% | up (flat) |
| Redemptions per user | 1.7300 | 1.7302 | +0.0002 | +0.01% | up (flat) |
| 1:1 meetings engagement | 0.4469 | 0.4472 | +0.0003 | +0.06% | up (flat) |
| Pulse check engagement | 0.6006 | 0.5086 | -0.0920 | -15.31% | down |

Largest relative move: pulse check engagement, -15.31% overall.

Driver: the data supports size_band as the driver. Enterprise fell from 0.5500 to 0.2743 (-0.2757, -50.13%), e.g. C-0B2895EF 0.5619 -> 0.2864, C-0D0B047C 0.5398 -> 0.2619. By contrast, smb was 0.6588 -> 0.6573 (-0.22%) and mid_market 0.5930 -> 0.5942 (+0.21%), both flat. The overall drop is entirely the enterprise segment.

The data does not support plan_tier as a driver: every row is tier_three, so no cross-tier comparison is possible.

Caveat: these are company-month aggregates; with only 10 companies per size_band, no weighting or significance testing is possible from the data given.
rewards-ytd-sectionPASS 1.00
Plain-English question: Produce the redemption section of the weekly rewards report through August.
Prompt — what the model saw
Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.4307 · 78s · in 47,063 / out 17,755 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

Plain-English answer: 378 redemptions, $27,846 spend, 235 unique redeemers at 1.61 each, custom rewards at 39% of spend, US leading all countries.
The answer in one line: redemption_count = 378; unique_redeemers = 235; redemptions_per_redeemer = 1.61; total_spend_usd = 27846.00
redemption_count378
redemptions_per_redeemer1.61
unique_redeemers235
provider_mix_pct_of_spendTangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_countcountry=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
total_spend_usd27846.00
graded checkscount · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
REDEMPTIONS — WEEKLY REWARDS REPORT (through 2026-08)

Last completed month: 2026-08. The file contains redemptions from 2026-01 through 2026-08 (latest row 2026-08-31) and zero rows for 2026-09, so 2026-08 is the last completed month. All 378 rows fall within the reporting window; nothing excluded.

Headline metrics (2026-01-01 through 2026-08-31)
- Redemptions: 378
- Spend: $27,846.00
- Unique redeemers: 235 (distinct user_key)
- Redemptions per redeemer: 378 / 235 = 1.61

Provider mix (% of spend; total $27,846.00)
- custom: $10,873.00 / $27,846.00 = 39.05% (37 redemptions)
- Tremendous: $8,505.00 / $27,846.00 = 30.54% (192 redemptions)
- Snappy: $5,238.00 / $27,846.00 = 18.81% (59 redemptions)
- TangoCard: $3,230.00 / $27,846.00 = 11.60% (90 redemptions)
- Sum: 39.05 + 30.54 + 18.81 + 11.60 = 100.00%

Top 5 countries by redemptions
1. US: 244 ($18,547.00)
2. CA: 24 ($2,286.00)
3. AU: 21 ($1,606.00)
4. GB: 17 ($944.00)
5. NL: 17 ($1,122.00)

GB and NL tie at 17 redemptions; both shown. GB placed 4th on lower spend, NL 5th — order between tied ranks is arbitrary.
customer-success 5 tests
churn-save-eligibilityPASS 1.00
Plain-English question: Which at-risk accounts actually qualify for a churn-save offer, and which play fits each?
Prompt — what the model saw
Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.1185 · 46s · in 3,324 / out 6,632 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

Plain-English answer: Only accounts meeting all three rules qualify — e.g. C-0F6C0F34 at $49,707 (executive touch) and C-0B827671 at $25,365 (usage revival) — with amount and play per account.
The answer in one line: n_accounts = 30; snapshot_date = 2026-09-05
n_accounts30
eligibleaccount_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
non_eligible_at_riskaccount_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
ruleshealth_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
snapshot_date2026-09-05
total_amount_at_stake224601.00
graded checkseligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
ELIGIBILITY (rules applied from eligibility_rules.csv against snapshot 2026-09-05; 120-day window ends 2027-01-03)

QUALIFY (pass R1 health<60, R2 eligible amount>0, R3 renewal within 120 days) — 8 accounts:

1. C-0B0F1BAB — at stake: $5,494.00 (ARR $15,391.00) — renewal in 18 days (2026-09-23), health 38, usage flat, 238/363 seats (65.6%), champion inactive
   Play: commercial concession. Signal: renewal 2026-09-23 is 18 days out — no runway for usage or exec motions, so pricing/terms is the only lever.
2. C-0E9C27D1 — at stake: $41,235.00 (ARR $75,093.00) — renewal in 19 days (2026-09-24), health 39, usage flat, 134/157 seats (85.4%), champion active
   Play: commercial concession. Signal: renewal 2026-09-24 in 19 days; utilization is already high (85.4%) so usage isn't the lever — the gap is commercial.
3. C-0F6C0F34 — at stake: $49,707.00 (ARR $86,741.00) — renewal in 28 days (2026-10-03), health 51, usage growing, 308/395 seats (78.0%), champion inactive
   Play: executive touch. Signal: champion_active=false with usage growing and 78% seat utilization — the product is landing but there's no senior sponsor 28 days before renewal.
4. C-0B360C78 — at stake: $35,748.00 (ARR $60,427.00) — renewal in 53 days (2026-10-28), health 57, usage growing, 246/327 seats (75.2%), champion active
   Play: usage revival. Signal: growing trend but 81 paid seats unused (246/327 = 75.2%) — seat-expansion risk at renewal despite active champion.
5. C-0D3278C7 — at stake: $17,602.00 (ARR $33,815.00) — renewal in 68 days (2026-11-12), health 54, usage declining, 126/380 seats (33.2%), champion active
   Play: usage revival. Signal: declining usage trend plus only 33.2% seat utilization (126 of 380 seats used).
6. C-0B827671 — at stake: $25,365.00 (ARR $72,088.00) — renewal in 70 days (2026-11-14), health 56, usage declining, 113/202 seats (55.9%), champion active
   Play: usage revival. Signal: declining usage trend with 55.9% utilization (113 of 202 seats).
7. C-0CEF69FD — at stake: $32,621.00 (ARR $79,324.00) — renewal in 77 days (2026-11-21), health 53, usage growing, 97/136 seats (71.3%), champion inactive
   Play: executive touch. Signal: champion_active=false; usage is growing (71.3% utilization), so the exposure is relationship/sponsorship, not product.
8. C-0CA21961 — at stake: $16,829.00 (ARR $31,501.00) — renewal in 114 days (2026-12-28), health 58, usage flat, 84/325 seats (25.8%), champion active
   Play: usage revival. Signal: flat usage and the lowest utilization of the qualified set — 84 of 325 seats (25.8%) — with 114 days of runway to fix it.

TOTAL AT STAKE (sum of churn_save_eligible_amount, qualified accounts):
5,494.00 + 41,235.00 + 49,707.00 + 35,748.00 + 17,602.00 + 25,365.00 + 32,621.00 + 16,829.00 = $224,601.00
(Corresponding total ARR of qualified accounts: $454,380.00)

AT RISK (health<60) BUT NOT QUALIFIED — 7 accounts:

- C-0BC71BDD (health 55, renewal 2026-10-27, 52 days): fails R2 — churn_save_eligible_amount = 0.00 despite flat usage and 29.9% utilization (59/197).
- C-0BE96399 (health 54, renewal 2026-10-29, 54 days): fails R2 — eligible amount 0.00 despite declining usage and 27.9% utilization (43/154).
- C-10A56B0F (health 54, renewal 2026-12-12, 98 days): fails R2 — eligible amount 0.00 despite declining usage and 48.3% utilization (85/176).
- C-0BA71F12 (health 52): fails R3 — renewal 2027-04-11 is 218 days out, beyond the 120-day window (has $6,824.00 eligible).
- C-0F876796 (health 47): fails R3 — renewal 2027-02-06 is 154 days out, beyond the window (has $19,958.00 eligible).
- C-0F6694C3 (health 43): fails R2 and R3 — eligible amount 0.00 and renewal 2027-03-21 is 197 days out.
- C-0FCCD2DF (health 43): fails R2 and R3 — eligible amount 0.00 and renewal 2027-04-23 is 230 days out.

Caveat: the three play names (usage revival / executive touch / commercial concession) appear only in your question — no play-mapping rules exist in the provided files. Assignments above are inferred from the cited signals: champion_active=false drives executive touch, declining/flat usage with low seat utilization drives usage revival, and renewals inside ~30 days (where other motions lack runway) drive commercial concession.
expansion-kitPASS 1.00
Plain-English question: Build the expansion kit for the customer who just replied to our outreach.
Prompt — what the model saw
A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0790 · 28s · in 14,933 / out 2,281 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

Plain-English answer: 37.5% seat coverage, 250 headroom seats worth $15,000 ARR at $60 a seat, usage up 43.2% in six months — and the replier cannot buy; Dana R. (VP People) is the right buyer.
The answer in one line: headroom_seats = 250; coverage_pct = 37.5; company_alias = C-0DDFC9A7 (+3 more below)
headroom_seats250
coverage_pct37.5
company_aliasC-0DDFC9A7
headroom_arr15000.00
per_seat_annual_rate60.00
replierMaria S.
replier_can_buy0
right_buyername=Dana R.; title=VP People
usage_factsactive_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84
usage_health_linesMonthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checkscoverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
EXPANSION KIT — C-0DDFC9A7

SEAT COVERAGE
Licensed 150 / headcount 400 = 37.5% of company covered.

USAGE HEALTH (2 lines)
1. MAU grew 88 (2026-03) -> 126 (2026-08): +38 users, +43.2% over 5 months ((126-88)/88), averaging +7.6 net users/month.
2. Aug MAU 126 vs 150 licensed seats = 84.0% seat utilization (126/150); at +7.6/mo the account hits its seat cap in ~3 months (~Nov-Dec 2026).

HEADROOM
- Seats: 400 headcount - 150 licensed = 250 seats unlicensed.
- Per-seat rate: $9,000.00 / 150 = $60.00/seat/yr.
- ARR headroom: 250 x $60.00 = $15,000.00 (full coverage = 400 x $60.00 = $24,000.00 total ARR).

WHO REPLIED / CAN THEY BUY
Maria S., People Operations Coordinator, replied 2026-09-02. She explicitly says she is NOT the buyer: "Budget and seat expansion sit with Dana R., our VP People." She offered to make an introduction and noted Dana R. "has been asking about our usage numbers lately."

RIGHT BUYER
Dana R., VP People. Caveat: last engaged 2026-05-18 — ~3.5 months cold, so Maria S.'s warm intro is the right path in rather than a direct cold touch. Sam K. (Office Manager, last engaged 2025-11-03) is not relevant to this motion.

REPLY EMAIL (to Maria S.; ~100 words)

Subject: RE: Growing your team's recognition program

Hi Maria,

Thank you — and glad the team is getting so much out of Bonusly. The numbers reflect it: monthly active users grew from 88 in March to 126 in August.

An intro to Dana would be great, and I appreciate the offer. Since she's been asking about usage, I'll put together a one-page summary she can skim — adoption trends, recognition activity, and where you stand on seats. Happy to send it to you first so you can share it however works best.

Either way, thanks for the warm handoff — no rush on timing.

Best,
Cole

NOTES / GAPS
- No data on contract renewal date, plan tier, or discount structure — per-seat rate assumes flat $60.00/seat/yr from current ARR.
- No engagement history for Dana R. beyond 2026-05-18; why she went quiet is unknown.
onboarding-statusPASS 1.00
Plain-English question: What is done and what is missing in this customer's onboarding before the mid-onboarding call?
Prompt — what the model saw
Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0367 · 50s · in 403 / out 2,076 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

Plain-English answer: 25 days live: Slack connected, allowance set, admins added — but the HRIS integration is not connected, with early engagement signals and three call priorities.
The answer in one line: days_live = 25; company_alias = C-0D284E42
days_live25
call_itemsConnect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
checklist{"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
company_aliasC-0D284E42
engagement_signalsavg_active_givers_week1=4.3; avg_active_givers_last_week=13; latest_active_givers=15; days_with_activity=25
graded checkscomplete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
MID-ONBOARDING PREP — C-0D284E42
Signed up 2026-08-11; 24 days elapsed as of the last usage date (2026-09-04).

COMPLETE (with the data field that proves it)
- Slack integration — integration_slack = 2026-08-12 (1 day after signup)
- Allowance set — allowance_set = 2026-08-13 (2 days after signup)
- Admins added — admins_added = 2 (no target count was provided, so "2" is all we can state; sufficiency unknown)
- First recognition given — first_recognition_at = 2026-08-15 14:22 (4 days after signup)

NOT COMPLETE (empty fields)
- HRIS integration — integration_hris is blank; no date, no evidence of connection
- First redemption — first_redemption_at is blank; 20 days have passed since first recognition (8/15 → 9/4) with zero redemptions recorded

EARLY ENGAGEMENT SIGNALS (onboarding_usage.csv, active_givers)
- Growth: 3 givers on 8/11 → 15 on 9/4. Arithmetic: 15 − 3 = +12 givers; 15 ÷ 3 = 5.0x
- First 7 days (8/11–8/17): 3+3+4+4+5+4+7 = 30; 30 ÷ 7 ≈ 4.3 avg
- Last 7 days (8/29–9/4): 11+13+11+13+13+15+15 = 91; 91 ÷ 7 = 13.0 avg → 13.0 ÷ 4.3 ≈ 3.0x first-week average
- Peak: 15 active givers on both 9/3 and 9/4 (most recent two days)
- Dips at 8/16 (4), 8/18 (5), 8/22 (8), 8/27 (10), 8/31 (11) — each recovered to a new high within 1–3 days; no downward trend anywhere in the series
- Data missing: no total employee count for C-0D284E42 was provided, so giver penetration (%) cannot be computed — do not quote one

THREE THINGS TO COVER ON THE CALL
1. HRIS integration — the one remaining unconnected integration (field blank). Find the owner and blocker, and set a connection date.
2. First redemption — recognition is flowing and givers are up 5x, but first_redemption_at is still empty 20 days after first recognition. Agree on a concrete action to get receivers redeeming.
3. Participation ceiling and admin coverage — givers grew from 3 to 15, but without a headcount number we can't tell how much room is left; confirm company size and whether 2 admins (admins_added = 2) is enough coverage to push the next cohort of givers.
renewal-risk-conflicting-datesPASS 1.00
Plain-English question: Two systems disagree on renewal dates — which do you trust per account, and who is actually renewing inside 90 days?
Prompt — what the model saw
Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.2328 · 100s · in 8,898 / out 13,066 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

Plain-English answer: Chargebee wins per account (multi-year contracts are known wrong in ChurnZero), with each trusted date, the reason, and a 90-day-window flag.
The answer in one line: n_accounts = 20; n_disagreements = 5; snapshot_date = 2026-09-05 (+2 more below)
n_accounts20
n_disagreements5
accounts20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
arr_at_risk_90d359409.00
disagreementsaccount_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
snapshot_date2026-09-05
total_arr_renewing_90d1048715.00
trust_rulemulti-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
window2026-09-05 to 2026-12-04
graded checkstotal_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-DAY RENEWAL RISK BRIEF
As-of date: 2026-09-05 (session date) | Window: 2026-09-05 to 2026-12-04 | Usage trend = active users 2026-06 -> 2026-08, last 3 months of data. All figures verified by computation.

═══════════════════════════════════════════════════════════════════════
1. WHICH SYSTEM TO TRUST (per account)
═══════════════════════════════════════════════════════════════════════

Rule: Chargebee dates come from actual billing contract terms (term_months, is_multi_year); ChurnZero multi-year dates are known to be wrong. So:
- Multi-year contracts (is_multi_year=true, 5 accounts): TRUST CHARGEBEE.
- 12-month contracts (15 accounts): ChurnZero and Chargebee agree exactly, so the date is confirmed by both systems — no conflict.

Note on the "wrong in ChurnZero" pattern: only 2 of the 5 multi-year accounts show ChurnZero pushing the date to 2027. The other 3 (C-0B7D2C30, C-0D2AB865, C-0F5D2323) have ChurnZero dates EARLIER than Chargebee — still wrong per the known defect, just wrong in the other direction. Every multi-year account disagrees; none agree.

Disagreements (5 of 20 accounts — all multi-year):
- C-0B7D2C30  CZ 2026-09-10 vs CB 2026-09-15  -> use CB 2026-09-15 (5-day gap)
- C-0BCDB8C2  CZ 2027-09-18 vs CB 2026-09-18  -> use CB 2026-09-18 (CZ 365 days late; the 2027 error)
- C-0D2AB865  CZ 2026-09-10 vs CB 2026-09-22  -> use CB 2026-09-22 (12-day gap)
- C-0BBE3E60  CZ 2027-09-26 vs CB 2026-09-26  -> use CB 2026-09-26 (CZ 365 days late; the 2027 error)
- C-0F5D2323  CZ 2026-09-10 vs CB 2026-09-29  -> use CB 2026-09-29 (19-day gap)

Operational impact of the CZ 2027 errors: had ChurnZero been trusted, C-0BCDB8C2 ($54,427) and C-0BBE3E60 ($30,993) would have been excluded from this window entirely — $85,420 of ARR renewing within ~3 weeks of today, invisible to the team.

═══════════════════════════════════════════════════════════════════════
2. RENEWAL REGISTER (sorted by trusted date)
═══════════════════════════════════════════════════════════════════════

SEPTEMBER — 5 renewals, $279,090 ARR. All five are HIGH risk. This month is the problem.

1) C-0B7D2C30 | Dana Mercer | $65,901 | 2026-09-15 (CB; disagrees w/ CZ)
   Utilization: 274/476 = 57.6% | 3-mo trend: 97 -> 84 = -13 (-13.4%)
   HIGH — usage down 13.4% in 3 months and down 45.8% over 12 (155 -> 84), a sustained decline with 10 days to renewal.

2) C-0BCDB8C2 | Cole Ingram | $54,427 | 2026-09-18 (CB; disagrees w/ CZ)
   Utilization: 232/424 = 54.7% | 3-mo trend: 127 -> 110 = -17 (-13.4%)
   HIGH — steady bleed in every month of the dataset (200 -> 110 over 12 months, -45.0%) and renewal is in 13 days.

3) C-0D2AB865 | Elena Sinclair | $38,022 | 2026-09-22 (CB; disagrees w/ CZ)
   Utilization: 250/407 = 61.4% | 3-mo trend: 125 -> 109 = -16 (-12.8%)
   HIGH — uninterrupted 12-month decline (199 -> 109, -45.2%) with no sign of leveling off.

4) C-0BBE3E60 | Dana Mercer | $30,993 | 2026-09-26 (CB; disagrees w/ CZ)
   Utilization: 74/114 = 64.9% | 3-mo trend: 39 -> 33 = -6 (-15.4%)
   HIGH — steepest recent decline in the book (-15.4% in 3 months) on top of a 47.6% 12-month drop (63 -> 33).

5) C-0F5D2323 | Cole Ingram | $90,647 | 2026-09-29 (CB; disagrees w/ CZ)
   Utilization: 111/390 = 28.5% | 3-mo trend: 20 -> 18 = -2 (-10.0%)
   HIGH — largest ARR in the September cohort with only 28.5% of seats in use and engagement stuck near zero (18-21 active users all year on a 390-seat contract).

OCTOBER — 5 renewals, $277,404 ARR

6) C-0EC6999D | Elena Sinclair | $79,419 | 2026-10-03 (systems agree)
   Utilization: 31/112 = 27.7% | 3-mo trend: 17 -> 15 = -2 (-11.8%)
   HIGH — worst seat utilization in the book (27.7%) and only 15 active users against a $79,419 contract.

7) C-0B20DB64 | Dana Mercer | $21,770 | 2026-10-07 (systems agree)
   Utilization: 214/378 = 56.6% | 3-mo trend: 294 -> 294 = 0 (0.0%)
   MEDIUM — engagement is rock-solid and flat for 12 months (293-298 band), but only 56.6% of seats are used, leaving a down-sell argument at renewal.

8) C-0BBC4E7A | Cole Ingram | $56,374 | 2026-10-10 (systems agree)
   Utilization: 228/337 = 67.7% | 3-mo trend: 142 -> 139 = -3 (-2.1%)
   LOW — usage essentially flat all year (139-142 band) with acceptable utilization; minor drift only.

9) C-0FD551AB | Elena Sinclair | $48,815 | 2026-10-14 (systems agree)
   Utilization: 210/376 = 55.9% | 3-mo trend: 123 -> 126 = +3 (+2.4%)
   MEDIUM — usage stable-to-slightly-up, but 44.1% of seats sit idle (376 seats vs ~126 active users), a classic right-size-at-renewal target.

10) C-0F9F8F13 | Dana Mercer | $46,230 | 2026-10-18 (systems agree)
    Utilization: 199/352 = 56.5% | 3-mo trend: 185 -> 182 = -3 (-1.6%)
    MEDIUM — engagement flat for 12 months (181-185 band) but utilization at 56.5% means roughly 153 unused seats are exposed to down-sell.

11) C-0BC34584 | Cole Ingram | $16,740 | 2026-10-22 (systems agree)
    Utilization: 327/494 = 66.2% | 3-mo trend: 104 -> 106 = +2 (+1.9%)
    LOW — smallest account in the book, usage stable and slightly rising.

12) C-0B7A7546 | Elena Sinclair | $35,062 | 2026-10-25 (systems agree)
    Utilization: 182/205 = 88.8% | 3-mo trend: 64 -> 63 = -1 (-1.6%)
    LOW — highest utilization in the book (88.8%) with usage up 8.6% over 12 months (58 -> 63).

13) C-0B369871 | Dana Mercer | $85,128 | 2026-10-29 (systems agree)
    Utilization: 317/422 = 75.1% | 3-mo trend: 326 -> 333 = +7 (+2.1%)
    LOW — usage grew every month of the dataset (289 -> 333, +15.2% over 12); a strong renewal and an expansion candidate (333 active users vs 317 seats_used suggests seat count may already be tightening).

NOVEMBER — 5 renewals, $492,221 ARR. All LOW risk.

14) C-0B144C78 | Cole Ingram | $30,899 | 2026-11-02 (systems agree)
    Utilization: 169/224 = 75.4% | 3-mo trend: 101 -> 106 = +5 (+5.0%)
    LOW — best recent growth rate in the book (+5.0% in 3 months), continuing a 12-month climb (90 -> 106).

15) C-0FC4DBB8 | Elena Sinclair | $94,732 | 2026-11-05 (systems agree)
    Utilization: 356/464 = 76.7% | 3-mo trend: 189 -> 193 = +4 (+2.1%)
    LOW — largest ARR in the entire window and trending up steadily all year (168 -> 193, +14.9%).

16) C-0D5BBE3A | Dana Mercer | $39,740 | 2026-11-09 (systems agree)
    Utilization: 85/102 = 83.3% | 3-mo trend: 88 -> 91 = +3 (+3.4%)
    LOW — high utilization plus uninterrupted 12-month growth (76 -> 91, +19.7%); expansion candidate.

17) C-0FB9D5AF | Cole Ingram | $63,158 | 2026-11-13 (systems agree)
    Utilization: 144/199 = 72.4% | 3-mo trend: 173 -> 176 = +3 (+1.7%)
    LOW — consistent 12-month growth (154 -> 176, +14.3%) with solid utilization.

18) C-0B344485 | Elena Sinclair | $64,384 | 2026-11-16 (systems agree)
    Utilization: 224/287 = 78.0% | 3-mo trend: 238 -> 244 = +6 (+2.5%)
    LOW — grew every month of the year (211 -> 244, +15.6%).

19) C-0CB2C1B4 | Dana Mercer | $40,628 | 2026-11-20 (systems agree)
    Utilization: 386/473 = 81.6% | 3-mo trend: 47 -> 49 = +2 (+4.3%)
    LOW — utilization above 80% and usage trending up (43 -> 49 over 12 months).

20) C-22170CA1 | Cole Ingram | $45,646 | 2026-11-24 (systems agree)
    Utilization: 251/294 = 85.4% | 3-mo trend: 143 -> 146 = +3 (+2.1%)
    LOW — steady growth all year (130 -> 146, +12.3%) with high utilization.

═══════════════════════════════════════════════════════════════════════
3. TOTALS (arithmetic shown)
═══════════════════════════════════════════════════════════════════════

Total ARR renewing in the 90-day window (all 20 accounts fall inside it):
65,901 + 54,427 + 38,022 + 30,993 + 90,647 + 79,419 + 21,770 + 56,374 +
48,815 + 46,230 + 16,740 + 35,062 + 85,128 + 30,899 + 94,732 + 39,740 +
63,158 + 64,384 + 40,628 + 45,646 = $1,048,715

ARR at risk (HIGH-rated accounts):
65,901 + 54,427 + 38,022 + 30,993 + 90,647 + 79,419 = $359,409  (34.3% of renewing ARR)

Medium-rated ARR (down-sell exposure, not modeled as at-risk):
21,770 + 48,815 + 46,230 = $116,815  (11.1%)
If mediums are included, total elevated-exposure ARR = 359,409 + 116,815 = $476,224 (45.4%).

LOW-rated ARR: $572,491 (54.6%).

Key takeaways:
- Risk is front-loaded: all $359,409 of HIGH-risk ARR renews between 2026-09-15 and 2026-10-03. September alone holds $279,090 across five accounts, every one HIGH risk, and two of them (C-0BCDB8C2, C-0BBE3E60) would have been missed entirely on ChurnZero dates.
- Every HIGH account shows either utilization under 30% (C-0F5D2323, C-0EC6999D) or a double-digit 3-month usage decline (the other four, all -10.0% or worse); the four September decliners have each lost 45%+ of active users over 12 months.
- The November cohort ($492,221, 47% of the book) is uniformly healthy — high utilization and growth — so save-team capacity should be concentrated on the September six (incl. C-0EC6999D on 10/03).

Data caveats: only the fields provided were used — no health scores, NPS, support tickets, or billing/collections status were available, so risk ratings rest solely on seat utilization and usage trend. The usage file ends at 2026-08; no September usage data exists yet. "ARR at risk" is the full contract ARR of HIGH-rated accounts, not a probability-weighted figure.
ticket-theme-synthesis0.80
Plain-English question: What are the real themes in this quarter's support tickets, ignoring the existing tags?
Prompt — what the model saw
Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.2011 · 177s · in 18,284 / out 9,749 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

Plain-English answer: 80 tickets in 4-7 themes — points-not-posting leads (20 tickets, 25%, $31,100 ARR affected) — each with counts, distinct accounts, ARR, and a recommendation.
The answer in one line: total_tickets = 80; quarter_window = 2026-06-01 through 2026-08-31
total_tickets80
noteexisting tags are noisy/unreliable; classify by text
quarter_window2026-06-01 through 2026-08-31
ranking_by_arr_exposurehris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
sample_ticket_ids80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
single_account_noise_themebilling_dispute
themes{"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
graded checkstheme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
SUPPORT TICKET SYNTHESIS — 80 tickets (IC-460001–IC-460080), 2026-06-01 to 2026-08-29
Existing tags ignored as instructed (they are noisy: identical billing texts are tagged how-to/question/bug/billing; HRIS texts are tagged billing/feedback/how-to). Themes below are built from body text. 24 distinct accounts total; no account appears in more than one theme. Ranked by ARR exposure.

────────────────────────────────────────
1. HRIS PROVISIONING FAILURES (new hires not created) — broad pattern
   Tickets: 12 → 12/80 = 15.0%
   Accounts: 3 — C-0B2213A9 (7 tickets), C-0DDFC9A7 (3), C-0F6C0F34 (2)
   ARR: $36,000 + $48,000 + $30,000 = $114,000 (40% of total ARR exposure)
   Example IDs: IC-460062, IC-460059
   Note: worst variant is silent failure — "HRIS sync skipped 12 new hires; provisioning log shows no errors" (IC-460062, IC-460060, IC-460064) means no alerting fired.
   Recommendation: P1 engineering escalation — audit the HRIS sync pipeline, add failure alerting, and backfill missed hires at all 3 accounts.

2. REDEMPTION / CHECKOUT FAILURES — broad pattern
   Tickets: 18 → 18/80 = 22.5%
   Accounts: 7 — C-0CEF69FD, C-0B827671, C-0F876796, C-14264ABD, C-0FCCD2DF, C-0D9CA315, C-0B0F1BAB
   ARR: $8,900 + $10,700 + $8,700 + $11,000 + $9,600 + $9,600 + $10,300 = $68,800
   Example IDs: IC-460025, IC-460024
   Sub-variants: checkout spins then fails (4), redemption failed / gift card email never arrived (6), order errored but points still deducted (5), gift card code never arrived (3).
   Recommendation: make redemption transactional — roll back points when fulfillment fails — and fix the checkout timeout.

3. INVOICE / BILLING ERRORS — SINGLE-ACCOUNT ISSUE (C-0E9C27D1 only)
   Tickets: 16 → 16/80 = 20.0%
   Accounts: 1 — C-0E9C27D1 (all 16 tickets)
   ARR: $52,000
   Example IDs: IC-460069, IC-460078
   Single-account by breadth, but not dismissible noise: repeat seat-count errors ("charged for 200 seats but we license 150," "third invoice in a row"), wrong renewal tier pricing, unapproved seat counts — 16 tickets in ~10 weeks on one $52,000 account is a churn-risk signal.
   Recommendation: full billing audit and corrected invoices for C-0E9C27D1, assign an exec sponsor, and fix seat-count/tier logic before renewal.

4. POINTS NOT POSTING / BALANCE NOT UPDATING — broad pattern
   Tickets: 20 → 20/80 = 25.0% (highest volume, only 4th by ARR)
   Accounts: 9 — C-0D3278C7, C-0BE96399, C-0DD0626C, C-0B2895EF, C-0D284E42, C-0BF20542, C-0D0B047C, C-0D6CC8E3, C-21FEBCBB
   ARR: $3,500 + $2,700 + $2,500 + $2,900 + $3,400 + $4,500 + $4,500 + $4,200 + $2,900 = $31,100
   Example IDs: IC-460001, IC-460004
   Concentrated in small accounts ($2,500–$4,500); includes team-wide outages ("points not posting for our whole team").
   Recommendation: fix the recognition→points posting pipeline and add a reconciliation job for delivered-but-unposted recognitions.

5. SLACK INTEGRATION FAILURES — broad pattern
   Tickets: 14 → 14/80 = 17.5%
   Accounts: 4 — C-0B843542, C-0BA71F12, C-10A56B0F, C-8C2E8F00
   ARR: $4,400 + $3,900 + $5,400 + $5,200 = $18,900
   Example IDs: IC-460047, IC-460046
   Sub-variants: slash command errors (5), sync toggle resets itself (4), sync stopped (3), re-auth does not stick (2) — points to OAuth token persistence.
   Recommendation: fix OAuth token persistence / re-auth flow and slash-command error handling.

────────────────────────────────────────
Validation: 12 + 18 + 16 + 20 + 14 = 80 tickets ✓. Total ARR exposure: $114,000 + $68,800 + $52,000 + $31,100 + $18,900 = $284,800.
Volume vs. ARR inversion to flag: the points theme is #1 by ticket count (25%) but #4 by ARR; HRIS is last by count (15%) but #1 by ARR — prioritize HRIS and the C-0E9C27D1 billing fix over the high-volume points queue.
marketing 5 tests
case-study-matchPASS 1.00
Plain-English question: Which three existing customers are the best case-study matches for this prospect?
Prompt — what the model saw
Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0380 · 25s · in 1,035 / out 2,039 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

Plain-English answer: Three ranked matches on industry, size, use case, and region (e.g. C-11C31562), with the higher-scoring decoy explicitly excluded for a mismatched field.
The answer in one line: prospect_alias = C-82AF3719
constraintsno_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
decoys_excludedcompany_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
prospect_aliasC-82AF3719
prospect_fields_usedindustry, size_band, use_case, region
top3company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
graded checkstop3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
TOP 3 LOOKALIKE CUSTOMERS (case-study-eligible only)

Prospect C-82AF3719: Technology / Mid-Market / employee_recognition / NA-West

Scoring: 1 point per matching field (industry, size_band, use_case, region) across the 8 customers where has_case_study=true. Tie-break rule: use_case match > industry match > region match (size_band equal across all finalists).

1. C-64171065 — 3/4
   Matched: industry=Technology, size_band=Mid-Market, use_case=employee_recognition
   Missed: region (NA-East vs NA-West)
   Wins the 3/4 tie on use_case + industry matches.

2. C-11C31562 — 3/4
   Matched: size_band=Mid-Market, use_case=employee_recognition, region=NA-West
   Missed: industry (Manufacturing vs Technology)
   Wins use_case tie-break; loses to #1 on industry.

3. C-A13C193D — 3/4
   Matched: industry=Technology, size_band=Mid-Market, region=NA-West
   Missed: use_case (retention vs employee_recognition)
   Loses tie-break: no use_case match.

Notes:
- No 4/4 match exists among case-study customers. Two customers match all four fields exactly, but both have has_case_study=false and are excluded (not named, per rules).
- Next-best eligible backups at 2/4, if needed: C-CD4829A7 (Technology, Mid-Market) and C-9D69D147 (Mid-Market, NA-West).
channel-attribution-mathPASS 1.00
Plain-English question: Compute cost per SQM, cost per SQO, and pipeline per dollar for every channel over six months.
Prompt — what the model saw
Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.1551 · 134s · in 17,438 / out 6,850 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

Plain-English answer: LinkedIn Ads at $960/SQM and $3,000/SQO, organic search at $90,000 pipeline on zero spend, paid social flagged at $18,000 with zero SQMs, and a channel winner named.
The answer in one line: best_pipeline_per_dollar_channel = paid_search
best_pipeline_per_dollar_channelpaid_search
confidence_notewebinars and referral have small SQM samples (<20); treat rates as directional.
flagged_rows_sqo_before_sqmCT-000041, CT-000044
months2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel{"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
reallocation_recommendationfrom=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
undefined_channelspaid_social
graded checksundefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
CHANNEL PERFORMANCE — TRAILING 6 MONTHS (2026-03 through 2026-08)

PAID CHANNELS

Spend per channel = monthly spend x 6 months.

channel       spend      SQMs  SQOs  $/SQM    $/SQO    SQM->SQO   pipeline    pipeline/$
paid_search   $36,000    40    18    $900     $2,000   45.0%      $720,000    20.00
linkedin_ads  $24,000    25    8     $960     $3,000   32.0%      $96,000     4.00
webinars      $9,000     12    5     $750     $1,800   41.7%      $60,000     6.67
paid_social   $18,000    0     0     undefined  undefined  undefined  $0      0.00
TOTAL         $87,000    77    31    $1,130   $2,806   40.3%      $876,000    10.07

Arithmetic:
- paid_search: 6,000x6 = 36,000; 36,000/40 = 900; 36,000/18 = 2,000; 18/40 = 45.0%; 18 x 40,000 = 720,000; 720,000/36,000 = 20.00
- linkedin_ads: 4,000x6 = 24,000; 24,000/25 = 960; 24,000/8 = 3,000; 8/25 = 32.0%; 8 x 12,000 = 96,000; 96,000/24,000 = 4.00
- webinars: 1,500x6 = 9,000; 9,000/12 = 750; 9,000/5 = 1,800; 5/12 = 41.7%; 5 x 12,000 = 60,000; 60,000/9,000 = 6.67
- paid_social: 3,000x6 = 18,000 spend; zero contacts in contacts_first_touch.csv, so cost-per-SQM, cost-per-SQO, and SQM->SQO are undefined (division by zero), reported as undefined per instruction. Pipeline = 0 (no records); pipeline/$ = 0/18,000 = 0.00 arithmetically, but treat as no-signal rather than a true zero-efficiency channel.

ORGANIC / NO-SPEND CHANNELS
(No spend rows exist for these in channel_spend.csv, so cost metrics cannot be computed — data missing, not zero.)

channel        volume (SQMs)  SQOs  SQO rate   pipeline
organic_search 30             10    33.3%      $90,000
referral       15             6     40.0%      $48,000

Arithmetic: organic_search 10/30 = 33.3%, 10 x 9,000 = 90,000; referral 6/15 = 40.0%, 6 x 8,000 = 48,000.

DATA QUALITY FLAGS — SQO date precedes SQM date (2 rows, both linkedin_ads):
- CT-000041: sqm_date 2026-06-14, sqo_date 2026-06-09
- CT-000044: sqm_date 2026-07-23, sqo_date 2026-07-18
Both are counted in the linkedin_ads totals above as given. If excluded: linkedin_ads SQOs = 6, rate = 24.0%, $/SQO = 4,000, pipeline = 72,000, pipeline/$ = 3.00 — it looks even worse.
(Note: CT-000007 has SQO on the same day as SQM, 2026-03-28 to 2026-03-28 — same-day, not preceding, so not flagged.)

REALLOCATION RECOMMENDATION

1. Cut paid_social entirely. $18,000 over 6 months produced zero SQMs — that is a complete no-signal channel, not a small-sample problem. This is the highest-confidence move in the dataset.
2. Shift that $3,000/month to paid_search. It leads on every efficiency metric: $2,000/SQO, 45.0% SQM->SQO, and 20.00 pipeline/$ (5x the linkedin_ads return). It already carries 72% of paid pipeline.
3. Put linkedin_ads on a one-quarter watch. Worst producer among channels with volume: $3,000/SQO, 32.0% rate, 4.00 pipeline/$, plus both data-quality flags land here. If it doesn't converge toward webinars-level economics (~$1,800/SQO), cut it next and split between paid_search and webinars.
4. Fund webinars modestly. Best $/SQM ($750) and second-best rate (41.7%), but only 12 SQMs — scale carefully until the sample grows.
5. organic_search (30 SQMs, 33.3%, $90k) and referral (15 SQMs, 40.0%, $48k) generated $138,000 pipeline with no tracked spend. No cost data exists, so ROI can't be computed, but referral's 40.0% rate argues for a formal referral program; treat investment sizing as a judgment call, not a data-driven one.

CONFIDENCE (given sample sizes)
- High: paid_social cut (6 months of zero output) and paid_search as the top performer (n=40 SQMs / 18 SQOs, consistent $40k deal sizes).
- Moderate: linkedin_ads underperformance (n=25/8; the two flagged rows only strengthen the case).
- Low-to-moderate: webinars ranking (n=12/5 — a swing of 2 SQOs moves its rate by ~17 points) and referral (n=15/6). Don't over-index on either until next quarter's data lands.
competitor-battlecard-updatePASS 1.00
Plain-English question: Update the Rivally battlecard with sourced pricing and where each side wins.
Prompt — what the model saw
Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.1269 · 95s · in 15,844 / out 5,288 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

Plain-English answer: Rivally at $7 per user per month (the August source wins over the stale $5, conflict noted), a 13-7 win/loss record, and two unverified old claims flagged.
The answer in one line: competitor = Rivally
citation_ids_available25 items — e.g. S01, S02, S03, S04
competitorRivally
pricingvalue_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
rep_opinion_snippet_ids_not_factsS09, S21
unverified_old_claimsRivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
win_loss_12mowins=13; losses=7
graded checkscompetitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Battlecard: Rivally — updated 2026-09-05

Data basis: competitor_snippets.csv (S01–S25), deals_with_competitor.csv (20 deals), existing_battlecard.md. Rep opinions (S09, S21) excluded as competitor facts.

## 1. One-line positioning
Points-based employee recognition vendor with an engaging recognition feed and fast mid-market setup, now expanding upmarket into EU enterprise and adjacent engagement surveys. (S02, S04, S12, S15, S06/S23)

## 2. Pricing
- Current list: Recognition Starter at $7 per user/month, annual billing required, per pricing page dated 2026-08-12 (S17).
- Conflict (newer source wins): pricing pages dated 2026-01-20 (S03) and 2026-04-01 (S08) showed $5 per user/month, annual billing. The 2026-08-12 page supersedes both; treat $5 as the old price, retired sometime between 2026-04-01 and 2026-08-12.
- Field-reported quotes (call notes, not list price): $6.50/user/mo to a 500-seat prospect on annual term, 2026-06-02 (S13); $7/user/mo list with a 15% discount for a 3-year term, 2026-08-14 (S18).
- Rivally Pulse (survey add-on) is priced as an add-on, not bundled (S23). Price point for Pulse is not in the data.

## 3. Where they win
- Recognition feed is engaging; points-based model praised by reviewers (S02, S16).
- Fast deployment: setup under a week, Slack integration works out of the box (S04).
- Distributed EU teams: multi-language support praised by an EU enterprise reviewer (S12); EU data residency now generally available (S15).
- Support response time under 4 hours praised (S22).
- Balance sheet: $40M Series C led by Northgate Ventures, 2025-11-04 (S01); EMEA leadership hire (ex-Workday VP) (S11).

## 4. Where we win
- Analytics depth: limited analytics (S02), basic reporting dashboards vs. enterprise tools (S07), analytics exports are CSV-only (S20). Deal evidence: an 800-seat prospect picked Bonusly over Rivally citing analytics depth, 2026-09-03 (S25).
- Enterprise readiness: no SCIM provisioning; manual user management called painful (S10).
- Admin tooling: lags peers (S16); admin console still lacks bulk recognition editing (S24).
- EMEA rewards catalog thinner than US catalog (S14) — undercuts their EU pitch on the rewards side.
(Note: S05 and S25 both show prospects evaluating or choosing Bonusly against Rivally, which implies we are Bonusly in this dataset. Flagged as inference, not used as a standalone fact.)

## 5. Objections and responses
- "Rivally is cheaper — $5/user."
  That price is retired. Current list is $7/user/mo, annual only (S17; old $5 pages S03/S08 superseded). Field reports show discounting off the new list (S13, S18) — anchor on value, not their old price.
- "Rivally has EU data residency."
  True — GA since 2026-07-01 (S15). Do not dispute. Redirect to what EU customers still give up: thinner EMEA rewards catalog (S14), no SCIM (S10), CSV-only analytics exports (S20).
- "Their recognition feed is more engaging."
  Concede the feed (S02, S16); counter with the admin and reporting burden behind it: basic dashboards (S07), admin tooling lags peers (S16), no bulk recognition editing (S24).
- "They're well funded and expanding."
  Acknowledge the $40M Series C (S01) and Dublin opening (S15). Point to what the expansion hasn't fixed per reviewers: analytics (S02, S07), provisioning (S10), admin tooling (S24).
- "Rivally is discounting aggressively / their UI is clunky."
  Both are rep opinions only (S21, S09) — do not present to prospects as fact.

## 6. Recent changes (newest first)
- 2026-09-01: Rivally Pulse exits beta; sold as a paid add-on, not bundled (S23).
- 2026-08-20: Microsoft Teams app v2 in public preview (S19).
- 2026-08-12: Starter list price raised $5 → $7/user/mo (S17, vs. S03/S08).
- 2026-07-01: Dublin office opened; EU data residency GA (S15).
- 2026-05-09: Hired ex-Workday VP EMEA to lead European expansion (S11).
- 2026-03-05: Launched Rivally Pulse survey add-on (S06).
- 2025-11-04: $40M Series C led by Northgate Ventures (S01).

## 7. Our 12-month win/loss vs. Rivally (2025-09 through 2026-08)
All 20 deals in the file fall inside this window.
- Record: 13 wins, 7 losses.
- Arithmetic: 13 + 7 = 20 deals; win rate = 13/20 = 65%.
- Wins: Deal-A9FD43, Deal-7AA785, Deal-44C524, Deal-0D0CD6, Deal-D5B790, Deal-5C636E, Deal-67BE14, Deal-1B6969, Deal-F03E7B, Deal-072E31, Deal-F65C8F, Deal-E46EAB, Deal-1D2392.
- Losses: Deal-7767F5, Deal-D263E0, Deal-935746, Deal-5C636E is a win — losses are Deal-7767F5, Deal-D263E0, Deal-935746, Deal-5645A5, Deal-C6FFAA, Deal-9066A6, Deal-72A02F.
- Pattern: losses cluster 2026-03 to 2026-05 (4 of 7 losses: Deal-9066A6, Deal-5645A5, Deal-72A02F, Deal-C6FFAA); then three straight wins 2026-06 to 2026-08 (Deal-67BE14, Deal-1B6969, Deal-F03E7B). No deal data past 2026-08.

## 8. Old-card carryover status
- "Points-based recognition for mid-market" — partially re-sourced: points-based feed (S02); mid-market evidence limited to a mid-market reviewer (S04).
- "Starts at $5/user/month, annual (2026-01)" — outdated: superseded by $7 on 2026-08-12 (S17).
- "Rivally lacks a Slack integration" — contradicted by S04 (Slack integration works out of the box). Removed.
- "Acquired by WorkHuman in 2025" — UNVERIFIED: no source in the provided data. Note tension with S01 (independent $40M Series C raise in 2025-11), but S01 does not disprove it.
- "Strong in EU enterprise with multi-language support" — re-sourced (S12).

No other data gaps beyond those flagged above.
nurture-sequence-diagnosisPASS 1.00
Plain-English question: What is broken in each email sequence — rates, overlaps, tracking errors?
Prompt — what the model saw
Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.1285 · 38s · in 10,794 / out 6,120 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

Plain-English answer: Per-sequence open, reply, and meeting rates with the weakest step (all three weakest at step 3), an opened-above-sent tracking error, and audience overlap flagged.
The answer in one line: fix_priority_first = Cold Outbound - HR Leaders
all_pairwise_overlaps['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
audience_overlap_pairCold Outbound - HR Leaders, Cold Outbound - People Ops
failure_modeCold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_firstCold Outbound - HR Leaders
per_sequence{"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_errorsequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
under_2pct_replyCold Outbound - HR Leaders
graded checkstracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
SEQUENCE PERFORMANCE

New Logo Nurture — sent 1,386 (500+458+428); opens 490/1,386 = 35.4%; replies 90/1,386 = 6.5%; meetings 27/1,386 = 1.9%. Weakest step: 3 (reply 18/428 = 4.2% vs 8.4% step 1, 6.6% step 2).

Expansion Nurture — sent 875; opens 565/875 = 64.6% (inflated, see tracking error); replies 59/875 = 6.7%; meetings 12/875 = 1.4%. Weakest step: 3 (reply 12/275 = 4.4% vs 7.3% and 8.3%).

Cold Outbound - HR Leaders — sent 1,785; opens 545/1,785 = 30.5%; replies 8/1,785 = 0.45%; meetings 0/1,785 = 0.0%. Weakest step: all fail; step 3 lowest (1/590 = 0.17%).

Cold Outbound - People Ops — sent 1,163; opens 340/1,163 = 29.2%; replies 29/1,163 = 2.5%; meetings 6/1,163 = 0.5%. Weakest step: 3 (reply 6/377 = 1.6%, decay 3.5% → 2.3% → 1.6%).

TRACKING ERRORS
Expansion Nurture step 2: opened 340 > sent 300 — impossible; duplicate open events or unlogged sends. Only opened-above-sent instance in the file.

AUDIENCE OVERLAP
21 contacts sit in two sequences.
New Logo Nurture ∩ Expansion Nurture (2): CT-000301, CT-000624.
Cold Outbound - HR Leaders ∩ Cold Outbound - People Ops (19): CT-000849, CT-000884, CT-000890, CT-000908, CT-001033, CT-001097, CT-001101, CT-001103, CT-001105, CT-001153, CT-001159, CT-001217, CT-001227, CT-001236, CT-001255, CT-001258, CT-001277, CT-001285, CT-001311.
No nurture ∩ cold overlap.

FAILURE MODE (<2% REPLY)
Cold Outbound - HR Leaders: message/offer failure, not deliverability. Step-1 open 240/600 = 40% proves emails land and subject lines work; reply 5/600 = 0.83% and 0 meetings means the body/value prop/CTA isn't resonating with HR Leaders. Cold Outbound - People Ops step 3 (1.6%) is classic sequence fatigue on top of a marginal 2.5% baseline.

ONE CHANGE PER WEAK SEQUENCE
- Cold Outbound - HR Leaders: rewrite step-1 value prop and CTA (single, specific ask); keep list and subject lines constant so the test isolates messaging.
- Cold Outbound - People Ops: replace step 3 with a new-angle/breakup email to stop the 3.5→2.3→1.6% decay.
- Expansion Nurture: fix open-event deduping on step 2, then re-baseline before judging step 3.

FIX FIRST
Cold Outbound - HR Leaders: largest volume (1,785 sends) producing 0 meetings — the biggest source of wasted sends and burned domain reputation.
weekly-marketing-goals-rollupPASS 1.00
Plain-English question: Are we on pace on the quarter's marketing goals?
Prompt — what the model saw
Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0378 · 25s · in 638 / out 2,109 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

Plain-English answer: SQMs ahead of pace, SQOs on pace, DS2s behind (35 short of expected at day 66 of 92), with deltas and same-quarter closes.
The answer in one line: days_in_quarter = 92; days_elapsed = 66; coverage_ratio_active_pipeline = 0.75; elapsed_fraction = 0.7174 (+1 more below)
days_in_quarter92
coverage_ratio_active_pipeline0.75
days_elapsed66
elapsed_fraction0.7174
metrics{"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
pace_ruleexpected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required1
graded checksmetrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 GOALS UPDATE (Week of day 66 of 92 — 71.7% of quarter elapsed; pace basis = target × 66/92 = target × 0.7174)

Volume metrics (higher is better)

SQMs
- QTD actual: 230 | Target: 300 | Delta: 230 - 300 = -70
- Pace: expected by now = 300 × 0.7174 = 215.2. 230 vs 215.2 = +14.8 → AHEAD

SQOs
- QTD actual: 84 | Target: 120 | Delta: 84 - 120 = -36
- Pace: expected = 120 × 0.7174 = 86.1. 84 vs 86.1 = -2.1 → BEHIND (marginally)

DS2s
- QTD actual: 40 | Target: 75 | Delta: 40 - 75 = -35
- Pace: expected = 75 × 0.7174 = 53.8. 40 vs 53.8 = -13.8 → BEHIND

Same-quarter closes
- QTD actual: 10 | Target: 20 | Delta: 10 - 20 = -10
- Pace: expected = 20 × 0.7174 = 14.3. 10 vs 14.3 = -4.3 → BEHIND

Rate metric (lower is better)

Closed-lost MIA rate
- QTD actual: closed_lost_mia / closed_lost_total = 5 / 25 = 0.20 (20%)
- Target: 0.10 (10%) | Delta: 0.20 - 0.10 = +0.10 (10 pts above ceiling) → BEHIND
- Note: this is a point-in-time ratio, not a cumulative count, so days-elapsed pacing does not apply. At 20%, the rate is currently 2x the 10% target.

Pipeline coverage

Active pipeline
- QTD actual: $3,000,000 | Target: $4,000,000 | Delta: 3,000,000 - 4,000,000 = -$1,000,000
- Coverage: 3,000,000 / 4,000,000 = 0.75 → 75% of target
- Pace: expected = 4,000,000 × 0.7174 = $2,869,565. 3,000,000 vs 2,869,565 = +$130,435 → AHEAD

What moved this week
Week-over-week movement cannot be computed from the data provided: the files contain only cumulative QTD figures as of day 66, with no prior-week snapshot, so any claim about this week's changes would be invented. What the current QTD state shows: SQMs (230, +14.8 vs pace) and active pipeline (75% coverage, +$130K vs pace) are the two metrics tracking ahead; SQOs (84) are essentially on pace, 2 behind expectation; DS2s (40, -13.8 vs pace) and same-quarter closes (10, -4.3 vs pace) are materially behind with 26 days remaining; and the closed-lost MIA rate sits at 20%, double the 10% ceiling, which is a quality issue independent of pacing. If you can supply last week's QTD figures, I can produce the actual weekly deltas.
communication 5 tests
ceo-slack-compressionPASS 1.00
Plain-English question: Compress the forecast analysis into one Slack message the CEO can read in ten seconds.
Prompt — what the model saw
Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0584 · 20s · in 920 / out 3,424 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

Plain-English answer: Three sentences, under 80 words, decision first, every distinct number kept plus the one caveat — all checked by deterministic text rules.
The answer in one line: weighted_number = 115976; must_keep = data-quality caveat — deleting the bad news fails
weighted_number115,976
limitmax 3 sentences, max 80 words, no em-dashes
must_keepdata-quality caveat — deleting the bad news fails
graded checksmax_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Q3 weighted forecast: $44,729 COMMIT (7 deals) + 0.35 x $203,565 BEST_CASE (24 deals) = $115,976.75, but treat it as a ceiling, not a callable number. All 7 COMMITs have empty why-buys, so none of the $44,729 has a documented reason to buy, and 32 of 86 deals ($227,575, including COMMIT Deal-D348E1 at $13,770, 2026-10-15) already sit past quarter end. Have sales leadership re-qualify the 7 COMMITs and the slipped deals before we call this number.
stalled-deal-followup-emailPASS 1.00
Plain-English question: Write a follow-up email for this stalled deal — under 120 words, one prior touch, one ask.
Prompt — what the model saw
Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0803 · 29s · in 14,402 / out 2,464 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

Plain-English answer: An email that references the July 28 milestone-awards demo, makes one clear ask, and invents nothing — Deal-0D2F7A, 31 days since last engagement.
The answer in one line: amount = 5100; word_cap = 120; days_since_last_engagement = 31; deal_alias = Deal-0D2F7A (+4 more below)
amount5,100
days_since_last_engagement31
word_cap120
allowed_facts14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_newENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
deal_aliasDeal-0D2F7A
must_referenceengagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
ownerAlex Franklin
single_cta1
stageDS3
tonewarm, concise, professional (matches prior thread)
graded checksword_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Deal: Deal-0D2F7A
From: Alex Franklin (owner)
To: [prospect first name — not provided in data]
Subject: Re: Your 150-seat pricing recap

---

Hi [First Name],

I'm following up on the pricing recap for 150 seats I sent over on August 5.

When we met on July 28, your team reacted strongly to the automated milestone awards and the points catalog, and you asked for pricing — so I want to make sure this doesn't fall through the cracks.

Would you have 15 minutes next week to walk through the numbers and agree on next steps? Happy to adjust timing if priorities have shifted.

Best,
Alex Franklin

---

Word count: 87 (under 120)
Prior touch referenced: ENG-00002, the July 28 demo (milestone awards, points catalog, pricing request)
Clear ask: one 15-minute call next week
funnel-digest-paragraphPASS 1.00
Plain-English question: Write the weekly GTM digest — one paragraph per team, every number traceable.
Prompt — what the model saw
Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0162 · 14s · in 606 / out 670 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

Plain-English answer: Four paragraphs (Marketing, Sales, CS, PLG) using only the allowed numbers, no contact names, no performance commentary, under 400 words.
The answer in one line: word_cap = 400
word_cap400
allowed_numbers16 items — e.g. 2, 3, 9, 08
bannedcustomer contact names, employee performance commentary, compensation
required_paragraphsMarketing, Sales, CS, PLG
graded checksfour_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing
A big round of recognition for the webinar channel this week, which delivered 18 of our 46 SQMs — nearly four in ten from a single source. Overall volume came in at 46 against a weekly target of 52, so we're closing the week 6 SQMs short of goal and looking to make that up. On the efficiency side, cost per SQM on paid search held steady at $150, keeping spend predictable while we work the top of funnel.

Sales
Sales earned their kudos this week, converting 14 SQOs and setting 9 DS2 meetings to keep momentum building. The team created $310,000 in new pipeline, and 3 deals closed within the same quarter. That's the kind of follow-through worth celebrating — every converted SQO is a customer story in the making.

Customer Success
CS showed up big: 2 renewals saved this week, and Team NPS moved to 61. Two saves is two relationships protected, and that deserves a shout-out. As we head into next week, the team is carrying 3 open red-flag accounts — the early-warning radar is doing its job, and next week is about turning those flags into wins.

PLG
Self-serve kept growing the family: 412 new signups joined this week, with activation holding at 31 percent. The moment we love most — 38 companies reached the aha milestone of 10 recognition gives. Each one of those companies is now building the habit of appreciation in the flow of work, and that's exactly the flywheel we want spinning.
partner-digest-sectionPASS 1.00
Plain-English question: Write the partner digest so every partner in the roster appears exactly once.
Prompt — what the model saw
Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0588 · 10s · in 14,450 / out 1,020 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

Plain-English answer: Four active partners with status and numbers, four quiet partners with one line each, and partner-sourced pipeline that reconciles to the deal data.
The answer in one line: rule = every roster partner named exactly once; quiet partners get one line; no partners outside roster.
active_no_dealsWorkWell Group
partner_sourced_pipelinecount=5; amount=550000
per_partner_pipeline{"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
quiet_partnersRecogniq, KudosWave, PeopleFirst Advisors, TotalPerk
rosterpartner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
ruleevery roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checksall_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
PARTNER DIGEST — 2026-08-24 to 2026-09-04

Partner-sourced pipeline: 5 deals, $550,000
(180,000 + 95,000 + 140,000 + 60,000 + 75,000 = 550,000)

ACTIVE PARTNERS

Apex Rewards Co — 2 sourced opps, $275,000 (Deal-DDAAF2 $180,000 + Deal-2CF33E $95,000). Both in DS1. Co-webinar locked for 09-15.

HRCloud Partners — 1 sourced opp, $140,000 (Deal-F1CDA5). Integration referral passed security review; opp moved to DS2.

CultureBridge — 2 sourced opps, $135,000 (Deal-096E1D $60,000 + Deal-067213 $75,000). Both early stage, from the lunch-and-learn. Deal data does not specify a DS stage.

WorkWell Group — 0 sourced deals this period, $0. Wants to restart the joint playbook in Q4; planning call booked for 09-09.

QUIET PARTNERS

Recogniq — quiet; no activity or sourced deals this period.
KudosWave — quiet; no activity or sourced deals this period.
PeopleFirst Advisors — quiet; no activity or sourced deals this period.
TotalPerk — quiet; no activity or sourced deals this period.

Reconciliation: Slack-sourced counts (Apex 2, HRCloud 1, CultureBridge 2, WorkWell 0) match the 5 Partner-UTM deals in partner_deals.csv. All 8 roster partners accounted for; no deals exist for quiet partners.
rfp-answer-from-policyPASS 1.00
Plain-English question: Answer the 10 RFP items from the policy excerpts only — no guessing.
Prompt — what the model saw
Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0197 · 7s · in 789 / out 869 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

Plain-English answer: Seven answered with excerpt citations (Q1-Q7), three flagged unanswerable with the document that would answer them (Q8-Q10).
The answer in one line: answered_count = 7; rule = unanswerable items must be flagged and the document that would answer them named; do not guess.
answered_count7
answer_map{"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
ruleunanswerable items must be flagged and the document that would answer them named; do not guess.
unanswerableQ8, Q9, Q10
valid_citation_idsP1, P2, P3, P4, P5, P6, P7, P8
graded checksanswered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1 — Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2 — Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3 — Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. The report is available to prospects under NDA. [P2]

Q4 — Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. [P3] Note on backups: production backups are retained for 35 days, so residual data in backups expires within that window. [P6]

Q5 — A current list of subprocessors is maintained at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

Q6 — Yes. SAML 2.0 single sign-on is available on the Enterprise plan. SCIM user provisioning is also available on that plan. [P5]

Q7 — Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8 — Unanswerable from the excerpts provided. None of P1–P8 references ISO 27001. A current ISO 27001 certificate (with certificate number, scope, and issuing body) or the certifications page of the trust portal would answer this.

Q9 — Unanswerable from the excerpts provided. None of P1–P8 states an uptime commitment. The Master Services Agreement or its SLA/Service Availability schedule would answer this.

Q10 — Unanswerable from the excerpts provided. None of P1–P8 addresses HIPAA or Business Associate Agreements. Legal/privacy documentation — a BAA template or HIPAA compliance statement — would answer this.
ops-maintenance 5 tests
skill-file-reconciliationPASS 1.00
Plain-English question: Audit a skill set against its manifest and flag every conflict with a severity and an action.
Prompt — what the model saw
Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.4336 · 97s · in 79,482 / out 13,013 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

Plain-English answer: A severity-ranked list of TRIM_DESC / UPDATE_BODY / DELETE_SKILL / MERGE / REVIEW actions covering every rule violation found.
The answer in one line: files_audited = 14; rank = CRITICAL / WARNING / INFO with one fix action per issue
files_audited14
findingsduplicate trigger, circular delegation chain, dangling file reference, oversized description, hardcoded values, manifest drift
rankCRITICAL / WARNING / INFO with one fix action per issue
graded checksduplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
SKILL SET vs MANIFEST RECONCILIATION — 14 files, 14 manifest rows
All evidence below comes only from the provided files and manifest. Manifest char/line counts are taken as given.

------------------------------------------------------------------
F1 · CRITICAL · MERGE — Duplicated ALWAYS-trigger phrases: comms-drafter ↔ email-drafter
------------------------------------------------------------------
Evidence (both descriptions, verbatim): "write me an email", "draft a follow-up", "help me reply (to this)", "what should I say", "bump email", "contract nudge", plus identical scope lists (outbound, follow-ups, post-demo recaps, pricing/contract follow-up, EOQ pushes, renewal/expansion, QBR follow-ups, onboarding). Both bodies carry the same contract follow-up benchmark email word-for-word, the same Recommended/Softer/Firmer output contract, the same 1–10 review scheme, and the same deal-strategy-coach lane marker. Unlike the validator → claim-compressor → feedback chain (which self-sequences explicitly), these two have no ordering or routing rule — a "draft a follow-up" request fires both nondeterministically.
Proposal: MERGE — keep comms-drafter (scope superset: it alone covers Intercom/support and partner comms), port email-drafter's unique assets (Gmail signature retrieval, no-markdown-in-body rule, HubSpot→Granola→Gong transcript order) into it, retire email-drafter, and repoint deal-strategy-coach's manager-email handoff to the merged skill.

------------------------------------------------------------------
F2 · WARNING · TRIM_DESC — Second trigger collision: weekly-pipeline-report ↔ pipeline-intelligence-report
------------------------------------------------------------------
Evidence: "pipeline update" appears in both ALWAYS-trigger lists; "run the pipeline report" (PIR) vs "generate the pipeline report" / "do the pipeline report" (weekly); "what's the pipeline look like" (PIR) vs "what does pipeline look like" (weekly). Deliverables differ (weekly SQM/SQO/bookings numbers vs scored/tiered pipeline), but the triggers don't.
Proposal: TRIM_DESC — partition phrases by deliverable: weekly-pipeline-report owns weekly numbers phrasing ("this week's numbers", "mid-month pipeline check"); pipeline-intelligence-report owns scoring phrasing ("score the pipeline", "tiered pipeline"). Remove "pipeline update" from one of the two.

------------------------------------------------------------------
F3 · WARNING · REVIEW — Circular delegation chain: deal-strategy-coach ↔ email-drafter
------------------------------------------------------------------
Evidence: deal-strategy-coach: "When drafting manager-to-prospect emails, use the email-drafter skill which automatically retrieves your Gmail signature." email-drafter: "For deal strategy, diagnosis, or coaching... use deal-strategy-coach instead" and "do the draft here and suggest deal-strategy-coach for the deeper analysis." No terminal condition on either side. comms-drafter → deal-strategy-coach ("For deep deal strategy, use deal-strategy-coach") adds a third entry point into the same loop.
Proposal: REVIEW — break the cycle by making the handoff one-way (deal-strategy-coach drafts manager emails inline using its own signature logic and drops the delegation), or add a handoff-origin guard so email-drafter's lane marker is suppressed when invoked from deal-strategy-coach.

------------------------------------------------------------------
F4 · CRITICAL · REVIEW — Dangling delegation target: prospect-research-multithreading
------------------------------------------------------------------
Evidence: delegated to by 3 of 14 skills — comms-drafter ("invoke prospect-research-multithreading in Contact Lookup mode first"), email-drafter ("invoke prospect-research-multithreading in Contact Lookup mode first"), deal-strategy-coach ("Cross-skill handoff — prospect-research-multithreading... always offer the handoff"). No manifest row; no file provided. Every unknown-recipient and multithreading path dead-ends unconditionally.
Proposal: REVIEW — either create prospect-research-multithreading and add a manifest row, or strip the handoffs from all three bodies and inline a manual verify-name/title step.

------------------------------------------------------------------
F5 · WARNING · REVIEW — Dangling delegation targets: analysis-validator's specialist octet + skill-orchestrator
------------------------------------------------------------------
Evidence: §12.4 mandates delegation ("always delegate to specialist skill") to 8 skills with no manifest row and no file: bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, bonusly-datadog-questions. §11 cascades corrections to skill-orchestrator, and signalforge-feedback's activation checklist requires registration in skill-orchestrator — also absent from the manifest.
Proposal: REVIEW — if these exist outside this skill set, note that scope explicitly; if not, downgrade §12.4's "always delegate" rules to inline validation, since they are currently unexecutable.

------------------------------------------------------------------
F6 · INFO · REVIEW — Out-of-manifest org skills: bonusly-brand, signalforge-reports
------------------------------------------------------------------
Evidence: bonusly-brand is a mandatory Step 0 for comms-drafter and email-drafter and is referenced by sales-forecast and signalforge-claim-compressor. signalforge-reports (via /mnt/skills/organization/ paths) is a mandatory pre-build read for pipeline-intelligence-report and weekly-pipeline-report. Both are described as org skills, so they may legitimately live outside this manifest — but that cannot be confirmed from the data provided.
Proposal: REVIEW — add manifest rows flagged as org-scoped/external so dependency audits are complete.

------------------------------------------------------------------
F7 · WARNING · UPDATE_BODY — Version conflict: AE roster (Core 6 vs 5)
------------------------------------------------------------------
Evidence: analysis-validator §12.3 ("GTM Team Roster, Updated May 4, 2026") defines "Core 6 AEs" including Hugo Lindqvist (77260721) and states any full-AE-team filter "must include all six IDs." pipeline-intelligence-report Phase 1 ("AE owner IDs, verified May 2026") lists only 5 — Hugo Lindqvist is omitted. Same vintage, different content: PIR's By-AE tab and name resolution will silently drop or mislabel Lindqvist's deals.
Survivor: analysis-validator §12.3 — explicitly dated, strictly a superset, and already the canonical lookup table for G2-F ID resolution.
Proposal: UPDATE_BODY — add Hugo Lindqvist 77260721 to pipeline-intelligence-report's owner map; longer-term, single-source the roster (stale-pipeline-report's dynamic owner resolution is the correct pattern).
Sub-conflicts noted: loss-taxonomy coverage target 95%+ (closed-lost-analysis) vs 97%+ (pipeline-intelligence-report Loss Intel spec); sales-forecast Step 1A header still reads "Open Q2 Deals" although v1.1 made the skill quarter-agnostic.

------------------------------------------------------------------
F8 · INFO · TRIM_DESC — Manifest descriptions exceeding 1,024 chars: 0
------------------------------------------------------------------
Arithmetic: largest values are pipeline-intelligence-report 1006 and signalforge-claim-compressor 1006; 1006 < 1024 → count exceeding = 0. Headroom: 18 chars for those two, 20 chars for partner-digest (1004). Any additive edit to those three descriptions breaches the cap.
Proposal: TRIM_DESC (prophylactic) — shave the three descriptions above 1,000 chars to create headroom; no compliance action required today.

------------------------------------------------------------------
F9 · WARNING · UPDATE_BODY — Hardcoded page IDs, dates, and person names in skill bodies
------------------------------------------------------------------
Notable instances (not exhaustive):
- partner-digest: cloud ID 73fe98de-..., space ID 1958248479, folder ID 2286616609, canonical issue page 2286321666, five partner page IDs (2265382925, 2236940297, 2237825028, 2239365136, 2238283777), Slack user ID U03QLMBL7AR, person names in the output template and pull specs (Amani Phipps, Kelli, Jen Lee, Hani, Bryce, Sara), dated title examples.
- signalforge-feedback: page ID 2295136266, parent 2234417154, Build Log 2247295002, cloudId.
- sales-forecast: space ID 2232811524, parent page 2232582148, cloudId; person name Alaina (and "Elena" survives in the 1.0 changelog entry).
- weekly-pipeline-report: Ben Lavin in the H1 and throughout, two spreadsheet IDs, static Q1 2026 actuals ($365,152 / $2,490,532) and hardcoded Q2 2026 business-day math that will go stale.
- analysis-validator: full named roster with owner IDs (§12.3), "Escalate to Finance (Manish or Amani)" in G1-K and §10, date-stamped constants (May 4, 2026 roster; March 28, 2023 DEALS staleness).
- closed-lost-analysis: named customer losses with embedded dates as taxonomy examples (MinIO May 4–12, Estee Lauder RFP, Softheon May 2026, Aurora Innovation, GCash, LIFTOFF, Nestlé, Ozinga, Ethos Cannabis, StickerYou).
- stale-pipeline-report: Slack channel ID C0561C1JCPJ, owner ID 55483190, org ID 1973303, example dates (5/7, 5/15, 5/19). next-to-close and pipeline-intelligence-report also hardcode org ID 1973303 and named owner IDs.
Proposal: UPDATE_BODY — externalize volatile identity data (page/folder/channel IDs, named individuals, dated rosters, static actuals) into a reference file or live-resolution step; keep structural constants (stage IDs, pipeline = 'default', org ID) hardcoded. Person names in example prose (claim-compressor's Panopto/Felix Construction) are acceptable if relabeled as illustrative.

------------------------------------------------------------------
F10 · INFO · REVIEW — Manifest drift, both directions: none at file level
------------------------------------------------------------------
Arithmetic: files provided = 14; manifest rows = 14; declared_name matches the frontmatter name for all 14. Files with no manifest row = 0. Manifest rows with no file = 0. description_chars and body_lines were accepted as provided (not independently recomputed against file contents).
The drift that actually exists is referential, covered in F4–F6: 12 skill names are invoked from bodies with no manifest row (1 prospect-research-multithreading + 8 bonusly-*-questions specialists + skill-orchestrator + bonusly-brand + signalforge-reports).
Proposal: REVIEW — reconcile the manifest against invoked skill names, not just files on disk; either add rows for real external skills or remove the invocations.

------------------------------------------------------------------
Summary: 10 findings — 2 CRITICAL (F1 merge, F4 dangling handoff), 5 WARNING (F2, F3, F5, F7, F9), 3 INFO (F6, F8, F10). No files were rewritten, per instruction.
runbook-from-slack-threadPASS 1.00
Plain-English question: Turn the resolved incident thread into a runbook.
Prompt — what the model saw
Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0306 · 13s · in 958 / out 1,563 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

Plain-English answer: Ordered steps from the thread, each with the exact command, who ran it, how success was verified, and rollback where state changed.
The answer in one line: rule = no steps beyond what the thread contains; every step traces to a message id.
ambiguous_steporder=4; msg=M05; mark=needs-confirmation
ruleno steps beyond what the thread contains; every step traces to a message id.
stepsorder=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
graded checksordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
RUNBOOK: reward-worker queue depth incident (#incident-2026-08-29-reward-queue, 2026-08-29, resolved)

1. Acknowledge PagerDuty alert, take Incident Commander role
   - Who: Bryce Harmon [M01]
   - Action: acknowledge alert (reward-worker queue depth > 10k), declare IC
   - State change: no system state change
   - Verification: none recorded in thread
   - Rollback: N/A

2. Check queue depth
   - Who: Farid Osman [M02]
   - Command: bundle exec rake sidekiq:queue_depth
   - State change: no (diagnostic)
   - Result: reward queue at 48,213 pending jobs; normal is under 500
   - Rollback: N/A

3. Inspect dead set
   - Who: Farid Osman [M03]
   - Command: none recorded — NEEDS CONFIRMATION (how the dead set was inspected)
   - State change: no (diagnostic)
   - Result: 112 dead jobs, all Redis::TimeoutError from around 13:58
   - Rollback: N/A

4. Pause enqueue to stop the bleed
   - Who: Farid Osman [M04]
   - Command: bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'
   - State change: yes (feature flag disabled)
   - Verification: NEEDS CONFIRMATION — thread does not record direct confirmation that the flag took effect; queue later fell to 9,400 at 14:33 [M07], but that was after worker scale-up in step 6
   - Rollback: bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)' [M04]

5. Clear the dead set
   - Who: Elena Sinclair [M05]
   - Command: NEEDS CONFIRMATION — thread says only "cleared out the dead set" from the console; no command recorded
   - State change: yes (112 dead jobs removed)
   - Verification: NEEDS CONFIRMATION — none recorded in thread
   - Rollback: NEEDS CONFIRMATION — none recorded in thread; clearing a dead set is typically destructive, so whether rollback was possible at all needs confirmation

6. Scale workers up
   - Who: Bryce Harmon [M06]
   - Command: kubectl scale deployment/reward-worker --replicas=6 (from 3)
   - State change: yes (3 -> 6 replicas)
   - Verification: queue depth down to 9,400 and falling ~1,200/min, observed by Farid Osman at 14:33 [M07]
   - Rollback: kubectl scale deployment/reward-worker --replicas=3 [M06]

7. Verify queue drained
   - Who: Cole Ingram [M08]
   - Command: bundle exec rake sidekiq:queue_depth
   - State change: no (verification)
   - Result: returns 0; Datadog error rate back to baseline
   - Rollback: N/A

8. Re-enable enqueue
   - Who: Bryce Harmon [M09]
   - Command: bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'
   - State change: yes (feature flag re-enabled)
   - Verification: 40 new jobs processed cleanly in the next 3 minutes [M09]
   - Rollback: NEEDS CONFIRMATION — none recorded for this step (the disable command from step 4 / M04 would presumably apply, but the thread does not state that)

9. Scale workers back down
   - Who: Bryce Harmon [M10]
   - Command: kubectl scale deployment/reward-worker --replicas=3
   - State change: yes (6 -> 3 replicas)
   - Verification: queue stable at 0; incident declared resolved [M10]
   - Rollback: NEEDS CONFIRMATION — none recorded for the scale-down (the scale-up command from step 6 / M06 would presumably apply, but the thread does not state that)

Open items requiring confirmation: exact dead-set inspection and clearing commands (steps 3, 5), verification and rollback for dead-set clearing (step 5), direct verification that the flag disable took effect (step 4), and explicit rollback statements for steps 8 and 9.
datadog-incident-timelinePASS 1.00
Plain-English question: Sequence the failure from the logs — what broke first, and what cascaded after it?
Prompt — what the model saw
From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0492 · 28s · in 1,618 / out 2,664 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

Plain-English answer: A reward-service Redis timeout at 14:01:12Z cascades to sidekiq at 14:01:40 and api-gateway at 14:03:05, with the Datadog query that confirms the first error.
The answer in one line: job_class_involved = RewardGiveJob
cascade_orderservice=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
clean_servicespostgres
confirming_queryservice:reward-service status:error "@message:Redis::TimeoutError"
first_errortimestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
job_class_involvedRewardGiveJob
logs_do_not_showroot cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
graded checksfirst_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
FAILURE SEQUENCE — datadog_logs.csv + sidekiq_jobs.csv

1. FIRST ERROR
- 2026-09-03T14:01:12Z | reward-service | ERROR | "Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s"
- Prior log lines (13:55:13Z–13:59:30Z) are all INFO across reward-service, web-app, api-gateway — no warning signs in the slice.

2. CASCADE (in order, timestamps from the slice)
1. 14:01:12Z — reward-service: Redis::TimeoutError connecting to redis-primary:6379 (5s timeout).
2. 14:01:20Z / 14:01:30Z / 14:01:40Z — reward-service: "retry exhausted for RewardGiveJob" x3.
3. 14:01:40Z–14:06:47Z — sidekiq: "RewardGiveJob failed: Redis::TimeoutError; retrying" x6 (first at 14:01:40Z says "retrying in 60s").
4. 14:02:30Z — sidekiq WARN: "Queue reward depth above 10,000" (backlog forming ~1m18s after first error).
5. 14:03:05Z — api-gateway: first "502 upstream timeout calling reward-service /gives" (1m53s after first error); repeats x5 through 14:06:52Z.
6. 14:03:30Z — web-app: first "Give form submission failed: upstream 502 from api-gateway" (2m18s after first error — first user-facing symptom); repeats x4 through 14:06:49Z.
7. 14:06:52Z–14:22:10Z — gap with no application logs; only postgres "checkpoint complete" INFO lines.
8. 14:22:10Z — reward-service: "Redis connection restored; resuming job processing" (20m58s after first error).
9. 14:24:45Z — sidekiq: "Queue reward depth below 500" (drain confirmed 2m35s after restore).

3. SERVICE AND JOB
- Service: reward-service (Redis client failing against redis-primary:6379).
- Primary job: RewardGiveJob (log messages + J-00001 through J-00012 in sidekiq_jobs.csv, all Redis::TimeoutError, failed_at 14:01:46Z–14:02:58Z; earliest is J-00005 at 14:01:46Z).
- Secondary: RecognitionDigestJob (J-00013–J-00016, failed_at 14:02:36Z–14:05:50Z, same error) — present in sidekiq_jobs.csv only; no log line names it.
- Note: sidekiq_jobs.csv failed_at values (14:01:46Z–14:01:57Z for the first six) fall between the "retry exhausted" log lines and the first sidekiq "failed; retrying" line — the logs do not link job IDs to log events.

4. DATADOG QUERY TO CONFIRM THE FIRST ERROR
Log Explorer:
  service:reward-service status:error "Redis::TimeoutError"
  Time range: 2026-09-03 13:55:00 UTC – 14:05:00 UTC, sort by timestamp ASCENDING.
  Expected first row: 2026-09-03T14:01:12Z — "Connection to Redis at redis-primary:6379 timed out after 5s".
To prove nothing earlier exists, run service:reward-service status:error over the wider window 13:00–14:01 UTC and confirm zero results.

5. WHAT THE LOGS DO NOT SHOW
- Root cause of the Redis timeout: no redis-primary logs, no network/infra/failover events in the slice.
- Why Redis recovered at 14:22:10Z ("restored" is stated, cause is not).
- What happened 14:06:52Z–14:22:10Z (~16 min): no reward-service, sidekiq, web-app, or api-gateway entries at all.
- Job IDs, payloads, retry counts, or final disposition (retried-to-success vs. dead/discarded) for J-00001–J-00016.
- Any mention of RecognitionDigestJob in log text, despite 4 failures in sidekiq_jobs.csv.
- User/request impact: no request counts, user IDs, or error rates.
- Metrics: no latency, memory, CPU, or Redis connection-pool telemetry — the Datadog slice is logs only.
- Whether postgres was affected: its "checkpoint complete" INFO lines continue unchanged, suggesting no, but no data confirms it.
feature-flag-state-summaryPASS 1.00
Plain-English question: Summarize what each feature flag controls and who has it on.
Prompt — what the model saw
Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0356 · 19s · in 480 / out 1,985 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: Recognition streaks on for 42 beta companies, redeem-flow redesign off with a 12-company target list, plus any flag with no code reference or no targets.
The answer in one line: rule = company counts must equal the export; do not describe flags absent from the export.
code_referencedrecognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
flags{"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
flags_with_no_targetspaused_offboard_cleanup
orphan_flags_no_code_referencelegacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
rulecompany counts must equal the export; do not describe flags absent from the export.
graded checksorphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
FEATURE FLAG SUMMARY (based solely on flags_export.csv + flag_code.md)

1) recognition_streaks_v2 — ON
   Controls: streak tracking on recognition events. Per app/models/recognition.rb, when enabled it calls StreakTracker.record(give).
   Targeting: segment:beta_companies — 42 companies.

2) points_budget_guardrails — ON
   Controls: points budget enforcement. Per app/services/budget_service.rb, when enabled it calls BudgetService.new(company).enforce!(giver, points).
   Targeting: all_companies — 220 companies.

3) slack_dm_nudges — ON
   Controls: Slack DM nudge delivery. Per app/jobs/nudge_job.rb, the job exits early unless the flag is enabled; when on, it calls SlackDm.send_nudge(user).
   Targeting: segment:region_na — 87 companies.

4) redeem_flow_redesign — OFF
   Controls: redeem flow version. Per app/controllers/redeem_controller.rb, enabled renders RedeemV2Component; otherwise renders RedeemV1Component. Since state is OFF, targeted companies get RedeemV1Component.
   Targeting: targeted_list — 12 companies.

5) analytics_dashboard_v3 — ON
   Controls: analytics dashboard version. Per app/controllers/analytics_controller.rb, when enabled it sets @dashboard = AnalyticsV3.new(company).
   Targeting: segment:tier_three — 65 companies.

6) ms_teams_app_v2 — OFF
   Controls: MS Teams app v2 install. Per app/services/teams_installer.rb, when enabled it calls TeamsAppV2.install(company).
   Targeting: targeted_list — 9 companies.

7) legacy_give_modal — OFF
   Controls: unknown — no code reference in flag_code.md.
   Targeting: segment:legacy_plan — 14 companies.

8) survey_boosters_q3 — ON
   Controls: unknown — no code reference in flag_code.md.
   Targeting: segment:legacy_plan — 7 companies.

9) paused_offboard_cleanup — OFF
   Controls: unknown — no code reference in flag_code.md.
   Targeting: none listed (targeting_rules blank) — 0 companies.

FLAGS WITH NO CODE REFERENCE
- legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup — these appear in the export but have no usage in the provided code excerpt.

FLAGS WITH NO TARGETS
- paused_offboard_cleanup — blank targeting_rules and company_count 0.

NOTES / DATA LIMITS
- The export lists segments and rule types only (beta_companies, all_companies, region_na, tier_three, legacy_plan, targeted_list). No individual company names are provided, so I can't say which specific companies have any flag.
- legacy_give_modal (14) and survey_boosters_q3 (7) both target segment:legacy_plan, but the data doesn't show whether those groups overlap.
- company_count is per flag; the same company may appear in multiple flags, so counts are not additive to a unique-company total.

ARITHMETIC (counts as given in the export)
- Rows in export: 9
- ON flags: 42 + 220 + 87 + 65 + 7 = 421 company-flag assignments
- OFF flags: 12 + 9 + 14 + 0 = 35 company-flag assignments
- Total: 421 + 35 = 456 (sum of all company_count values)
nda-triagePASS 1.00
Plain-English question: Triage each NDA green, yellow, or red — and quote the clause that decides it.
Prompt — what the model saw
Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0195 · 6s · in 875 / out 839 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

Plain-English answer: NDA-1 green (standard mutual terms), NDA-2 yellow for an embedded 18-month non-solicit, with the driving clause quoted for every classification.
The answer in one line: non_solicit_in = NDA-2
checksclassification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
classifications{"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_inNDA-2
graded checksgreen · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — [PARTY A] and Bonusly: GREEN
Driving clause: "No license, no obligation to proceed, no exclusivity." (Clause 5), supported by full standard carve-outs in Clause 3 (a–e) and Delaware governing law.
Reasoning: Mutual, 2-year term with 3-year survival, complete carve-outs, familiar US governing law — nothing non-standard.

NDA-2 — [PARTY B] and Bonusly: YELLOW
Driving clause: "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement." (Clause 4)
Reasoning: Otherwise-standard mutual NDA (3-year term, full carve-outs, Delaware law) but carries an embedded employee non-solicit surviving 18 months post-term — a hiring restriction counsel should approve before signature.

NDA-3 — [PARTY C] and Bonusly: RED
Driving clause: "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business." (Clause 2)
Reasoning: Broad 3-year non-compete on Bonusly as Recipient, compounded by one-way confidentiality in favor of [PARTY C], a 5-year term, no stated confidentiality carve-outs (Clause 4 states none), and exclusive Irish governing law/jurisdiction — multiple compounding red flags requiring full legal review.