← back to all models ← RevenueBench home
mistralai

mistralai/mistral-large-4-0

Mean 0.932 · 33/40 perfect tests · $1.86 total · modalities in: text, image · out: text · each card: the prompt → the correct answer (gold) → this model's actual answer

What do the modalities mean?

Modalities in means what you can send this model: text, images, files, video, or audio. Modalities out means what it can send back. Bench tasks feed text extracts, so modalities do not change the score. They are here so you know what this model can handle in real work.

data-crm 5 tests
stage-hygiene-auditPASS 1.00
Plain-English question: How much of the open DS2+ pipeline is missing a substantive why-buys, and is anything missing DS1 timestamps or out of order?
Prompt — what the model saw
Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.1544 · 72s · in 75,638 / out 13,964 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

Plain-English answer: 81.5% of the 124 DS2+ deals are missing a substantive why-buys — Alex Franklin 48, Cole Ingram 19, Dana Mercer 18, Bryce Harmon 15, Farid Osman 1 — and every deal has a DS1 timestamp with none regressed.
The answer in one line: n_deals = 156; ds2plus = 124; noncomp_pct = 81.5; bryce_noncomp = 15; no_ds1 = 0; regressed = 0 (+1 more below)
n_deals156
bryce_noncomp15
ds2plus124
no_ds10
noncomp_pct81.5
regressed0
per_owner_noncompAlex Franklin=48 of 61 (78.7%); Cole Ingram=19 of 22 (86.4%); Dana Mercer=18 of 22 (81.8%); Bryce Harmon=15 of 18 (83.3%); Farid Osman=1 of 1 (100%)
graded checksnoncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- DS2+ = stage DS2, DS3, DS4, or DS5.
- why-buys compliant = why_buys has 15+ characters and is not tbd / n/a / see notes; LONG(n) counts as compliant per rule.
- No DS1 = t_ds1 is blank.
- Regressed = any present earlier-stage timestamp later than a present later-stage timestamp (t_ds1..t_ds5 out of order).

Numbers:
- DS2+ deals: 124 of 156 open deals.
- why-buys non-compliant overall: 101 / 124 = 81.5%. All 101 are blank why_buys; no tbd / n/a / see notes values appear, 23 are LONG(n) compliant.
- By owner:
  - Bryce Harmon: 15 / 18 = 83.3% non-compliant
  - Alex Franklin: 48 / 61 = 78.7% non-compliant
  - Dana Mercer: 18 / 22 = 81.8% non-compliant
  - Cole Ingram: 19 / 22 = 86.4% non-compliant
  - Farid Osman: 1 / 1 = 100.0% non-compliant
- Deals with no DS1 timestamp: 0.
- Deals regressed: 0.

Process fixes:
1. Make why-buys required to advance to DS2; block stage save when blank, under 15 characters, or placeholder.
2. Add a DS2+ forecast hygiene report filtering blank why-buys by owner for weekly cleanup.
3. Stamp t_ds1 automatically at creation and lock stage timestamps from manual edit to preserve order.
owner-resolution0.83
Plain-English question: Every open deal has a blank owner name — resolve each one to a real person using the owners table.
Prompt — what the model saw
The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0400 · 361s · in 48,800 / out 1,414 tok
✓ resolved_count✗ unmatched_none✓ deactivated_flagged✓ bryce_amount✓ alex_amount✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: All 156 deals resolve to six named owners; jake_amount maps to Bryce Harmon ($1,054,144) and justin_amount to Alex Franklin ($624,310).
The answer in one line: jake_amount = 1,054,144; justin_amount = 624,310; resolved = 156
jake_amount1,054,144
justin_amount624,310
resolved156
deactivated
per_ownerBryce Harmon=1,054,144; Alex Franklin=624,310; Dana Mercer=341,195; Cole Ingram=288,161.43; Farid Osman=4,134; Elena Sinclair=2,100
graded checksresolved_count · unmatched_none · deactivated_flagged · bryce_amount · alex_amount · no_invented

Model's actual answer

show the model's full answer
All 156 open deals in `deals_open.csv` resolved against the owners table. Arithmetic shown; aliases cited exactly.

**Deals resolved:** 156 of 156 (every `owner_id` in the file has a match in the owners table).

**Deals with no owner match:** 0. None to list.

**Owner ids mapping to archived/deactivated owners:** 0 deals. The owners table has two archived owners — `1520255671` Gavin Porter (archived=true) and `77260721` Hugo Lindqvist (archived=true) — but neither id appears on any open deal, so no deal aliases to list.

**Total pipeline per resolved owner** (amount summed per owner, all stages/categories included):

| Owner | owner_id | Deals | Pipeline total |
|---|---|---|---|
| Bryce Harmon | 119337721 | 35 | $1,054,144.00 |
| Alex Franklin | 84342457 | 67 | $624,310.00 |
| Dana Mercer | 83155923 | 24 | $341,195.00 |
| Cole Ingram | 83155924 | 22 | $288,161.43 |
| Farid Osman | 716654662 | 7 | $4,134.00 |
| Elena Sinclair | 701163055 | 1 | $2,100.00 |

Arithmetic check: 35 + 67 + 24 + 22 + 7 + 1 = 156 deals. $1,054,144.00 + $624,310.00 + $341,195.00 + $288,161.43 + $4,134.00 + $2,100.00 = $2,314,044.43, which equals the grand total of all 156 `amount` values in the file. No names were guessed; every name above comes directly from the owners table row for that id.
gong-hubspot-join-integrityPASS 1.00
Plain-English question: How many call records point at deal ids that do not exist in the open pipeline?
Prompt — what the model saw
Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.1135 · 36s · in 73,884 / out 4,870 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

Plain-English answer: 30 Gong call rows are orphans keyed to dead deal ids, none are duplicates, and DS3 call coverage is 29.4%.
The answer in one line: orphans = 30; ds3_cov_pct = 29.4; dup = 0
orphans30
ds3_cov_pct29.4
dup0
orphan_aliasesDeal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
graded checksorphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
(a) Orphans: 30 of 67 gong rows have no match in deals_open.

Arithmetic: 67 gong rows - 37 matched = 30 orphans. 30/67 = 44.8%.

Sample orphan aliases (all from gong file, absent from deals_open): Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42.

Note: gong_owner_id not used for matching; it is a Gong user id, not a HubSpot owner id.

(b) Duplicate conversation keys: 0.

Arithmetic: 0 rows where calls_90d > distinct_conversation_keys. In every gong row, calls_90d = distinct_conversation_keys.

(c) Open deals at DS3 or later with at least one logged call: 25 of 85, = 29.4%.

Arithmetic: 85 open deals at DS3/DS4/DS5; 25 of those deal ids appear in gong table. 25/85 = 0.2941 = 29.4%.
snowflake-sql-trapped-catalog0.83
Plain-English question: Write one Snowflake query for first-month usage per customer using only the provided, trap-laden catalog.
Prompt — what the model saw
Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0316 · 283s · in 21,056 / out 8,251 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✗ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

Plain-English answer: A single correct SQL query that uses the right catalog tables and avoids the trap columns — graded on the exact tables and columns.
The answer in one line: use_table = PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
avoid_stalePRODUCTION.HUBSPOT.DEALS (stale)
avoid_unpopulatedV2_LIVE.OBJECTS_DEALS (unpopulated)
forbidden_filterdeleted-giver exclusion must NOT be applied
outputone fenced SELECT with per-FROM/JOIN table-choice comment
required_filtersucceeded redemptions only
use_tablePRODUCTION.PLG.COMPANY_COHORT_SUMMARY
graded checksavoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- ============================================================================
-- STATUS: THE REQUESTED QUERY CANNOT BE WRITTEN FROM THE PROVIDED CATALOG.
-- No combination of tables in schema_catalog.md can return, per customer
-- company, for its first calendar month as a customer: unique givers,
-- recognition count, successful redemption count. Missing inputs:
--
-- MISSING 1 - Recognition/giving events. No table in this catalog records
--   giving/recognition events with a giver ID, a company ID, and an event
--   timestamp. Unique givers and recognition counts are impossible without
--   it. The documented rule about "a filter excluding deleted givers" belongs
--   to giving queries against such a table; that table is absent here, and
--   per the documented rule the filter must NOT be applied to historical
--   giving counts in any case. No invented giving table is used below.
--
-- MISSING 2 - Company dimension / deal-to-company linkage. Needed for "per
--   customer company". V2_DAILY.OBJECTS_DEALS documents no company column,
--   and the only association table provided (ASSOCIATIONS_DEALS_TO_CONTACTS)
--   links deals to CONTACTS, not companies. No companies table and no
--   company aliases are provided in the catalog.
--
-- MISSING 3 - Redemption event columns. PRODUCTION.DEPRECATED_RECOGNITION.
--   REDEMPTION_RECORDS_V2 is the documented redemption source (only
--   STATE = 'succeeded' rows count), but the excerpt documents no company
--   or date columns for it, so a per-company, per-month redemption count
--   cannot be safely written against it from this catalog.
--
-- TABLES REVIEWED AND EXCLUDED (per catalog notes):
--   HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS - UNPOPULATED. Do not use.
--   PRODUCTION.HUBSPOT.DEALS                  - STALE, last sync 2023-03. Do not use.
--   PRODUCTION.HUBSPOT.GONG_HUBSPOT_MAP_FAST  - RETIRED pattern. Do not use.
--   PRODUCTION.PLG.COMPANY_COHORT_SUMMARY     - Self-serve companies only;
--     the excerpt documents no company key column, M1_USERS is users (not
--     unique givers) and must not be substituted for givers, and the M1
--     month anchor is not defined. Cannot satisfy this request.
--
-- INTENDED AGGREGATION once the missing tables are supplied:
--   first calendar month as a customer = DATE_TRUNC('month', <first customer date>)
--   unique givers          = COUNT(DISTINCT GIVER_ID)  -- NO deleted-giver filter
--   recognition count      = COUNT(*)                   -- NO deleted-giver filter
--   successful redemptions = COUNT(*) filtered to STATE = 'succeeded'
--
-- No FROM or JOIN appears below because no table in the provided catalog
-- contains the required inputs; every candidate table's disposition is
-- documented above instead.
-- ============================================================================

SELECT 'unique givers' AS requested_metric,
       'MISSING SOURCE DATA' AS status,
       'No recognition/giving events table with giver IDs exists in the provided catalog' AS reason
UNION ALL
SELECT 'recognition count',
       'MISSING SOURCE DATA',
       'No recognition/giving events table in the provided catalog; per documented rule, the deleted-giver exclusion filter must NOT be applied to historical giving counts'
UNION ALL
SELECT 'successful redemption count',
       'MISSING SOURCE DATA',
       'REDEMPTION_RECORDS_V2 company/date columns are not documented in the catalog excerpt; PLG M1_REDEMPTIONS covers self-serve companies only and has no documented company key'
UNION ALL
SELECT 'per customer company (all metrics)',
       'MISSING SOURCE DATA',
       'No company dimension table and no deal-to-company association exists in the provided catalog';
```
crm-completeness-enrichmentPASS 1.00
Plain-English question: Audit the CRM for missing and invalid fields, and propose the fixes.
Prompt — what the model saw
Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0576 · 257s · in 38,004 / out 14,507 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

Plain-English answer: 9 companies missing employee count, 6 missing HQ country, 4 invalid emails, 1 domain mismatch, 2 duplicate clusters, and 25 enrichment matches.
The answer in one line: n_contacts = 52; n_companies = 34; enrichment_matches = 25; invalid_email_count = 4; n_dup_clusters = 2; domain_mismatch_count = 1 (+4 more below)
n_contacts52
domain_mismatch_count1
enrichment_matches25
invalid_email_count4
n_companies34
n_dup_clusters2
company_missingindustry=0; employee_count=9; hq_country=6
domain_mismatchesCT-0011
duplicate_clusters['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
invalid_emailsCT-0010, CT-0080, CT-0081, CT-0192
graded checksdup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
Audit complete. One input is missing: **no deals file was provided**, so deal completeness (owner, stage, amount, close date, why-buys) cannot be computed, and the final top-10 cannot be ranked by pipeline amount. The top-10 below is ranked by data-quality impact instead. All numbers below are computed from the three files provided; nothing is invented.

## 1. Completeness per field

Companies (34 rows):

| Field | Complete | Arithmetic | Missing rows |
|---|---|---|---|
| industry | 100% | 34/34 | none (but 10 rows use nonstandard variants, see §5) |
| employee_count | 73.5% | 25/34 | C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386, C-93C8BF |
| hq_country | 82.4% | 28/34 | C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB |

Contacts (52 rows):

| Field | Complete | Arithmetic | Notes |
|---|---|---|---|
| email (non-empty) | 100% | 52/52 | but 4 are invalid, see §4 |
| email (valid format) | 92.3% | 48/52 | invalid: CT-0010, CT-0080, CT-0081, CT-0192 |
| title | 75.0% | 39/52 | 13 missing: CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170 |
| persona | 71.2% | 37/52 | 15 missing: CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181 |

Deals: not provided. Cannot report.

Other gaps: 14 of 34 companies have zero contacts (C-2C60E5, C-B97B4E, C-2D7423, C-7BBDFA, C-50D386, C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20, C-0A092931, C-0A092932, C-0A092933, C-0A092934). No orphan contacts (every contact's company_alias exists in companies.csv).

## 2. Duplicate company clusters (shared domain)

Cluster 1 — acme-corp.com (2 rows):
- C-0A092931: Technology, 500, US
- C-0A092932: tech, 510, USA
- Conflicts: employee_count 500 vs 510 (real conflict); industry and country are the same after normalization.
- Neither row is in the enrichment export, so the 500 vs 510 conflict cannot be resolved from provided data.
- Survivor: C-0A092931 (canonical spellings "Technology"/"US"). Employee count must be verified manually — do not auto-pick 500 or 510.

Cluster 2 — globex.io (2 rows):
- C-0A092933: SaaS, 200, US
- C-0A092934: Technology, 200, US
- Conflict: industry SaaS vs Technology. Employee count and country agree.
- Not in the enrichment export.
- Survivor: C-0A092933 — "SaaS" is more specific than "Technology". Flag for verification since no third-party source is available.

No other shared domains; all other domains are unique.

## 3. Invalid emails and domain mismatches

Invalid emails (truncated, no domain) — 4:
- CT-0010: "user0@" (C-66D1FC)
- CT-0080: "user0@" (C-92D97D)
- CT-0081: "user1@" (C-92D97D)
- CT-0192: "user2@" (C-425E2A)

Domain mismatch — 1:
- CT-0011 (C-66D1FC): email user1@other-domain.com, but company/contact domain is 66d1fc.com.

All other 47 valid emails match their company domain, and every contact's domain column matches its company's domain.

## 4. Enrichment fills (matched by domain, fill only where ZI has a row and a value)

employee_count — 8 fillable, all from ZoomInfo = 400:
- C-EC3025 (ec3025.com) <- 400
- C-96039F (96039f.com) <- 400
- C-44EA29 (44ea29.com) <- 400
- C-D04904 (d04904.com) <- 400
- C-B23205 (b23205.com) <- 400
- C-60C75F (60c75f.com) <- 400
- C-7BBDFA (7bbdfa.com) <- 400
- C-50D386 (50d386.com) <- 400
- C-93C8BF: NOT fillable — no enrichment row for 93c8bf.com.

hq_country — 0 fillable:
- C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5 each have an enrichment row but zi_hq_country is blank.
- C-EE9FFB has no enrichment row (ee9ffb.com not in export).
- All 6 stay missing; no value may be invented.

## 5. CRM vs enrichment disagreements (both present, matched by domain)

Industry — 10 rows where CRM says "Technology"/"tech"/"Tech " and ZoomInfo says "Computer Software":
- C-66D1FC, C-EC3025, C-44EA29, C-92D97D, C-D04904, C-77A95A, C-AA8DDA, C-B25F40, C-60C75F, C-425E2A
- Recommended source: ZoomInfo. It is the third-party firmographic provider, the value is more specific, and the CRM side is internally inconsistent (3 spellings of the same industry across 10 rows: "tech" x4, "Tech " x4 with trailing space, "Technology" x9 in the full file).

Employee count — no disagreements: on all 16 matched rows where both sides have a value, they are equal.

hq_country — no true disagreements: CRM "US" (9 rows), "USA" (6), "United States" (2) vs ZI "United States" are the same country. Recommended fix is normalization, not a source pick: standardize on one label (ZI's "United States" if ZI is the canonical source).

Additional normalization (not CRM-vs-ZI, but dirty in CRM alone): industry variants "tech" x4, "Tech " x4 (trailing space), "health care" x2 (C-7BBDFA, C-50D386 — note CRM and ZI agree on the nonstandard "health care"; recommend "Healthcare").

## 6. Top 10 fixes (impact-ranked; NOT amount-ranked — no deals/pipeline data was provided)

1. Merge duplicate cluster acme-corp.com (C-0A092931 / C-0A092932); keep C-0A092931 as survivor; manually verify employee count (500 vs 510, no enrichment row to arbitrate).
2. Merge duplicate cluster globex.io (C-0A092933 / C-0A092934); keep C-0A092933 as survivor; manually verify industry (SaaS vs Technology, no enrichment row).
3. Repair 4 invalid emails: CT-0010, CT-0080, CT-0081, CT-0192 (all truncated at "userN@", no domain) — without these, outreach to C-66D1FC, C-92D97D, C-425E2A is broken.
4. Fix domain mismatch on CT-0011 (user1@other-domain.com vs company domain 66d1fc.com) — likely a personal email or a typo; verify before keeping.
5. Fill 8 missing employee_count values from ZoomInfo (all 400): C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386. (C-93C8BF has no enrichment row — leave missing.)
6. Normalize industry taxonomy: map "tech"/"Tech " variants and adopt ZI "Computer Software" for the 10 matched rows listed in §5; map "health care" -> "Healthcare" on C-7BBDFA, C-50D386.
7. Normalize hq_country: collapse "US"/"USA"/"United States" (17 rows) to one standard label.
8. Source personas for 15 contacts (71.2% complete) — no enrichment source covers persona; needs CRM/outreach input.
9. Source titles for 13 contacts (75.0% complete) — same, no provided source.
10. Source hq_country for 6 companies (C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB) — enrichment is blank or absent; cannot be filled from provided data.

To rank these by pipeline amount at stake, send the deals extract (with amount and company linkage) and I'll re-rank.
deal-intelligence 5 tests
closed-lost-classification0.60
Plain-English question: Why did we lose the last six months of deals — one primary category per deal, and whose side is the reason on?
Prompt — what the model saw
Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.1270 · 117s · in 60,063 / out 12,206 tok
✗ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

Plain-English answer: Every closed-lost deal gets exactly one category (pricing, competitor, no decision, timing, product gap, champion left, other) plus a side, per the computed classification.
The answer in one line: total_lost_6mo = 90; top_reason = competitor (26)
total_lost_6mo90
categoriespricing, competitor, no decision, timing, product gap, champion left, other
patternstwo most actionable, tag-vs-text disagreements counted
sideBonusly / buyer / unknown per deal
top_reasoncompetitor (26)
graded checkstotal_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Side rule, applied only from tag + free text:
Bonusly = evaluated and beaten on feature, price, or named vendor. Buyer = buyer delay, priority, budget approval, scope, or leadership call. Unknown = unresponsive, spam, or vague "different direction/vendor/platform" with no confirmed reason.

Every deal, alias | primary category | side:

Deal-DB0AAC | timing | buyer
Deal-F7F635 | competitor | unknown — "another direction", no vendor/reason named
Deal-AC944F | no decision | unknown — "unresponsive"
Deal-214060 | no decision | unknown — "unresponsive"
Deal-91A056 | timing | buyer — reconnect early 2027
Deal-29326C | timing | buyer
Deal-5DB9B0 | other | unknown — tag Does not fit ICP, "Spam."
Deal-831B7B | timing | buyer — revisit new year
Deal-F97C37 | competitor | Bonusly — other vendor, more diversified offerings
Deal-13E9CF | no decision | buyer — R&R deprioritized, explicitly not budget
Deal-39E25C | timing | buyer — reconnect next year
Deal-7ED004 | pricing | buyer — did not get budget approval
Deal-21B045 | no decision | unknown — "MIA"
Deal-B3ABED | timing | buyer — revisit Q2 next year for 2028 budget
Deal-422BA6 | competitor | Bonusly — competing vendor, ADP TotalSource partner differentiator
Deal-ED9AE7 | timing | buyer — "Timing, budget, authority."
Deal-988493 | no decision | unknown — "mia"
Deal-381C8C | competitor | unknown — tag Competitor, text only "not moving forward"
Deal-F308CA | no decision | unknown — no contact since April, ignored outreach
Deal-F1E8A6 | competitor | unknown — "not moving forward", no reason
Deal-B6AC09 | timing | buyer — revisiting in 2027
Deal-70F704 | no decision | unknown — anniversary-awards scope then MIA
Deal-E6E80A | timing | buyer — pushed early 2027
Deal-B038F0 | timing | buyer — pushed early 2027
Deal-4664E1 | no decision | unknown — no contact after intro
Deal-175756 | timing | buyer — on hold until 2027, other priorities
Deal-E74A73 | no decision | buyer — test manually before investing, maybe next year
Deal-DDAB52 | competitor | Bonusly — Rippl, more at same cost, no exchange-rate issue
Deal-ACE061 | competitor | unknown — "I feel they went with HeyTaco", unconfirmed
Deal-BB78F3 | timing | buyer — plant action items first
Deal-D48E0B | no decision | unknown — "MIA"
Deal-15DA99 | timing | buyer — early 2027
Deal-F4AF5D | timing | buyer — early next year
Deal-79B7A1 | timing | buyer
Deal-583ADB | no decision | unknown — "MIA"
Deal-8E27DA | no decision | buyer — swag only, didn't want R&R
Deal-2D2F8D | competitor | unknown — "different direction", no detail
Deal-E0441F | other | unknown — stale inherited from departed rep, no contact
Deal-7CB44D | no decision | unknown — no contact since demo
Deal-0F96AA | competitor | Bonusly — lost RFP before finalist demo
Deal-1BCA50 | competitor | Bonusly — budget/gift-card details, stakeholder down path with another vendor
Deal-7CC678 | competitor | unknown — "Nothing specific provided."
Deal-FAC17C | no decision | buyer — no final approval from Executive IT Director
Deal-242273 | product gap | Bonusly — lost on digitizing points currency + onsite spend
Deal-50E5D8 | no decision | buyer — leadership pause
Deal-A2C349 | competitor | Bonusly — stuck with Awardco + surveying
Deal-9F176A | timing | buyer — pause until end of year
Deal-7B2236 | pricing | buyer — budget + want simpler/cheaper Kudos board
Deal-AFA56C | no decision | unknown — "unresponsive"
Deal-C7156E | competitor | Bonusly — selected another vendor after evaluation
Deal-C33D91 | pricing | buyer — budget cuts, not approved
Deal-9048EB | product gap | Bonusly — bad fit, multiple feature gaps
Deal-5E64CE | timing | buyer — Nectar exit fee, locked through Oct 2027
Deal-8A0992 | competitor | Bonusly — Canadian provider aligns
Deal-D0C698 | competitor | Bonusly — client wants Kudos again
Deal-69CF3D | timing | buyer — "On Hold"
Deal-ECBF89 | timing | buyer — "On Hold"
Deal-3618CC | product gap | Bonusly — "Wanted Surveys"
Deal-EECC02 | competitor | unknown — "Went another direction."
Deal-5AD03E | product gap | Bonusly — wanted more defined budget access
Deal-D1A623 | timing | buyer
Deal-413C56 | timing | buyer — back-to-school priority, CEO not ready
Deal-47F1A1 | competitor | Bonusly — staying with WorkTango 12 months
Deal-BF2A98 | competitor | Bonusly — deployed HiThrive
Deal-2A292B | no decision | buyer — build simple internally
Deal-D1AABF | no decision | unknown — "No response."
Deal-FEDBCB | timing | buyer — reconnect end of year
Deal-1E7DA9 | competitor | Bonusly — selected another platform after evaluation
Deal-2BBA21 | no decision | unknown — no contact since intro, ignored nudges
Deal-286F9C | competitor | Bonusly — another platform, "not a good fit"
Deal-7FBAC6 | no decision | buyer — leadership pause again
Deal-369281 | competitor | Bonusly — went with Paylocity
Deal-386F6E | no decision | unknown — "No response."
Deal-9FCD0D | competitor | Bonusly — Canadian company, CEO preference
Deal-55867E | timing | buyer — "not moving forward at this time"
Deal-DAFB82 | pricing | buyer — priorities need budget, not budgeted until 2028
Deal-2FEDDB | timing | buyer — unsure on timing
Deal-64B19A | competitor | unknown — "Likely stayed with Motivosity"
Deal-3F86A0 | no decision | unknown — "unresponsive"
Deal-096750 | no decision | unknown — no contact after intro
Deal-F325A5 | champion left | buyer — layoffs and change in leadership
Deal-ABD14C | no decision | buyer — not interested in signing up
Deal-79E61A | no decision | unknown — "Unresponsive."
Deal-8A119B | pricing | buyer — "Didn't get approval."
Deal-AE7C4E | no decision | unknown — "Unresponsive."
Deal-DAB4F1 | no decision | unknown — "Unresponsive."
Deal-B4B50F | no decision | unknown — "Unresponsive"
Deal-981AD4 | product gap | Bonusly — doesn't fit UI, not UK focused
Deal-DC77FE | product gap | Bonusly — price not a factor, lost on customization/label-points-as-dollars
Deal-5885B9 | no decision | unknown — "MIA"

Summary counts, N=89:
timing 24 + competitor 23 + no decision 28 + pricing 5 + product gap 6 + other 2 + champion left 1 = 89.
timing 27.0%, competitor 25.8%, no decision 31.5%.

Side split: Bonusly 21 + buyer 38 + unknown 30 = 89.

Tag-vs-text clear disagreements: 3 deals.
Deal-8E27DA — tag Feature Request, text states no R&R want at all, swag only.
Deal-9048EB — tag MIA, text confirms move to closed/lost plus bad fit and multiple feature gaps.
Deal-3618CC — tag Lost DM, text only "Wanted Surveys", no DM/authority mention.

Two patterns worth acting on:
1. 2027/2028 timing wall: 24 timing losses repeatedly name early-2027 and 2028, e.g. Deal-91A056, Deal-B6AC09, Deal-E6E80A, Deal-B038F0, Deal-15DA99, Deal-B3ABED, Deal-5E64CE, Deal-DAFB82. Without dated re-open cadence this becomes phantom pipeline.
2. Dark loss reasons: 30 unknown-side deals — pure MIA/ghosting plus vague competitor closes like Deal-F7F635, Deal-381C8C, Deal-F1E8A6, Deal-2D2F8D, Deal-7CC678, Deal-EECC02. Separately, decidable product gaps recur around surveys, budget-access controls, points-currency/onsite-spend, customization, and regional fit, e.g. Deal-3618CC, Deal-A2C349, Deal-5AD03E, Deal-242273, Deal-DC77FE, Deal-DDAB52, Deal-8A0992, Deal-9FCD0D, Deal-981AD4.
pipeline-tieringPASS 1.00
Plain-English question: Score all 156 open deals into exactly one of LOCK / ACTION / BUILD / REVIVE / WATCH / RISKY.
Prompt — what the model saw
Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.1591 · 72s · in 103,930 / out 4,583 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

Plain-English answer: All 156 open deals tiered exactly once across the six tiers using forecast category, stage, meetings, contacts, and recency.
The answer in one line: total = 156
total156
graded checksjson_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{"tier_counts": {"LOCK": 3, "ACTION": 33, "BUILD": 39, "REVIVE": 20, "WATCH": 53, "RISKY": 8}, "tier_examples": {"LOCK": ["Deal-D348E1", "Deal-C26D20", "Deal-403845"], "ACTION": ["Deal-25F752", "Deal-C6FE92", "Deal-7E2131"], "BUILD": ["Deal-D73B89", "Deal-012CB1", "Deal-523604"], "REVIVE": ["Deal-0660B4", "Deal-BA571A", "Deal-2D1F1B"], "WATCH": ["Deal-6787C2", "Deal-C9C286", "Deal-332637"], "RISKY": ["Deal-547B2B", "Deal-B7EBD1", "Deal-A2B47C"]}, "risky_deals": ["Deal-547B2B", "Deal-B7EBD1", "Deal-A2B47C", "Deal-2465CE", "Deal-584EE5", "Deal-A5E80A", "Deal-FD9F4E", "Deal-499BF6"], "lock_violations": 0, "pipeline_shape": "Top-heavy early pipeline: 3+8+33+39+20+53=156 deals; 92/156 with zero meetings_30d; only 3 COMMIT DS4/DS5 with meetings support LOCK while 6 COMMIT DS5 with zero meetings plus 2 early-stage COMMIT drive RISKY; bulk sits in PIPELINE DS1-DS3 WATCH/BUILD with scattered recent meetings."}
call-transcript-extractionPASS 1.00
Plain-English question: Extract the CRM write-back fields from each call transcript as JSON.
Prompt — what the model saw
For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0148 · 160s · in 1,905 / out 5,754 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

Plain-English answer: Per transcript: prospect-only why-buys, pain points, stakeholders, budget and timeline signals, and competitors — e.g. TX-001: '$40k earmarked', live before open enrollment.
The answer in one line: rep = Alex Franklin
allowed_newTX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
per_transcript{"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
repAlex Franklin
rep_sourced_nulls{"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
graded checksjson_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "Automating anniversary and birthday awards — \"our HR team of three cannot keep up with it manually\" (Prospect (VP People))"
    ],
    "pain_points": [
      "Manual tracking in a spreadsheet; \"people slip through the cracks\" (Prospect (HR Admin))"
    ],
    "stakeholders": ["Prospect (VP People)", "Prospect (HR Admin)"],
    "budget_signal": "\"We have about $40k earmarked for engagement tools this fiscal year\" (Prospect (VP People))",
    "timeline_signal": "\"Ideally we would have this live before open enrollment in November\" (Prospect (VP People))",
    "competitor_mentioned": "Achievers — \"We looked at Achievers last year, but it was too heavy for a team our size\" (Prospect (VP People))",
    "next_step": "Security review on September 12 — \"Yes — let's do the security review on September 12\" (Prospect (VP People))",
    "objections": [
      "Needs SSO and audit logs for IT sign-off (Prospect (HR Admin))"
    ],
    "confidence": "high — budget earmarked, go-live deadline stated, security review scheduled"
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "Tie recognition to retention for hourly workforce (Prospect (Head of Total Rewards))"
    ],
    "pain_points": [
      "\"Regretted turnover there is over 30%\" among hourly workforce (Prospect (Head of Total Rewards))"
    ],
    "stakeholders": ["Prospect (Head of Total Rewards)", "Prospect (CFO)"],
    "budget_signal": "\"Finance has approved a $25k pilot budget for this quarter\" (Prospect (CFO))",
    "timeline_signal": "\"We want a decision by end of September\" (Prospect (CFO))",
    "competitor_mentioned": null,
    "next_step": "Send pilot agreement; prospect will route it to legal this week — \"Yes — send the pilot agreement and we'll route it to legal this week\" (Prospect (CFO))",
    "objections": [
      "\"Integration with Workday has to be rock solid — that's my one condition\" (Prospect (CFO))"
    ],
    "confidence": "high — finance-approved pilot budget, decision date stated, pilot agreement agreed"
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "\"We need to make recognition visible across our 12 retail locations\" (Prospect (People Ops Manager))"
    ],
    "pain_points": [
      "\"Store managers have zero budget autonomy for on-the-spot recognition today\" (Prospect (People Ops Manager))"
    ],
    "stakeholders": ["Prospect (People Ops Manager)"],
    "budget_signal": null,
    "timeline_signal": "\"Honestly there's no rush on our side until Q1\" (Prospect (People Ops Manager))",
    "competitor_mentioned": "Bucketlist — \"My CEO used Bucketlist at her last company and liked it\" (Prospect (People Ops Manager))",
    "next_step": "Schedule a call with the CEO — \"Yes, let's schedule a call with our CEO — I'll send two times\" (Prospect (People Ops Manager))",
    "objections": [
      "\"The CEO has to be sold first — she decides anything people-related\" (Prospect (People Ops Manager))"
    ],
    "confidence": "low — no budget stated, timeline deferred to Q1, decision-maker (CEO) not yet engaged"
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "\"We want to consolidate three separate recognition tools into one\" (Prospect (VP People))"
    ],
    "pain_points": [
      "\"We're paying for three tools and none of them talk to our HRIS\" (Prospect (VP People))"
    ],
    "stakeholders": ["Prospect (VP People)", "Prospect (IT Security Lead)"],
    "budget_signal": "\"If it's under $15k annually, I can approve it without going to the board\" (Prospect (VP People))",
    "timeline_signal": "\"Our procurement cycle runs six to eight weeks minimum\" (Prospect (IT Security Lead))",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "\"The security review took three months for our last vendor — that's my hesitation\" (Prospect (IT Security Lead))",
      "Six-to-eight-week procurement cycle (Prospect (IT Security Lead))"
    ],
    "confidence": "medium — budget authority under $15k and clear consolidation pain, but long procurement, security hesitation, and no confirmed next step"
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "\"Automate service milestones\" (Prospect (HR Director))",
      "\"Give us analytics on recognition equity across departments\" (Prospect (HR Director))"
    ],
    "pain_points": [
      "\"Our night-shift teams feel invisible — their engagement scores run 20 points lower\" (Prospect (People Ops Coordinator))",
      "\"Our exec team is skeptical after a failed rollout two years ago\" (Prospect (HR Director))"
    ],
    "stakeholders": ["Prospect (HR Director)", "Prospect (People Ops Coordinator)"],
    "budget_signal": "\"We have $12k approved under our engagement line\" (Prospect (HR Director))",
    "timeline_signal": "\"We need this running before our January all-hands\" (Prospect (HR Director))",
    "competitor_mentioned": "Nectar — \"We're mid-pilot with Nectar right now, so you'd need to beat that experience\" (Prospect (HR Director))",
    "next_step": "Present to exec team on October 2 — \"Yes — come present to our exec team on October 2\" (Prospect (HR Director))",
    "objections": [
      "\"We're mid-pilot with Nectar right now, so you'd need to beat that experience\" (Prospect (HR Director))",
      "\"Our exec team is skeptical after a failed rollout two years ago\" (Prospect (HR Director))"
    ],
    "confidence": "medium — approved budget and exec presentation scheduled, but active Nectar pilot and exec skepticism"
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "\"I'd love to cut the admin time on service awards\" (Prospect (HR Manager))"
    ],
    "pain_points": [
      "\"I personally spend five hours a month ordering and shipping plaques\" (Prospect (HR Manager))"
    ],
    "stakeholders": ["Prospect (HR Manager)"],
    "budget_signal": "\"Budget isn't the issue — time is\" (Prospect (HR Manager)) — no amount stated",
    "timeline_signal": "\"Our fiscal year resets in February, so a Q1 start is realistic\" (Prospect (HR Manager))",
    "competitor_mentioned": null,
    "next_step": "Send one-page overview; HR Manager will forward it to the COO this week — \"Yes — send the one-page overview and I'll forward it to our COO this week\" (Prospect (HR Manager))",
    "objections": [
      "\"Fair warning, our COO usually prefers building things in-house\" (Prospect (HR Manager))",
      "Comparing against doing it internally — \"Nobody else — we're comparing this against just doing it internally\" (Prospect (HR Manager))"
    ],
    "confidence": "medium — acute personal pain, budget not a constraint, next step agreed; but single-threaded on the HR Manager and COO bias toward in-house build"
  }
]
```

Notes on judgment calls:
- TX-004 next_step is null: the prospect said "Maybe — I need to check her calendar, no promises" — not an explicit agreement.
- TX-003 budget_signal is null: the only budget-adjacent prospect statement is that store managers have zero budget autonomy, which is a pain point, not a budget figure. The $8/employee/month figure came from the rep and was excluded.
- TX-006 competitor_mentioned is null: "doing it internally" is not a named competitor; it is captured under objections.
- Stakeholders list only prospect speakers from the speaker list; the rep (Alex Franklin) is excluded.
- No arithmetic was required — all figures are prospect-stated and quoted verbatim.
next-to-close0.75
Plain-English question: Which 3 deals are closest to signature right now?
Prompt — what the model saw
Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0063 · 73s · in 1,214 / out 1,936 tok
✗ exactly_three✓ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

Plain-English answer: Deal-547B2B ($11,200), Deal-B7EBD1 ($9,000), and Deal-403845 ($9,000) — all DS5 COMMIT deals closing September 10-11, with what is left on each.
blocked_excludeddeal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
exactly_three1
ranking_ruleSlack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
top3rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
graded checksexactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
Closest 3 to signature (CRM COMMIT/DS5 cross-checked against #sales-sparring, 2026-09-04):

1. Deal-547B2B — $11,200 — Alex Franklin, close 2026-09-11. Why close: Slack 09:12 — redlines came back clean, signing page is out, their VP People said they are signing tomorrow; Alex calls it "signature-imminent." Left: the signature itself.

2. Deal-403845 — $9,000 — CRM owner Alex Franklin (Dana Mercer reported it in Slack 10:02), close 2026-09-11. Why close: the order form is with their finance team — one step from signature. Left: finance approval and countersignature.

3. Deal-A2B47C — $6,360 — Alex Franklin, close 2026-09-11. Why close: Slack 09:44 confirms it is warm at normal legal-review pace, and CRM has it DS5/COMMIT with a next-day close date. Left: legal review completion.

Arithmetic: 11,200 + 9,000 + 6,360 = $26,560 combined.

Excluded despite CRM COMMIT:
- Deal-2465CE ($5,400, close 2026-09-10): Slack 09:20 — champion left the company and procurement froze new vendors; Dana is pulling it out of commit. Blocked, not close, even though CRM still shows COMMIT.
- Deal-B7EBD1 ($9,000, close 2026-09-10, earliest close date in CRM): no Slack signal either way, so unverified; ranked below Deal-A2B47C, which Slack confirms is actively moving.
- Deal-D348E1 ($13,770, close 2026-10-15): warm per Slack 09:44 but later close date and still in legal review.
product-gap-detectionPASS 1.00
Plain-English question: Which prospect complaints are true product gaps versus rollout or enablement issues?
Prompt — what the model saw
Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0191 · 183s · in 7,557 / out 5,332 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

Plain-English answer: Two real gaps (SCIM provisioning, ADP Workforce Now integration) separated from plan gates and enablement issues, each with the prospect quote and deal.
allowed_newTG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
enablement_issuestranscript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voicetranscript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
plan_gatestranscript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
real_gapstranscript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
graded checksreal_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Amounts are not in the provided files — `transcripts_gaps.csv` has no amount column and `product_docs.md` has none either. I am not pulling amounts from other workspace files, per "use only the data provided." Deals are named from the transcript's `deal_alias`.

CANDIDATES (all 5 transcripts)

1) TG-001 — Deal-EC3025 — Prospect (IT Security Lead):
"We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
Classification: REAL GAP. Docs: "SCIM user provisioning and ADP Workforce Now integrations are NOT currently listed as supported capabilities." SCIM is absent on every tier (not Core/Pro/Enterprise-gated), so this is a missing capability, not a plan gate. Amount: not provided in provided data.

2) TG-002 — Deal-D0D6B5 — Prospect (HRIS Manager):
"Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
Classification: REAL GAP. Docs list supported HRIS as Workday, BambooHR, Gusto, Rippling (Pro and above) and state ADP Workforce Now is "NOT currently listed." ADP is absent on all tiers — a missing integration, not a plan gate. Amount: not provided in provided data.

3) TG-003 — Deal-CFE7F4 — Prospect (VP People):
"I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?"
Classification: PLAN GATE. Docs: "Custom report builder" is yes on Enterprise, — on Core and Pro. The capability exists; it is tier-gated. Not a real gap. Amount: not provided in provided data.

4) TG-004 — Deal-84DBA6 — Prospect (People Ops Manager):
"We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it."
Classification: ROLLOUT/ENABLEMENT ISSUE. Docs: Slack integration is supported on all plans. The complaint is lack of manager training, not a missing capability. Not a real gap. Amount: not provided in provided data.

5) TG-005 — Deal-36C33F — Prospect (HR Manager):
"Good to know. The web version should be fine for our office staff for now."
Classification: EXCLUDED — no prospect-raised gap. The mobile-app line ("we don't have a native mobile app for hourly workers yet") is spoken by Alex Franklin (rep); rep language does not count as prospect voice. The prospect's own line accepts the web version. (Mobile app is also not addressed in the docs excerpt.) Amount: not provided in provided data.

Arithmetic / tally: 5 candidates → 2 real gaps (TG-001, TG-002), 1 plan gate (TG-003), 1 rollout/enablement (TG-004), 1 excluded for rep-only voice (TG-005). Dollar total at risk cannot be computed: amounts are not present in the provided data, so none is summed or estimated.

SUMMARY — true product gaps only (2)

- Deal-EC3025 — SCIM user provisioning is not a supported capability on any tier. Prospect (IT Security Lead): security team will not approve any tool requiring manual account management. Amount: not provided in provided data.
- Deal-D0D6B5 — ADP Workforce Now integration is not supported (supported HRIS: Workday, BambooHR, Gusto, Rippling). Prospect (HRIS Manager) calls it a dealbreaker. Amount: not provided in provided data.
rep-performance 5 tests
stale-pipeline-by-rep0.67
Plain-English question: Which open deals have had no email, call, or meeting in the last 7 days?
Prompt — what the model saw
The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.1443 · 98s · in 51,415 / out 17,511 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ bryce_top_amount✓ cole_stale_amt✗ dana_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

Plain-English answer: 65 open deals are stale; the largest is Deal-2D1F1B at $240,000, with Bryce Harmon carrying the most stale dollars at $626,243.
The answer in one line: jake_amt = 626,243; jenn_amt = 261,645; kevin_amt = 252,905.03; top_amount = 240,000; stale_count = 65; top_alias = Deal-2D1F1B
jake_amt626,243
jenn_amt261,645
kevin_amt252,905.03
stale_count65
top_amount240,000
top_aliasDeal-2D1F1B
graded checksstale_recall_ge_80 · grouped_by_owner · bryce_top_amount · cole_stale_amt · dana_stale_amt · stale_count

Model's actual answer

show the model's full answer
Snapshot: 2026-09-05. Stale = max(last_email, last_call, last_meeting) <= 2026-08-28, i.e. days since >= 8.
Days = 2026-09-05 minus last contact date, verified: e.g. 09-05 − 08-28 = 8, − 08-27 = 9, − 08-26 = 10, − 08-25 = 11, − 08-24 = 12, − 08-21 = 15, − 08-20 = 16, − 08-19 = 17, − 08-18 = 18, − 08-17 = 19, − 08-13 = 23, − 08-12 = 24, − 08-10 = 26, − 08-03 = 33, − 07-21 = 46, − 06-16 = 81.
Future last_meeting dates after 2026-09-05 excluded as not yet occurred; recency uses most recent date <= snapshot.

Bryce Harmon:
Deal-2D1F1B, DS1, 240000, last 2026-06-16, 81 days
Deal-66D1FC, DS1, 99000, last 2026-08-20, 16 days
Deal-950043, DS1, 70000, last 2026-08-17, 19 days
Deal-B23205, DS1, 45000, last 2026-08-20, 16 days
Deal-7BBDFA, DS3, 37440, last 2026-07-21, 46 days
Deal-332637, DS2, 36000, last 2026-08-27, 9 days
Deal-1BEEBF, DS1, 31500, last 2026-08-17, 19 days
Deal-A414F6, DS1, 25200, last 2026-08-17, 19 days; future 2026-09-10 ignored
Deal-C5658B, DS1, 23400, last 2026-08-20, 16 days
Deal-40522D, DS3, 21000, last 2026-08-17, 19 days
Deal-C1FA6D, DS1, 18000, last 2026-08-20, 16 days; future 2026-09-15 ignored
Deal-01E193, DS1, 12600, last 2026-08-28, 8 days; future 2026-09-09 ignored
Deal-F0EBBB, DS3, 11400, last 2026-08-12, 24 days
Deal-927338, DS1, 10920, last 2026-08-18, 18 days; future 2026-09-17 ignored
Deal-E25A09, DS1, 6000, last 2026-08-27, 9 days
Deal-C9C286, DS2, 5502, last 2026-08-27, 9 days
Deal-012CB1, DS1, 1, last 2026-08-13, 23 days
Deal-3795AD, DS2, 1, last 2026-08-28, 8 days; future 2026-10-02 ignored

Dana Mercer:
Deal-44EA29, DS2, 60000, last 2026-08-26, 10 days
Deal-E51FB7, DS2, 43875, last 2026-08-24, 12 days
Deal-B42F46, DS1, 27000, last 2026-08-17, 19 days
Deal-BA3DDC, DS3, 23400, last 2026-08-21, 15 days
Deal-9DDE86, DS2, 20000, last 2026-08-21, 15 days
Deal-215CCA, DS3, 18900, last 2026-08-19, 17 days
Deal-5EED42, DS3, 16250, last 2026-08-25, 11 days
Deal-57887A, DS2, 15000, last 2026-08-28, 8 days
Deal-944310, DS4, 10500, last 2026-08-03, 33 days; future 2026-09-15 ignored
Deal-B7EBD1, DS5, 9000, last 2026-08-20, 16 days
Deal-3974EB, DS4, 9000, last 2026-08-28, 8 days
Deal-F40F04, DS2, 8100, last 2026-08-21, 15 days
Deal-7599B8, DS3, 7350, last 2026-08-18, 18 days; future 2026-09-10 ignored
Deal-87DDD1, DS1, 5000, last 2026-08-17, 19 days
Deal-F336B6, DS3, 4200, last 2026-08-21, 15 days
Deal-0660B4, DS4, 1920, last 2026-08-20, 16 days

Alex Franklin:
Deal-CC08D1, DS1, 24000, last 2026-08-20, 16 days
Deal-E73427, DS3, 18000, last 2026-08-26, 10 days
Deal-885F45, DS2, 9300, last 2026-08-24, 12 days
Deal-C2FF3C, DS1, 8316, last 2026-08-26, 10 days
Deal-0D2F7A, DS3, 5100, last 2026-08-24, 12 days
Deal-6C60D4, DS3, 4800, last 2026-08-24, 12 days
Deal-13FEBD, DS2, 4680, last 2026-08-24, 12 days
Deal-819506, DS1, 4400, last 2026-08-28, 8 days; future 2026-09-09 ignored
Deal-9D0060, DS3, 3840, last 2026-08-24, 12 days
Deal-690476, DS2, 3600, last 2026-08-18, 18 days
Deal-C6D97A, DS4, 3240, last 2026-08-28, 8 days
Deal-EE195F, DS3, 3120, last 2026-08-28, 8 days
Deal-278DEC, DS3, 2700, last 2026-08-28, 8 days
Deal-635B8E, DS3, 2600, last 2026-08-18, 18 days
Deal-6883F3, DS1, 2400, last 2026-08-20, 16 days
Deal-4A13AD, DS3, 2160, last 2026-08-10, 26 days
Deal-F67D31, DS2, 1800, last 2026-08-28, 8 days
Deal-5FDCE4, DS3, 1600, last 2026-08-24, 12 days
Deal-BA571A, DS4, 1080, last 2026-08-18, 18 days

Cole Ingram:
Deal-D04904, DS2, 58529.25, last 2026-08-25, 11 days
Deal-B25F40, DS3, 40000, last 2026-08-28, 8 days
Deal-813836, DS2, 32175, last 2026-08-25, 11 days
Deal-1BA595, DS2, 31750, last 2026-08-25, 11 days
Deal-CFE1E8, DS3, 18000, last 2026-08-25, 11 days
Deal-CD47A6, DS2, 12168, last 2026-08-25, 11 days
Deal-627646, DS3, 11193, last 2026-08-25, 11 days
Deal-FF809F, DS2, 7781.2, last 2026-08-25, 11 days
Deal-AF932D, DS2, 7225.4, last 2026-08-25, 11 days
Deal-A71728, DS2, 6947.5, last 2026-08-25, 11 days
Deal-8BC9F5, DS2, 5616, last 2026-08-26, 10 days
Deal-175395, DS3, 4779.88, last 2026-08-25, 11 days
Deal-481E24, DS3, 4140, last 2026-08-26, 10 days
Deal-C7F9BF, DS2, 3360, last 2026-08-25, 11 days
Deal-2F3A66, DS3, 3334.8, last 2026-08-25, 11 days
Deal-342E96, DS2, 2700, last 2026-08-12, 24 days
Deal-E568D5, DS3, 1875, last 2026-08-25, 11 days
Deal-FD9F4E, DS5, 1330, last 2026-08-26, 10 days

Farid Osman:
Deal-8BA24E, DS1, 2880, last 2026-08-28, 8 days; future 2026-09-15 ignored
Deal-8FDCD2, DS1, 1, last 2026-08-21, 15 days; future 2026-09-15 ignored

Missing data, no engagement row, recency cannot be computed:
Deal-3EED2C, Alex Franklin, DS2, 7200
Deal-57FF13, Elena Sinclair, DS1, 2100

Totals, stale only, amounts descending order already applied above:
Bryce Harmon: 18 stale deals; total = 240000+99000+70000+45000+37440+36000+31500+25200+23400+21000+18000+12600+11400+10920+6000+5502+1+1 = 692964
Dana Mercer: 16 stale deals; total = 60000+43875+27000+23400+20000+18900+16250+15000+10500+9000+9000+8100+7350+5000+4200+1920 = 279495
Alex Franklin: 19 stale deals; total = 24000+18000+9300+8316+5100+4800+4680+4400+3840+3600+3240+3120+2700+2600+2400+2160+1800+1600+1080 = 106736
Cole Ingram: 18 stale deals; total = 58529.25+40000+32175+31750+18000+12168+11193+7781.2+7225.4+6947.5+5616+4779.88+4140+3360+3334.8+2700+1875+1330 = 252905.03
Farid Osman: 2 stale deals; total = 2880+1 = 2881
Elena Sinclair: 0 stale with computable recency; 1 missing-data deal.
All stale with computable recency: 73 deals; 240000+99000+70000+45000+37440+36000+31500+25200+23400+21000+18000+12600+11400+10920+6000+5502+1+1 + 279495 + 106736 + 252905.03 + 2881 = 1334981.03
activity-mix-vs-outcomePASS 1.00
Plain-English question: Per rep, what did the last 30 days of emails, calls, and meetings look like next to the deals that advanced?
Prompt — what the model saw
The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.1240 · 392s · in 124,277 / out 13,909 tok
✓ alex_ds2_30d✓ alex_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

Plain-English answer: Alex Franklin is the standout signal with 18 deals entering DS2 in 30 days (next closest: Bryce Harmon with 4), alongside per-rep activity totals.
The answer in one line: alex_ds2_30d = 18; bryce_ds2_30d = 4; note = legacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
alex_ds2_30d18
bryce_ds2_30d4
notelegacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
graded checksalex_ds2_30d · alex_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Window used: t_ds2 from 2026-08-06 through 2026-09-05 inclusive, i.e. within 30 days of the 2026-09-05 snapshot. Activity totals use the provided emails_30d, calls_30d, and meetings_30d columns, summed across each owner's deals in deals_open.csv. Two open deals have no row in engagements_by_deal_90d.csv and therefore contribute 0 activities: Deal-3EED2C and Deal-57FF13. No company alias field was provided; deal aliases below are cited exactly. Gavin Porter and Hugo Lindqvist are archived=true and have no deals in deals_open.csv, so they are not ranked.

Per-rep arithmetic:

Alex Franklin (84342457): 307 emails + 36 calls + 41 meetings = 384 total activities. Mix: 307/384 = 79.9% emails; 36/384 = 9.4% calls; 41/384 = 10.7% meetings. DS2 entries in window: 18 — Deal-D9A72E, Deal-EE195F, Deal-7FA0C3, Deal-E531A6, Deal-36C33F, Deal-D1E6C2, Deal-317E6F, Deal-4F775F, Deal-F436DA, Deal-CA5E44, Deal-46988D, Deal-5296C9, Deal-898FC5, Deal-E73427, Deal-403845, Deal-92D97D, Deal-1FC049, Deal-3EED2C. Efficiency: 384/18 = 21.33 activities per DS2 entry.

Bryce Harmon (119337721): 162 emails + 0 calls + 43 meetings = 205 total activities. Mix: 162/205 = 79.0% emails; 0/205 = 0.0% calls; 43/205 = 21.0% meetings. DS2 entries in window: 4 — Deal-1CCE5C, Deal-25F752, Deal-CA7DC0, Deal-D73B89. Efficiency: 205/4 = 51.25 activities per DS2 entry.

Cole Ingram (83155924): 96 emails + 14 calls + 1 meeting = 111 total activities. Mix: 96/111 = 86.5% emails; 14/111 = 12.6% calls; 1/111 = 0.9% meetings. DS2 entries in window: 2 — Deal-1BA595, Deal-42326B. Efficiency: 111/2 = 55.50 activities per DS2 entry.

Farid Osman (716654662): 38 emails + 0 calls + 34 meetings = 72 total activities. Mix: 38/72 = 52.8% emails; 0/72 = 0.0% calls; 34/72 = 47.2% meetings. DS2 entries in window: 1 — Deal-499BF6. Efficiency: 72/1 = 72.00 activities per DS2 entry.

Dana Mercer (83155923): 84 emails + 18 calls + 11 meetings = 113 total activities. Mix: 84/113 = 74.3% emails; 18/113 = 15.9% calls; 11/113 = 9.7% meetings. DS2 entries in window: 1 — Deal-57887A. Efficiency: 113/1 = 113.00 activities per DS2 entry.

Elena Sinclair (701163055): 0 emails + 0 calls + 0 meetings = 0 total activities. Mix: N/A because total activities = 0. DS2 entries in window: 0 — none. Efficiency: N/A because DS2 entries = 0.

Efficiency ranking, lowest activities per DS2 entry first, among reps with at least one DS2 entry:
1. Alex Franklin — 21.33
2. Bryce Harmon — 51.25
3. Cole Ingram — 55.50
4. Farid Osman — 72.00
5. Dana Mercer — 113.00
Unranked: Elena Sinclair — N/A, 0 DS2 entries.

Most efficient rep: Alex Franklin. Highest-volume rep: Alex Franklin, with 384 total activities. They do not differ. This is descriptive only; no cause is attributed.
ae-qtd-scorecardFAIL 0.00
Plain-English question: Build Alex Franklin's QTD scorecard: bookings versus quota, pipeline, and conversion.
Prompt — what the model saw
Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0206 · 462s · in 25,764 / out 1,475 tok
✗ bookings✗ quota✗ attainment✗ new_expansion_split✗ top_loss_reason✗ coaching_obs

✓ Correct Answer (ground truth)

Plain-English answer: 75% attainment ($150,000 on a $200,000 quota), $113,500 new versus $36,500 expansion, pipeline by stage, and the rolling DS2-to-won rate.
The answer in one line: quota = 200000; bookings = 150000; new_bookings = 113500; expansion_bookings = 36500; open_deals_count = 125; attainment_pct = 75; owner = Alex Franklin (+12 more below)
quota200,000
attainment_pct75
bookings150,000
expansion_bookings36,500
expansion_count3
losses_qtd27
new_bookings113,500
new_count5
open_deals_count125
wins_qtd8
activity_30d_totalsemails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
allowed_new18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
ds2_to_wonwindow_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
ownerAlex Franklin
pipeline_by_stage{"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
prior_quarter_excludeddeal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
quarter2026-Q3
snapshot_date2026-09-05
top_loss_reasonreason=Lost- Timing (1 year or more); count=13
graded checksbookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
I'll compute this from the provided files with a script so the arithmetic is exact.
multithreading-gapPASS 1.00
Plain-English question: Which open deals are single-threaded or under-threaded?
Prompt — what the model saw
Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0340 · 208s · in 14,173 / out 10,282 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

Plain-English answer: Deals with fewer than 2 active contacts (single-threaded) or fewer than 3, or all in one persona (under-threaded), each flagged with the missing persona.
The answer in one line: active_window_start = 2026-07-07
active_window_start2026-07-07
allowed_new59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
best_persona_ruleDS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flag_ruleflag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
flaggeddeal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flaggedDeal-4B0BEB, Deal-84DBA6, Deal-D348E1
persona_universeeconomic buyer, champion, HR admin, IT security, finance
graded checksflagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Reference date: 2026-10-19 (today). "Active" = last_engaged_date on/after 2026-08-20 (today minus 60 days, inclusive) AND is_former=false. Verified by script: cutoff = 2026-08-20, exactly 60 days back.

Data gaps (stated, not invented): the provided files contain NO amount, NO stage, and NO open/closed status field. So amount and stage are "not provided" for every deal, and I treated all 14 rows as the candidate pool because no closed flag exists to filter on. Flagging is therefore based purely on contact threading.

Flag rule applied: single-threaded = active contacts < 2; under-threaded = active contacts < 3 OR all active contacts in one persona. 11 of 14 deals flagged. Not flagged: Deal-84DBA6 (3 active, 3 personas), Deal-4B0BEB (3 active, 3 personas), Deal-D348E1 (4 active, 4 personas).

Most-valuable-persona rule (stage not provided, so gap-based): if no active economic buyer, the economic buyer is the highest-value add (budget authority); where an economic buyer is already active but no champion exists, the champion is the gap. Only Deal-885F45 falls in the second case.

FLAGGED DEALS (11)

1) Deal-EC3025 | C-FDD0C7
- amount: not provided | stage: not provided
- active contacts: 1 (CT-047C54, champion, 2026-09-02)
- flagged: single-threaded AND under-threaded
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: CT-6827DB, Chief People Officer, economic buyer

2) Deal-92D97D | C-E23238
- amount: not provided | stage: not provided
- active contacts: 1 (CT-01F5B4, HR admin, 2026-08-28). CT-A902AE champion is stale (2026-06-01, 140d), not former.
- flagged: single-threaded AND under-threaded
- personas present: HR admin
- personas missing: economic buyer, champion, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: none on file

3) Deal-50D386 | C-EB10E4
- amount: not provided | stage: not provided
- active contacts: 2 (CT-AA41B2 champion 2026-09-01; CT-B9C35B HR admin 2026-08-25)
- flagged: under-threaded (<3)
- personas present: champion, HR admin
- personas missing: economic buyer, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: CT-A1C4B3, Chief People Officer, economic buyer

4) Deal-D0D6B5 | C-32918E
- amount: not provided | stage: not provided
- active contacts: 1 (CT-87CED4, champion, 2026-09-02). CT-DE6D7C (61d) and CT-FD70B2 (73d) are just outside/stale.
- flagged: single-threaded AND under-threaded
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: CT-1FA4DB, Chief People Officer, economic buyer

5) Deal-5BFE3B | C-535D36
- amount: not provided | stage: not provided
- active contacts: 1 (CT-57123B, champion, 2026-08-31). CT-5CE757 stale (68d).
- flagged: single-threaded AND under-threaded
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: none on file

6) Deal-36C33F | C-077A0E
- amount: not provided | stage: not provided
- active contacts: 0. CT-4FE556 IT security stale (65d); CT-405B45 champion former+stale; CT-86B22F economic buyer former+stale.
- flagged: single-threaded AND under-threaded (fully dark)
- personas present: none
- personas missing: economic buyer, champion, HR admin, IT security, finance
- most valuable to add: economic buyer (rebuild from budget authority)
- on-file unengaged fit: CT-1DB73E, Chief People Officer, economic buyer

7) Deal-885F45 | C-5E8EFB
- amount: not provided | stage: not provided
- active contacts: 1 (CT-51C81E, economic buyer, 2026-08-26). CT-D9A0E8 champion stale (69d).
- flagged: single-threaded AND under-threaded
- personas present: economic buyer
- personas missing: champion, HR admin, IT security, finance
- most valuable to add: champion (EB is in place; no internal advocate)
- on-file unengaged fit: CT-B3F25D, IT Security Lead, IT security (fills a missing persona, but NOT the champion gap; no champion on file)

8) Deal-FCBE5B | C-737030
- amount: not provided | stage: not provided
- active contacts: 1 (CT-4A5317, champion, 2026-08-29)
- flagged: single-threaded AND under-threaded
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: none on file

9) Deal-5408B0 | C-2AE3AA
- amount: not provided | stage: not provided
- active contacts: 1 (CT-D33AE4, champion, 2026-09-01). CT-8742FD HR admin stale (62d).
- flagged: single-threaded AND under-threaded
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: CT-07FA76, Chief People Officer, economic buyer

10) Deal-C6D97A | C-5A8FC2
- amount: not provided | stage: not provided
- active contacts: 2 (CT-223DDC champion 2026-08-31; CT-B03555 champion 2026-08-20, exactly 60d, counts active). CT-4E8A2B stale (75d).
- flagged: under-threaded (<3 AND all one persona: champion x2)
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: none on file

11) Deal-F9A08A | C-0D15DF
- amount: not provided | stage: not provided
- active contacts: 1 (CT-931B10, champion, 2026-09-03). CT-913581 economic buyer stale (121d), not former.
- flagged: single-threaded AND under-threaded
- personas present: champion
- personas missing: economic buyer, HR admin, IT security, finance
- most valuable to add: economic buyer
- on-file unengaged fit: CT-697541, Chief People Officer, economic buyer

Summary arithmetic: 14 deals total; 11 flagged (8 single-threaded, of which all 8 also under-threaded; 3 under-threaded only: Deal-50D386, Deal-C6D97A, plus Deal-50D386... recount). Single-threaded (<2 active): Deal-EC3025, Deal-92D97D, Deal-D0D6B5, Deal-5BFE3B, Deal-36C33F, Deal-885F45, Deal-FCBE5B, Deal-5408B0, Deal-F9A08A = 9. Under-threaded (<3 or one persona): all 11 flagged. Under-threaded-only (>=2 active but still flagged): Deal-50D386 (2 active) and Deal-C6D97A (2 active, one persona) = 2. 9 + 2 = 11. Not flagged = 3. Checks: 11 + 3 = 14.

Cross-cutting: 9 of 11 flagged deals have no active economic buyer; 7 of 11 have an on-file unengaged Chief People Officer (economic buyer) ready to engage: Deal-EC3025 (CT-6827DB), Deal-50D386 (CT-A1C4B3), Deal-D0D6B5 (CT-1FA4DB), Deal-36C33F (CT-1DB73E), Deal-5408B0 (CT-07FA76), Deal-F9A08A (CT-697541) = 6 economic-buyer contacts on file. The other on-file unengaged contact (CT-B3F25D, IT security) sits at Deal-885F45, whose top gap is a champion (not on file). Four flagged deals have no on-file unengaged contact at all: Deal-92D97D, Deal-5BFE3B, Deal-FCBE5B, Deal-C6D97A.
talk-track-from-transcriptsPASS 1.00
Plain-English question: From this rep's last 10 calls: how do they open, handle objections, and how often do they lock a next step?
Prompt — what the model saw
From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0082 · 83s · in 2,750 / out 2,322 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

Plain-English answer: 8 of 10 calls open with the same customer-win story; objection handling, next-step agreement rate, and every competitor raised, with quotes.
The answer in one line: n_transcripts = 10; objection_count = 3; rep = Alex Franklin (+3 more below)
n_transcripts10
objection_count3
allowed_new13 items — e.g. TT-001, TT-002, TT-003, TT-004
competitor_quotesAwardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
competitors_prospect_raisedAwardco, Kudos
next_stepagreed=7; total=10; rate=0.7
objectionsobjection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
opener_themetheme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
repAlex Franklin
rep_raised_competitors_excludedWorkhuman
graded checksobjection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
**Opening (first 5 minutes)**

8 of 10 calls open with the identical minute-0 hook (TT-001/Deal-D348E1, TT-002/Deal-5408B0, TT-003/Deal-547B2B, TT-005/Deal-C61CF7, TT-006/Deal-60C2C2, TT-007/Deal-EDC141, TT-008/Deal-D9A12F, TT-010/Deal-84DBA6):
"Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."

The other 2 are tailored:
- TT-004/Deal-403845 (min 0): "I put together a short agenda — security review first, then pricing."
- TT-009/Deal-1E2498 (min 0): "You asked for straight pricing last time, so let's start there."

TT-005/Deal-C61CF7 adds a proactive competitor jab at min 2: "And unlike Workhuman, our pricing includes the full rewards catalog with no extra margin."

**Three most common objections and handling**

1. Budget locked (4 of 10 calls: TT-001/Deal-D348E1, TT-003/Deal-547B2B, TT-006/Deal-60C2C2, TT-010/Deal-84DBA6). Prospect: "Honestly, budget is locked until next fiscal year — I can't add a new line item right now." Rep reframes to ROI (min 8): "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."

2. Revisit next quarter (3 of 10: TT-002/Deal-5408B0, TT-005/Deal-C61CF7, TT-008/Deal-D9A12F). Prospect: "This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater." Rep counters with a scoped pilot (min 8): "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"

3. Spreadsheet + gift cards is enough (3 of 10: TT-004/Deal-403845, TT-007/Deal-EDC141, TT-009/Deal-1E2498). Prospect: "We already do recognition with a spreadsheet and quarterly gift cards — why would we change?" Rep contrasts on automation + analytics (min 8): "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

**Concrete next step rate**

Asked in 7 calls (TT-001, TT-002, TT-003, TT-005, TT-006, TT-008, TT-009): "Should we lock the next step — a working session with your team this week?" Agreed in all 7: "Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager." → 7/7 = 100% when asked.
Across all 10 calls: 7 agreed / 10 calls = 70%. The 3 misses (TT-004/Deal-403845, TT-007/Deal-EDC141, TT-010/Deal-84DBA6) are exactly the 3 calls where the rep never asked — each ended with "Understood — I'll leave it with you." / "Fair enough." / "Understood, thanks for the candor."

**Competitors raised by prospects**

- Awardco — TT-003/Deal-547B2B (min 4): "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
- Kudos — TT-007/Deal-EDC141 (min 4): "How are you different from Kudos? Our CEO used them at her last company."

(Workhuman appears in TT-005/Deal-C61CF7, but the rep raised it unprompted at min 2 — not prospect-raised.)

**Coaching notes**

1. The ask is the whole game: 7/7 next steps land when the rep asks, 0/3 when they don't. Make the working-session ask a mandatory close on every call, including ones where the prospect deflects (TT-004, TT-007, TT-010 all ended with no next step and no follow-up scheduled).
2. The opener is a copy-paste: the same retailer/18% hook opens 8 of 10 calls, and the two tailored openers (TT-004 agenda-first, TT-009 pricing-first per the prospect's own request) show the rep can adapt. Vary the first five minutes by deal stage and by what the prospect has already asked for — and drop the proactive "unlike Workhuman" jab at min 2 of TT-005 unless a competitor is actually in the deal.
reporting-analytics 5 tests
q3-weighted-forecastPASS 1.00
Plain-English question: Build the Q3 2026 weighted forecast from the open pipeline, and show the math deal by deal.
Prompt — what the model saw
Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0593 · 100s · in 27,771 / out 4,935 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

Plain-English answer: A stage-weighted Q3 forecast number computed from the open pipeline, with per-stage math that reconciles to the deal data.
The answer in one line: excluded_amount = 227575; best_case_total = 203565; weighted_forecast = 115,976.75; commit_total = 44729; excluded_out_of_quarter = 32; best_case_count = 24; flags = blank owner column + empty why_buys fields must be surfaced (+2 more below)
excluded_amount227,575
best_case_count24
best_case_total203,565
commit_count7
commit_total44,729
excluded_out_of_quarter32
weighted_forecast115,976.75
flagsblank owner column + empty why_buys fields must be surfaced
top5_best_caseDeal-2D7423, Deal-25F752, Deal-E53952, Deal-5EED42, Deal-FA32A0
graded checkscommit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
Q3 2026 forecast (close_date 2026-07-01 to 2026-09-30 inclusive)

Inside-quarter counts:
COMMIT: 7 deals
BEST_CASE: 24 deals
PIPELINE: 23 deals (counts 0)
Total inside: 54 deals

COMMIT total:
11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = 44,729
Deals: Deal-547B2B, Deal-B7EBD1, Deal-403845, Deal-A2B47C, Deal-2465CE, Deal-A5E80A, Deal-499BF6

BEST_CASE total: 203,565

Weighted forecast = 100% x COMMIT + 35% x BEST_CASE
= 44,729 + 0.35 x 203,565
= 44,729 + 71,247.75
= 115,976.75

Excluded for close_date outside the quarter:
Count: 32 deals
Total amount: 227,575

Top 5 BEST_CASE deals inside the quarter by amount:
1. Deal-2D7423 — 38,935 — 2026-09-30
2. Deal-25F752 — 24,000 — 2026-09-25
3. Deal-E53952 — 19,656 — 2026-09-30
4. Deal-5EED42 — 16,250 — 2026-09-30
5. Deal-FA32A0 — 11,116 — 2026-09-25

## Data quality
Owner is blank in 85 of 86 rows (only Deal-C9C286 has Bryce Harmon). Stage contradicts forecast_category in 4 rows: Deal-A5E80A (DS1 COMMIT), Deal-499BF6 (DS2 COMMIT), Deal-6787C2 (DS4 PIPELINE), and Deal-C61CF7 (DS5 BEST_CASE). why_buys_chars is 0 in 71 of 86 rows, and 4 still-open deals have close_dates before the 2026-09-05 extract (Deal-333EBB 2026-08-28, Deal-57FF13 2026-09-02, Deal-31AD2C and Deal-7A2454 2026-09-04), plus Deal-42326B has a fractional amount 2480.4.
aha-moment-2x2PASS 1.00
Plain-English question: Do companies that hit 5+ givers and one redemption in month one actually retain better at 24 months?
Prompt — what the model saw
Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.1168 · 67s · in 52,177 / out 11,399 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

Plain-English answer: The four cohort cells are graded against computed 24-month retention; the numbers come from the scorer's cohort table, not opinion.
The answer in one line: both_5plus_givers_and_redemption = 66.0% (31 of 47)
both_5plus_givers_and_redemption66.0% (31 of 47)
exclusionsnone — claim bounded as correlation, not causation
givers_only46.9% (23 of 49)
largest_single_signal_liftunique givers
neither40.0% (38 of 95)
redemption_only31.0% (9 of 29)
graded checksboth_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
2x2 on 220 companies, retained = current_status = 'active':

Both signals (m1_users >= 5 AND m1_redemptions >= 1): 47 companies, 31 active. 31/47 = 66.0%
Givers-only (m1_users >= 5, m1_redemptions = 0): 49 companies, 23 active. 23/49 = 46.9%
Redemption-only (m1_users < 5, m1_redemptions >= 1): 29 companies, 9 active. 9/29 = 31.0%
Neither (m1_users < 5, m1_redemptions = 0): 95 companies, 38 active. 38/95 = 40.0%

Check: 47 + 49 + 29 + 95 = 220. 31 + 23 + 9 + 38 = 101 active. Matches file total.

Excluded from denominator: 0. No companies excluded. All 220 rows have m1_users, m1_redemptions, and current_status; cancelled and non_renewing (3 rows) counted as not retained.

Single signal with largest retention lift: 5+ givers.

Arithmetic:
Givers marginal: (31+23)/(47+49) = 54/96 = 56.3% vs non-givers (9+38)/(29+95) = 47/124 = 37.9%. Lift = +18.3pp.
Redemption marginal: (31+9)/(47+29) = 40/76 = 52.6% vs non-redemption (23+38)/(49+95) = 61/144 = 42.4%. Lift = +10.3pp.
Vs neither baseline: givers-only 46.9% - 40.0% = +6.9pp; redemption-only 31.0% - 40.0% = -9.0pp; both 66.0% - 40.0% = +26.0pp.

This proves: in this extract, both-signals companies retained better at 24 months than either single-signal group or neither.

This does not prove: causation. It does not prove 5+ givers or 1+ redemption causes retention, does not control for confounders (size, tier, industry, country, integrations), and does not generalize beyond this cohort. Redemption alone is associated with lower retention than neither (31.0% vs 40.0%).
arr-rollforward-reconciliationPASS 1.00
Plain-English question: Does CRM ARR match billing ARR — and if not, exactly where is the variance?
Prompt — what the model saw
Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0418 · 361s · in 15,711 / out 13,534 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

Plain-English answer: Billing $604,739.28 vs CRM $603,581.76: a $1,157.52 variance decomposed into status-mismatch, rounding, and missing-record buckets.
The answer in one line: n_companies = 40; n_mismatched_accounts = 7; as_of = 2026-09-05 (+4 more below)
n_companies40
n_mismatched_accounts7
as_of2026-09-05
billing_arr_total604739.28
bucket_sum_equals_variance1
buckets{"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
crm_arr_total603581.76
ruleterm_months != 12 requires cf_agreement_end_date
variance1157.52
variance_definitionbilling_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
violationssubscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
graded checksbilling_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
All arithmetic verified by script (Decimal-exact). Here is the reconciliation.

CONVENTIONS
- Billing ARR = MRR x 12, active subscriptions only (cancelled subs are not ARR as of 2026-09-05).
- Variance = CRM total - Billing total. Positive = CRM higher.

TOTALS
- CRM (HubSpot) total: $603,581.76 across 39 company records
- Billing (Chargebee) total: $604,739.28 across 37 active subscriptions
- Variance (CRM - Billing): -$1,157.52 (CRM is $1,157.52 under billing)
- Footnote: if cancelled subs were included, billing total = $617,897.76 (39 subs); the status-mismatch bucket below is exactly the $13,158.48 of cancelled ARR still carried in CRM.

VARIANCE DECOMPOSITION (sums exactly to -$1,157.52)
- Status mismatch: +$13,158.48
- Rounding: +$36.00
- Missing records: -$11,952.00
- Other: -$2,400.00
- Check: 13,158.48 + 36.00 - 11,952.00 - 2,400.00 = -1,157.52 ✓

MISMATCHED ACCOUNTS (7 of 39; the other 32 active accounts tie exactly, CRM = MRR x 12 to the penny)
1. C-0C8323BF — status mismatch. SUB-000E cancelled in billing; CRM still carries $4,905.24. Diff +$4,905.24. Suggested owner: RevOps/CRM admin (zero HubSpot ARR or confirm reactivation).
2. C-0DC4FB8C — status mismatch. SUB-000F cancelled in billing; CRM still carries $8,253.24. Diff +$8,253.24. Suggested owner: RevOps/CRM admin (same fix).
3. C-0D66DF9E — rounding. Billing 1,932.00 x 12 = $23,184.00; CRM $23,200.00 = billing rounded to nearest $100. Diff +$16.00. Suggested owner: RevOps (sync HubSpot ARR to exact billing figure).
4. C-14D70CE0 — rounding. Billing 1,515.00 x 12 = $18,180.00; CRM $18,200.00 = nearest $100. Diff +$20.00. Suggested owner: RevOps (same fix).
5. C-0F7269D7 — other. Billing 2,233.00 x 12 = $26,796.00; CRM $24,396.00 implies MRR $2,033.00. Diff -$2,400.00 (CRM under by $200/mo). Does not fit rounding (nearest $100 of $26,796.00 is $26,800.00). Suggested owner: RevOps with Billing admin (verify which MRR is contracted).
6. C-0D5BBE3A — missing record (CRM only). HubSpot ARR $16,497.24; no Chargebee subscription exists. Diff +$16,497.24. Suggested owner: RevOps/Billing admin (confirm whether a subscription is missing in Chargebee or the ARR is non-subscription revenue).
7. C-21629AA4 — missing record (billing only). Active SUB-0004, 2,370.77 x 12 = $28,449.24; no HubSpot company record. Diff -$28,449.24. Suggested owner: RevOps/CRM admin (create/link the company record).

No owner names exist in either file, so owners above are suggested roles, not named individuals.

BUCKET ARITHMETIC
- Status mismatch: 4,905.24 + 8,253.24 = +13,158.48
- Rounding: 16.00 + 20.00 = +36.00
- Missing records: +16,497.24 (C-0D5BBE3A) - 28,449.24 (C-21629AA4) = -11,952.00
- Other: -2,400.00 (C-0F7269D7)

BUSINESS RULE: term other than 12 months must have cf_agreement_end_date populated
Violations (2):
- SUB-0002 / C-1794A52C — term 24 months, cf_agreement_end_date blank, status active
- SUB-0019 / C-22170CA1 — term 36 months, cf_agreement_end_date blank, status active
Compliant non-12-month subs (for reference): SUB-000C / C-0DB48281 (24 mo, end 2027-11-30) and SUB-001A / C-0FC4DBB8 (36 mo, end 2027-11-30).

DATA GAPS
- No owner/rep fields in either file, so "suggested owner" is role-based.
- No subscription start dates or cancellation dates are provided, so "as of 2026-09-05" status is taken from the status field as given; the two cancelled subs are treated as not-ARR.
kvm-month-over-monthPASS 1.00
Plain-English question: How did the four core KVMs move in August versus July?
Prompt — what the model saw
Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0167 · 78s · in 8,555 / out 3,823 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

Plain-English answer: Giving rate 60.27% (up 0.07%), redemptions per user 1.7302 (up 0.01%), with 1:1 meetings and pulse engagement each up fractionally — direction reported for all four.
The answer in one line: n_company_rows_per_month = 30; aggregation = simple mean of company-level values per month
n_company_rows_per_month30
aggregationsimple mean of company-level values per month
biggest_moverpulse_engagement
driving_segmentsize_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
kvms{"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
segment_detail{"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
graded checksfour_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Method: simple average across the 30 company rows per month (10 smb, 10 mid_market, 10 enterprise; all plan_tier = tier_three). Arithmetic: e.g. pulse Aug = mean of the 30 enterprise+smb+mid_market 2026-08 pulse values = 0.5086; Jul = 0.6006; change = 0.5086 − 0.6006 = −0.0920; relative = −0.0920 / 0.6006 = −15.31%.

| KVM | 2026-07 | 2026-08 | Abs change | Rel change | Direction |
|---|---|---|---|---|---|
| Giving rate | 0.6023 | 0.6027 | +0.0004 | +0.07% | Up (flat) |
| Redemptions per user | 1.7300 | 1.7302 | +0.0002 | +0.01% | Up (flat) |
| 1:1 meetings engagement | 0.4469 | 0.4472 | +0.0003 | +0.06% | Up (flat) |
| Pulse check engagement | 0.6006 | 0.5086 | −0.0920 | −15.31% | Down |

Largest relative move: pulse check engagement, −15.31%. The data supports the driver: the enterprise size_band fell from 0.5500 to 0.2743 (−0.2757, −50.13%), while smb (−0.22%) and mid_market (+0.21%) were flat. All 10 enterprise aliases (C-0B2895EF, C-0B2213A9, C-0D6CC8E3, C-0D0B047C, C-0D3278C7, C-0FCCD2DF, C-0F6C0F34, C-8C2E8F00, C-0B827671, C-0BA71F12) show the same ~50% drop, so it is segment-wide, not one company. plan_tier cannot be assessed — every row is tier_three. Note: the drop is uniform across all enterprise companies, which is consistent with a measurement/definition change; the data does not say why.
rewards-ytd-sectionPASS 1.00
Plain-English question: Produce the redemption section of the weekly rewards report through August.
Prompt — what the model saw
Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0383 · 315s · in 36,169 / out 2,690 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

Plain-English answer: 378 redemptions, $27,846 spend, 235 unique redeemers at 1.61 each, custom rewards at 39% of spend, US leading all countries.
The answer in one line: redemption_count = 378; unique_redeemers = 235; redemptions_per_redeemer = 1.61; total_spend_usd = 27846.00
redemption_count378
redemptions_per_redeemer1.61
unique_redeemers235
provider_mix_pct_of_spendTangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_countcountry=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
total_spend_usd27846.00
graded checkscount · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
All 378 rows fall within 2026-01-01 to 2026-08-31, so "through the last completed month" includes every row.

**Last completed month: 2026-08 (August 2026).** The data runs 2026-01-01 through 2026-08-31; there is no September data, so August is the last completed month.

## Redemptions (YTD through 2026-08)

Source: `redemptions_ytd.csv` (378 redemption rows).

- **Redemption count:** 378
- **Spend:** $27,812.00 (sum of `amount_usd` over all 378 rows)
- **Unique redeemers:** 235 (distinct `user_key`)
- **Redemptions per redeemer:** 378 / 235 = **1.61**

**Provider mix (% of spend)** — shares sum to 100%:

| Provider | Spend | % of spend |
|---|---|---|
| custom | $10,873.00 | 39.1% |
| Tremendous | $8,505.00 | 30.6% |
| Snappy | $5,204.00 | 18.7% |
| TangoCard | $3,230.00 | 11.6% |
| **Total** | **$27,812.00** | **100.0%** |

Arithmetic: custom 10873/27812 = 39.09%; Tremendous 8505/27812 = 30.58%; Snappy 5204/27812 = 18.71%; TangoCard 3230/27812 = 11.61%. Rounded: 39.1 + 30.6 + 18.7 + 11.6 = 100.0%.

**Top 5 countries by redemptions (count):**

| Rank | Country | Redemptions |
|---|---|---|
| 1 | US | 244 |
| 2 | CA | 24 |
| 3 | AU | 21 |
| 4 | GB | 17 |
| 5 | NL | 17 |

Note: GB and NL are tied at 17 redemptions each; both hold top-5 slots (next is SG at 12).
customer-success 5 tests
churn-save-eligibilityPASS 1.00
Plain-English question: Which at-risk accounts actually qualify for a churn-save offer, and which play fits each?
Prompt — what the model saw
Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0243 · 155s · in 8,774 / out 7,389 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

Plain-English answer: Only accounts meeting all three rules qualify — e.g. C-0F6C0F34 at $49,707 (executive touch) and C-0B827671 at $25,365 (usage revival) — with amount and play per account.
The answer in one line: n_accounts = 30; snapshot_date = 2026-09-05
n_accounts30
eligibleaccount_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
non_eligible_at_riskaccount_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
ruleshealth_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
snapshot_date2026-09-05
total_amount_at_stake224601.00
graded checkseligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Eligibility = R1 AND R2 AND R3. Snapshot 2026-09-05 + 120 days = cutoff 2027-01-03 (renewal must be on/before this). 15 of the 30 rows are at risk (health < 60); the other 15 have health >= 60 and are not at risk under R1, so they are out of scope.

QUALIFY FOR CHURN-SAVE (8 accounts) — amount at stake = churn_save_eligible_amount

Play rule used, signal-cited per account:
- Usage revival = adoption is the problem (declining trend, or flat with severe under-utilization)
- Executive touch = champion/relationship gap (champion_active = false) when usage is not the issue
- Commercial concession = no adoption or relationship gap (usage growing/healthy AND champion active) -> lever is commercial

  C-0F6C0F34   $49,707.00   Executive touch        champion_active=false (usage_trend=growing, so not adoption); renewal 2026-10-03
  C-0B827671   $25,365.00   Usage revival          usage_trend_3m=declining; seats_used/seats = 113/202 = 55.9%
  C-0B360C78   $35,748.00   Commercial concession  usage_trend=growing AND champion_active=true (no adoption/relationship gap); eligible = 59.2% of ARR
  C-0B0F1BAB   $ 5,494.00   Executive touch        champion_active=false; health 38 (lowest), renewal 2026-09-23 (18 days out)
  C-0CA21961   $16,829.00   Usage revival          seats_used/seats = 84/325 = 25.8% (severe under-use) with flat trend
  C-0E9C27D1   $41,235.00   Commercial concession  seats_used/seats = 134/157 = 85.4% (healthy) AND champion_active=true; renewal 2026-09-24
  C-0CEF69FD   $32,621.00   Executive touch        champion_active=false (usage_trend=growing)
  C-0D3278C7   $17,602.00   Usage revival          usage_trend_3m=declining; seats_used/seats = 126/380 = 33.2%

TOTAL AT STAKE
  49,707 + 25,365 + 35,748 + 5,494 + 16,829 + 41,235 + 32,621 + 17,602 = $224,601.00

AT RISK BUT DO NOT QUALIFY (7 accounts) — health < 60 (R1 pass) but fail R2 and/or R3
  C-0BC71BDD   health 55   fails R2: churn_save_eligible_amount = 0.00 (renewal 2026-10-28 is in window)
  C-0BE96399   health 54   fails R2: churn_save_eligible_amount = 0.00 (renewal 2026-10-29 is in window)
  C-10A56B0F   health 54   fails R2: churn_save_eligible_amount = 0.00 (renewal 2026-12-12 is in window)
  C-0BA71F12   health 52   fails R3: renewal 2027-04-11 is past the 2027-01-03 cutoff (eligible amount $6,824 > 0)
  C-0F876796   health 47   fails R3: renewal 2027-02-06 is past the 2027-01-03 cutoff (eligible amount $19,958 > 0)
  C-0F6694C3   health 43   fails R2 (eligible = 0.00) and R3 (renewal 2027-03-21 past cutoff)
  C-0FCCD2DF   health 43   fails R2 (eligible = 0.00) and R3 (renewal 2027-04-23 past cutoff)

Note: the seven non-qualifiers are still at risk by health; five of them (C-0BC71BDD, C-0BE96399, C-10A56B0F, C-0F6694C3, C-0FCCD2DF) fail only because churn_save_eligible_amount = 0, i.e. no documented churn-save dollars are attached to them under these rules.
expansion-kitPASS 1.00
Plain-English question: Build the expansion kit for the customer who just replied to our outreach.
Prompt — what the model saw
A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0090 · 53s · in 2,683 / out 2,050 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

Plain-English answer: 37.5% seat coverage, 250 headroom seats worth $15,000 ARR at $60 a seat, usage up 43.2% in six months — and the replier cannot buy; Dana R. (VP People) is the right buyer.
The answer in one line: headroom_seats = 250; coverage_pct = 37.5; company_alias = C-0DDFC9A7 (+3 more below)
headroom_seats250
coverage_pct37.5
company_aliasC-0DDFC9A7
headroom_arr15000.00
per_seat_annual_rate60.00
replierMaria S.
replier_can_buy0
right_buyername=Dana R.; title=VP People
usage_factsactive_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84
usage_health_linesMonthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checkscoverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
Expansion kit — C-0DDFC9A7 (all figures from the three files provided)

SEAT COVERAGE
150 licensed / 400 headcount = 37.5% covered. 250 of 400 employees are outside the license.

USAGE HEALTH (two lines)
MAU climbed every month, 88 (Mar) to 126 (Aug): +38 users, +43.2%, with steady +7 to +8 adds per month and no dips. August utilization is 126/150 = 84% of licensed seats — the account is growing into its license, which is the expansion signal.

HEADROOM
Per-seat rate: $9,000.00 ARR / 150 seats = $60.00 per seat (assumes current_arr is annual; the file does not state billing period).
Seat headroom to full headcount: 400 - 150 = 250 seats.
ARR headroom at current rate: 250 x $60.00 = $15,000.00. Full-coverage ARR would be 400 x $60.00 = $24,000.00 (2.67x current).

WHO REPLIED / CAN THEY BUY
Maria S., People Operations Coordinator, replied 2026-09-02. She cannot buy — she states budget and seat expansion sit with Dana R., VP People, and offers to make an introduction.

RIGHT BUYER
Dana R., VP People (last engaged 2026-05-18). She is the budget and seat-expansion owner per Maria's reply, and per the reply she has been asking about usage numbers — the kit's usage trend answers her open question. Sam K., Office Manager, is not a buyer for this decision.

REPLY EMAIL (99 words)

Subject: RE: Growing your team's recognition program

Hi Maria,

Thanks for the reply, and glad to hear the team is getting value from Bonusly.

One number worth sharing: monthly active users grew from 88 in March to 126 in August, now 84% of your 150 licensed seats. With 400 people on the team, there is room to extend recognition to more of them.

Happy to take you up on the introduction to Dana R. When you connect us, I will keep it short: current coverage, the usage trend, and what expansion would look like at your existing per-seat rate. No pressure on timing.

Thanks again,
Cole

Missing data, stated explicitly: no contract renewal date, no Dana R. contact details beyond name/title, and no statement of whether $9,000 is annual or monthly ARR — the $60/seat rate and all headroom figures scale accordingly if it is monthly.
onboarding-statusPASS 1.00
Plain-English question: What is done and what is missing in this customer's onboarding before the mid-onboarding call?
Prompt — what the model saw
Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0052 · 41s · in 491 / out 1,641 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

Plain-English answer: 25 days live: Slack connected, allowance set, admins added — but the HRIS integration is not connected, with early engagement signals and three call priorities.
The answer in one line: days_live = 25; company_alias = C-0D284E42
days_live25
call_itemsConnect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
checklist{"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
company_aliasC-0D284E42
engagement_signalsavg_active_givers_week1=4.3; avg_active_givers_last_week=13; latest_active_givers=15; days_with_activity=25
graded checkscomplete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
CSM prep — C-0D284E42 (mid-onboarding call)

COMPLETE (each backed by a data field)
- Slack integration connected — integration_slack = 2026-08-12 (1 day after signup 2026-08-11)
- Allowance set — allowance_set = 2026-08-13
- Admins added — admins_added = 2
- First recognition given — first_recognition_at = 2026-08-15 14:22 (4 days after signup)

NOT COMPLETE (field is blank — no data)
- HRIS integration — integration_hris is empty
- First redemption — first_redemption_at is empty

EARLY ENGAGEMENT SIGNALS (active_givers, 2026-08-11 to 2026-09-04, 25 days)
- Day 1: 3 givers. Latest (2026-09-04): 15 givers. Change: 15 − 3 = +12, i.e. 12/3 = +400% (5x) over 24 elapsed days.
- Sum of daily givers = 214 across 25 days; average = 214/25 = 8.56 givers/day.
- First week (08-11 to 08-17) avg = 30/7 = 4.29. Last week (08-29 to 09-04) avg = 91/7 = 13.0. That's 13.0/4.29 = ~3.0x week-over-week at the tail.
- Peak = 15 (09-03 and 09-04). Trough = 3 (08-11, 08-12). Day-to-day dips (08-16, 08-18, 08-22, 08-27, 08-31) are single-day; no sustained decline.
- Signal: giving is adopted and growing; the loop is open only on the redemption side.

THREE THINGS TO COVER ON THE CALL
1. HRIS integration is not connected (integration_hris blank). Slack went in on 08-12; get HRIS scheduled with an owner and date on the call.
2. No first redemption yet (first_redemption_at blank) despite first recognition on 08-15 and givers up 5x. Walk the redemption path and rewards catalog live; treat this as the main unlock.
3. Lock in the giving habit while it's compounding: givers grew 3 → 15 (5x) in 24 days. Confirm what's driving it and set a cadence goal for the next 2 weeks.

Data limits: usage file contains only active_givers. No fields for recipients, logins, redemption attempts, or reward views — don't claim anything about those.
renewal-risk-conflicting-datesPASS 1.00
Plain-English question: Two systems disagree on renewal dates — which do you trust per account, and who is actually renewing inside 90 days?
Prompt — what the model saw
Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0555 · 664s · in 40,860 / out 12,577 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

Plain-English answer: Chargebee wins per account (multi-year contracts are known wrong in ChurnZero), with each trusted date, the reason, and a 90-day-window flag.
The answer in one line: n_accounts = 20; n_disagreements = 5; snapshot_date = 2026-09-05 (+2 more below)
n_accounts20
n_disagreements5
accounts20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
arr_at_risk_90d359409.00
disagreementsaccount_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
snapshot_date2026-09-05
total_arr_renewing_90d1048715.00
trust_rulemulti-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
window2026-09-05 to 2026-12-04
graded checkstotal_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-DAY RENEWAL RISK BRIEF

Window: usage data ends 2026-08, so the vantage point is early September 2026. All 20 renewals fall between 2026-09-15 and 2026-11-24 (a 71-day span), so all 20 are in the 90-day cohort.

Date source rule: Chargebee holds contract terms and is the billing system of record. ChurnZero multi-year renewal dates are known to be wrong, so for the 5 multi-year accounts (is_multi_year=true) I use the Chargebee date. For the 15 twelve-month accounts the two systems agree exactly, so the choice is moot there.

Risk rules (stated up front):
- HIGH: seat utilization < 50%, OR active users down > 30% over 12 months.
- MEDIUM: utilization 50-75%.
- LOW: utilization > 75%.
Seat utilization = seats_used / seats. 3-month trend = avg(Jun-Aug 2026) vs avg(Mar-May 2026). 12-month trend = Aug 2026 vs Sep 2025.

RENEWALS (sorted by date used)

1. C-0B7D2C30 | Dana Mercer | $65,901 | 2026-09-15 (Chargebee) | util 274/476 = 57.6% | 3m trend -18.2%, 12m -45.8% | HIGH — usage has fallen every month for 12 straight months (155 to 84 active users, -45.8%). DISAGREEMENT: ChurnZero says 2026-09-10; Chargebee says 2026-09-15. Used Chargebee (36-month multi-year; CZ multi-year dates known wrong).

2. C-0BCDB8C2 | Cole Ingram | $54,427 | 2026-09-18 (Chargebee) | util 232/424 = 54.7% | 3m -17.6%, 12m -45.0% | HIGH — active users down 45% over 12 months (200 to 110) with no plateau. DISAGREEMENT: ChurnZero says 2027-09-18, a full year late; Chargebee says 2026-09-18. Used Chargebee (36-month multi-year).

3. C-0D2AB865 | Elena Sinclair | $38,022 | 2026-09-22 (Chargebee) | util 250/407 = 61.4% | 3m -18.9%, 12m -45.2% | HIGH — steady 12-month decline (199 to 109, -45.2%). DISAGREEMENT: ChurnZero says 2026-09-10; Chargebee says 2026-09-22. Used Chargebee (24-month multi-year).

4. C-0BBE3E60 | Dana Mercer | $30,993 | 2026-09-26 (Chargebee) | util 74/114 = 64.9% | 3m -19.5%, 12m -47.6% | HIGH — worst 12-month decline in the book (63 to 33, -47.6%). DISAGREEMENT: ChurnZero says 2027-09-26, a full year late; Chargebee says 2026-09-26. Used Chargebee (24-month multi-year).

5. C-0F5D2323 | Cole Ingram | $90,647 | 2026-09-29 (Chargebee) | util 111/390 = 28.5% | 3m +3.5%, 12m -14.3% | HIGH — largest account in the cohort and only 28.5% of seats in use (111 of 390); active users flat at ~17-21, so the product is barely adopted. DISAGREEMENT: ChurnZero says 2026-09-10; Chargebee says 2026-09-29. Used Chargebee (24-month multi-year).

6. C-0EC6999D | Elena Sinclair | $79,419 | 2026-10-03 (both agree) | util 31/112 = 27.7% | 3m +6.7%, 12m 0.0% | HIGH — second-largest account, only 27.7% seat utilization (31 of 112); usage flat at ~14-17 active users, well under half of seats used. No disagreement.

7. C-0B20DB64 | Dana Mercer | $21,770 | 2026-10-07 (both agree) | util 214/378 = 56.6% | 3m +0.1%, 12m +0.3% | MEDIUM — utilization 56.6% with perfectly flat usage (~293-298) for 12 months; stable but over-provisioned. No disagreement.

8. C-0BBC4E7A | Cole Ingram | $56,374 | 2026-10-10 (both agree) | util 228/337 = 67.7% | 3m -0.9%, 12m -2.1% | MEDIUM — utilization 67.7%, usage flat (~139-142); healthy floor but a third of seats idle. No disagreement.

9. C-0FD551AB | Elena Sinclair | $48,815 | 2026-10-14 (both agree) | util 210/376 = 55.9% | 3m -1.6%, 12m +1.6% | MEDIUM — utilization 55.9%, usage flat (~122-127); stable but nearly half of seats unused. No disagreement.

10. C-0F9F8F13 | Dana Mercer | $46,230 | 2026-10-18 (both agree) | util 199/352 = 56.5% | 3m +0.2%, 12m 0.0% | MEDIUM — utilization 56.5%, usage flat (~181-185); no growth signal. No disagreement.

11. C-0BC34584 | Cole Ingram | $16,740 | 2026-10-22 (both agree) | util 327/494 = 66.2% | 3m +1.0%, 12m +2.9% | MEDIUM — utilization 66.2%, usage flat (~101-106); smallest account, one-third of seats idle. No disagreement.

12. C-0B7A7546 | Elena Sinclair | $35,062 | 2026-10-25 (both agree) | util 182/205 = 88.8% | 3m +4.3%, 12m +8.6% | LOW — highest utilization in the book (88.8%) and usage growing (58 to 63). No disagreement.

13. C-0B369871 | Dana Mercer | $85,128 | 2026-10-29 (both agree) | util 317/422 = 75.1% | 3m +4.3%, 12m +15.2% | LOW — utilization 75.1% and usage up 15.2% over 12 months (289 to 333). No disagreement.

14. C-0B144C78 | Cole Ingram | $30,899 | 2026-11-02 (both agree) | util 169/224 = 75.4% | 3m +3.0%, 12m +17.8% | LOW — utilization 75.4%, usage up 17.8% (90 to 106). No disagreement.

15. C-0FC4DBB8 | Elena Sinclair | $94,732 | 2026-11-05 (both agree) | util 356/464 = 76.7% | 3m +4.2%, 12m +14.9% | LOW — largest account overall, utilization 76.7%, usage up 14.9% (168 to 193). No disagreement.

16. C-0D5BBE3A | Dana Mercer | $39,740 | 2026-11-09 (both agree) | util 85/102 = 83.3% | 3m +3.9%, 12m +19.7% | LOW — utilization 83.3%, fastest 12-month growth in the book (+19.7%, 76 to 91). No disagreement.

17. C-0FB9D5AF | Cole Ingram | $63,158 | 2026-11-13 (both agree) | util 144/199 = 72.4% | 3m +4.6%, 12m +14.3% | MEDIUM — usage growing (+14.3%) but utilization still 72.4%, just under the LOW threshold. No disagreement.

18. C-0B344485 | Elena Sinclair | $64,384 | 2026-11-16 (both agree) | util 224/287 = 78.0% | 3m +4.2%, 12m +15.6% | LOW — utilization 78.0%, usage up 15.6% (211 to 244). No disagreement.

19. C-0CB2C1B4 | Dana Mercer | $40,628 | 2026-11-20 (both agree) | util 386/473 = 81.6% | 3m -2.0%, 12m +14.0% | LOW — utilization 81.6% and usage up 14.0% over 12 months; note active users (~49) run far below seats used (386), worth a CSM check on what "used" means here. No disagreement.

20. C-22170CA1 | Cole Ingram | $45,646 | 2026-11-24 (both agree) | util 251/294 = 85.4% | 3m +2.6%, 12m +12.3% | LOW — utilization 85.4%, usage up 12.3% (130 to 146). No disagreement.

DISAGREEMENT FLAGS (5 of 20 accounts — all multi-year, all resolved to Chargebee)

- C-0B7D2C30: CZ 2026-09-10 vs CB 2026-09-15 (5 days early in CZ). Used CB.
- C-0BCDB8C2: CZ 2027-09-18 vs CB 2026-09-18 (CZ a full year late — if CZ were trusted this $54,427 renewal would be missed entirely). Used CB.
- C-0D2AB865: CZ 2026-09-10 vs CB 2026-09-22 (12 days early in CZ). Used CB.
- C-0BBE3E60: CZ 2027-09-26 vs CB 2026-09-26 (CZ a full year late). Used CB.
- C-0F5D2323: CZ 2026-09-10 vs CB 2026-09-29 (19 days early in CZ). Used CB.

The other 15 accounts (all 12-month terms) have identical dates in both systems — no disagreement. Pattern: ChurnZero multi-year dates are wrong in both directions (two a year late, three up to 19 days early), consistent with the known multi-year defect. Chargebee is trusted for all multi-year accounts.

Arithmetic check:
- Total ARR renewing = 65,901 + 54,427 + 38,022 + 30,993 + 90,647 + 79,419 + 21,770 + 56,374 + 48,815 + 46,230 + 16,740 + 35,062 + 85,128 + 30,899 + 94,732 + 39,740 + 63,158 + 64,384 + 40,628 + 45,646 = $1,048,715.00
- HIGH (6 accounts: C-0B7D2C30, C-0BCDB8C2, C-0D2AB865, C-0BBE3E60, C-0F5D2323, C-0EC6999D) = 65,901 + 54,427 + 38,022 + 30,993 + 90,647 + 79,419 = $359,409.00
- MEDIUM (6 accounts: C-0B20DB64, C-0BBC4E7A, C-0FD551AB, C-0F9F8F13, C-0BC34584, C-0FB9D5AF) = 21,770 + 56,374 + 48,815 + 46,230 + 16,740 + 63,158 = $253,087.00
- LOW (8 accounts) = 35,062 + 85,128 + 30,899 + 94,732 + 39,740 + 64,384 + 40,628 + 45,646 = $436,219.00
- Cross-check: 359,409 + 253,087 + 436,219 = $1,048,715.00 ✓

CLOSE:
- Total ARR renewing in the 90-day window: $1,048,715.00 (20 accounts)
- ARR at risk (HIGH + MEDIUM): $612,496.00 = 58.4% of renewing ARR
- Of which HIGH (active usage collapse or sub-30% utilization): $359,409.00 = 34.3% of renewing ARR

Caveat: no health scores, support tickets, or engagement data were provided, so risk is rated on seat utilization and usage trend only. The two full-year ChurnZero errors (C-0BCDB8C2, C-0BBE3E60) mean any renewal list built off ChurnZero alone would have dropped $85,420 of renewals from this window.
ticket-theme-synthesisPASS 1.00
Plain-English question: What are the real themes in this quarter's support tickets, ignoring the existing tags?
Prompt — what the model saw
Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0465 · 304s · in 17,930 / out 14,923 tok
✓ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids fabricated: C-0C2E8F00

✓ Correct Answer (ground truth)

Plain-English answer: 80 tickets in 4-7 themes — points-not-posting leads (20 tickets, 25%, $31,100 ARR affected) — each with counts, distinct accounts, ARR, and a recommendation.
The answer in one line: total_tickets = 80; quarter_window = 2026-06-01 through 2026-08-31
total_tickets80
noteexisting tags are noisy/unreliable; classify by text
quarter_window2026-06-01 through 2026-08-31
ranking_by_arr_exposurehris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
sample_ticket_ids80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
single_account_noise_themebilling_dispute
themes{"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
graded checkstheme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Verified by script: 80 tickets, 24 distinct accounts, $284,800 combined ARR. Every ticket mapped to exactly one of 5 themes (no conflicts, no unmatched), and the 24 accounts partition cleanly across themes (3+7+1+9+4 = 24), so ARR shares sum to 100%.

Ranked by ARR exposure (not volume):

1. HRIS provisioning / new-hire sync failures — BROAD (3 large accounts)
- Count: 12 (15.0% of tickets)
- Distinct accounts: 3 — C-0B2213A9 ($36,000, 7 tickets), C-0DDFC9A7 ($48,000, 3), C-0F6C0F34 ($30,000, 2)
- ARR affected: 36,000 + 48,000 + 30,000 = $114,000 (40.0% of total ARR)
- Ticket IDs: IC-460059, IC-460060
- Recommendation: Provisioning is silently skipping new hires (log shows no errors) at the three largest accounts — fix the HRIS sync job, add failure alerting, and contact C-0DDFC9A7 / C-0B2213A9 / C-0F6C0F34 before renewal.

2. Redemption / gift-card checkout failures — BROAD (7 accounts)
- Count: 18 (22.5%)
- Distinct accounts: 7 — C-0B0F1BAB ($10,300), C-0B827671 ($10,700), C-0CEF69FD ($8,900), C-0D9CA315 ($9,600), C-0F876796 ($8,700), C-0FCCD2DF ($9,600), C-14264ABD ($11,000)
- ARR affected: 10,300+10,700+8,900+9,600+8,700+9,600+11,000 = $68,800 (24.2%)
- Ticket IDs: IC-460025, IC-460030
- Recommendation: Redemption fails at checkout and in some cases points are deducted with no gift card issued — make the redemption transaction idempotent with a points hold, and re-issue/refund across the 7 accounts.

3. Billing / invoicing errors — SINGLE-ACCOUNT (concentrated, not broad)
- Count: 16 (20.0%) — all from one account
- Distinct accounts: 1 — C-0E9C27D1 ($52,000, all 16 tickets)
- ARR affected: $52,000 (18.3%)
- Ticket IDs: IC-460071, IC-460069
- Recommendation: Single-account noise by breadth, but material: C-0E9C27D1 is on its third straight seat-count error plus wrong-tier renewal pricing — escalate to billing ops for a corrected invoice and a root-cause fix on seat-count sourcing.

4. Points not posting to balances — BROAD (9 accounts, all small)
- Count: 20 (25.0% — highest volume)
- Distinct accounts: 9 — C-0B2895EF ($2,900), C-0BE96399 ($2,700), C-0BF20542 ($4,500), C-0D0B047C ($4,500), C-0D284E42 ($3,400), C-0D3278C7 ($3,500), C-0D6CC8E3 ($4,200), C-0DD0626C ($2,500), C-21FEBCBB ($2,900)
- ARR affected: 2,900+2,700+4,500+4,500+3,400+3,500+4,200+2,500+2,900 = $31,100 (10.9%)
- Ticket IDs: IC-460004, IC-460016
- Recommendation: Highest ticket volume but lowest ARR per account — recognitions show delivered yet balances never update; fix the points-posting pipeline and proactively credit the 9 affected balances.

5. Slack integration failures — NARROW (4 small accounts)
- Count: 14 (17.5%)
- Distinct accounts: 4 — C-0B843542 ($4,400), C-0BA71F12 ($3,900), C-10A56B0F ($5,400), C-0C2E8F00→C-8C2E8F00 ($5,200)
- ARR affected: 4,400+3,900+5,400+5,200 = $18,900 (6.6%)
- Ticket IDs: IC-460041, IC-460047
- Recommendation: Sync stops, the toggle resets, re-auth doesn't stick, and the slash command errors team-wide — fix token persistence and slash-command error handling across the 4 accounts.

Cross-checks: tickets 12+18+16+20+14 = 80 ✓; accounts 3+7+1+9+4 = 24 ✓; ARR 114,000+68,800+52,000+31,100+18,900 = $284,800 ✓.

Broad vs. noise: themes 1, 2, 4, 5 are multi-account patterns. Theme 3 (billing) is single-account noise by breadth — all 16 tickets from C-0E9C27D1 — but it still ranks #3 on ARR because that one account is $52,000 and the errors are recurring. Note the inverse case at the bottom: points-not-posting is the highest-volume theme (25% of tickets) but only 10.9% of ARR, because it sits entirely in small accounts.
marketing 5 tests
case-study-matchPASS 1.00
Plain-English question: Which three existing customers are the best case-study matches for this prospect?
Prompt — what the model saw
Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0057 · 28s · in 755 / out 1,815 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

Plain-English answer: Three ranked matches on industry, size, use case, and region (e.g. C-11C31562), with the higher-scoring decoy explicitly excluded for a mismatched field.
The answer in one line: prospect_alias = C-82AF3719
constraintsno_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
decoys_excludedcompany_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
prospect_aliasC-82AF3719
prospect_fields_usedindustry, size_band, use_case, region
top3company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
graded checkstop3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Prospect: C-82AF3719 — Technology, Mid-Market, employee_recognition, NA-West

Scoring: 1 point per matching field (industry, size_band, use_case, region), max 4. Only the 8 customers with has_case_study=true are eligible.

Scorecard (all case-study customers):
  C-64171065   Technology / Mid-Market / employee_recognition / NA-East   = 3/4 (misses region)
  C-A13C193D   Technology / Mid-Market / retention / NA-West            = 3/4 (misses use_case)
  C-11C31562   Manufacturing / Mid-Market / employee_recognition / NA-West = 3/4 (misses industry)
  C-9D69D147   Financial Services / Mid-Market / retention / NA-West   = 2/4
  C-CD4829A7   Technology / Mid-Market / retention / NA-East           = 2/4
  C-FEFCC020   Education / Enterprise / wellness / NA-East              = 0/4
  C-11209339   Manufacturing / SMB / peer_milestones / APAC             = 0/4
  C-0CE315B2   Education / SMB / wellness / EMEA                       = 0/4

Three-way tie at 3/4 broken by field weight in the order given (industry > size_band > use_case > region): each candidate misses exactly one field, so the one missing the lowest-weighted field ranks highest.

Ranked social-proof matches:

1. C-64171065 — Technology, Mid-Market, employee_recognition, NA-East
   Matches: industry, size_band, use_case. Differs only on region (NA-East vs NA-West).

2. C-A13C193D — Technology, Mid-Market, retention, NA-West
   Matches: industry, size_band, region. Differs on use_case (retention vs employee_recognition).

3. C-11C31562 — Manufacturing, Mid-Market, employee_recognition, NA-West
   Matches: size_band, use_case, region. Differs on industry (Manufacturing vs Technology).

All three have has_case_study=true. No other case-study customer scores above 2/4, so no additional names are included.
channel-attribution-mathPASS 1.00
Plain-English question: Compute cost per SQM, cost per SQO, and pipeline per dollar for every channel over six months.
Prompt — what the model saw
Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0403 · 249s · in 30,011 / out 7,907 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

Plain-English answer: LinkedIn Ads at $960/SQM and $3,000/SQO, organic search at $90,000 pipeline on zero spend, paid social flagged at $18,000 with zero SQMs, and a channel winner named.
The answer in one line: best_pipeline_per_dollar_channel = paid_search
best_pipeline_per_dollar_channelpaid_search
confidence_notewebinars and referral have small SQM samples (<20); treat rates as directional.
flagged_rows_sqo_before_sqmCT-000041, CT-000044
months2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel{"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
reallocation_recommendationfrom=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
undefined_channelspaid_social
graded checksundefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Channel performance, trailing 6 months (2026-03 through 2026-08). All spend rows and all 122 contact rows fall inside this window, so nothing is excluded.

PAID CHANNELS

paid_search
- Spend: 6 months x $6,000 = $36,000
- SQMs: 40 | SQOs: 18
- Cost per SQM: 36,000 / 40 = $900.00
- Cost per SQO: 36,000 / 18 = $2,000.00
- SQM-to-SQO rate: 18 / 40 = 45.0%
- Pipeline: 18 SQOs x $40,000 = $720,000
- Pipeline per dollar: 720,000 / 36,000 = $20.00

linkedin_ads
- Spend: 6 x $4,000 = $24,000
- SQMs: 25 | SQOs: 8
- Cost per SQM: 24,000 / 25 = $960.00
- Cost per SQO: 24,000 / 8 = $3,000.00
- SQM-to-SQO rate: 8 / 25 = 32.0%
- Pipeline: 8 x $12,000 = $96,000
- Pipeline per dollar: 96,000 / 24,000 = $4.00

paid_social
- Spend: 6 x $3,000 = $18,000
- SQMs: 0 | SQOs: 0
- Cost per SQM: undefined (spend with zero SQMs)
- Cost per SQO: undefined (spend with zero SQMs)
- SQM-to-SQO rate: undefined (zero SQMs)
- Pipeline: $0
- Pipeline per dollar: 0 / 18,000 = $0.00

webinars
- Spend: 6 x $1,500 = $9,000
- SQMs: 12 | SQOs: 5
- Cost per SQM: 9,000 / 12 = $750.00
- Cost per SQO: 9,000 / 5 = $1,800.00
- SQM-to-SQO rate: 5 / 12 = 41.7%
- Pipeline: 5 x $12,000 = $60,000
- Pipeline per dollar: 60,000 / 9,000 = $6.67

Paid total
- Spend: 36,000 + 24,000 + 18,000 + 9,000 = $87,000
- SQMs: 40 + 25 + 0 + 12 = 77 | SQOs: 18 + 8 + 0 + 5 = 31
- Cost per SQM: 87,000 / 77 = $1,129.87
- Cost per SQO: 87,000 / 31 = $2,806.45
- SQM-to-SQO rate: 31 / 77 = 40.3%
- Pipeline: 720,000 + 96,000 + 0 + 60,000 = $876,000
- Pipeline per dollar: 876,000 / 87,000 = $10.07

ORGANIC / NON-PAID CHANNELS (no spend data provided, so cost metrics cannot be computed — data missing)

organic_search
- Volume (SQMs): 30 | SQOs: 10
- SQO rate: 10 / 30 = 33.3%
- Pipeline: 10 x $9,000 = $90,000

referral
- Volume (SQMs): 15 | SQOs: 6
- SQO rate: 6 / 15 = 40.0%
- Pipeline: 6 x $8,000 = $48,000

Combined non-paid: 45 SQMs, 16 SQOs, SQO rate 16 / 45 = 35.6%, pipeline $138,000.

DATE-INTEGRITY FLAGS (SQO date precedes SQM date)
- CT-000044 | linkedin_ads | SQM 2026-07-23 | SQO 2026-07-18 | pipeline $12,000
- CT-000041 | linkedin_ads | SQM 2026-06-14 | SQO 2026-06-09 | pipeline $12,000

These two rows are included in the linkedin_ads figures above as reported. Excluding them, linkedin_ads would be: 6 SQOs, $72,000 pipeline, cost per SQO 24,000 / 6 = $4,000.00, pipeline per dollar 72,000 / 24,000 = $3.00, SQM-to-SQO rate 6 / 25 = 24.0% — worse than reported.

REALLOCATION RECOMMENDATION

1. Cut paid_social entirely. $18,000 spent over 6 months produced zero attributed SQMs. Its efficiency metrics are undefined, not zero, and there is no outcome data supporting continued spend. Reallocate that $18,000.
2. Split the freed $18,000 toward webinars and paid_search. Webinars has the best unit economics of any channel with outcomes (cost per SQM $750, cost per SQO $1,800). paid_search has the best pipeline yield ($20.00 pipeline per dollar, $40,000 pipeline per SQO). A rough split: ~$10,000 to webinars (scaling a $9,000 channel more than 2x may be aggressive; treat as a test with a checkpoint) and ~$8,000 to paid_search.
3. Hold or restructure linkedin_ads rather than cut outright, pending a data fix. Its reported economics are the weakest among channels with outcomes (cost per SQO $3,000, pipeline per dollar $4.00), and 2 of its 8 SQOs ($24,000 of $96,000 pipeline) have impossible dates, so even those weak numbers are unreliable. Fix the date attribution first, re-measure for one quarter, then decide.

CONFIDENCE: Moderate on cutting paid_social — zero attributed outcomes against $18,000 of spend for 6 straight months is a strong signal, though it could reflect an attribution/tracking gap rather than true zero performance, so verify tracking before finalizing. Low-to-moderate on the split between webinars and paid_search — samples are small (webinars: 12 SQMs / 5 SQOs; paid_search: 40 SQMs / 18 SQOs; linkedin_ads: 25 / 8), and pipeline is open pipeline, not closed revenue, so pipeline per dollar overstates realized return. Treat the reallocation as a one-quarter test with a checkpoint, not a permanent shift.
competitor-battlecard-updatePASS 1.00
Plain-English question: Update the Rivally battlecard with sourced pricing and where each side wins.
Prompt — what the model saw
Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0412 · 296s · in 30,501 / out 9,086 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

Plain-English answer: Rivally at $7 per user per month (the August source wins over the stale $5, conflict noted), a 13-7 win/loss record, and two unverified old claims flagged.
The answer in one line: competitor = Rivally
citation_ids_available25 items — e.g. S01, S02, S03, S04
competitorRivally
pricingvalue_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
rep_opinion_snippet_ids_not_factsS09, S21
unverified_old_claimsRivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
win_loss_12mowins=13; losses=7
graded checkscompetitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
Updated battlecard: Rivally. Built only from competitor_snippets.csv and deals_with_competitor.csv. Every factual claim carries a snippet id. Rep opinions (S09, S21) are excluded as facts and flagged. Old-card lines that cannot be re-sourced are marked.

ONE-LINE POSITIONING
Points-based employee recognition platform with an engaging recognition feed, sold on fast setup and Slack/Teams integrations, expanding in the EU with data residency and multi-language support, plus a paid engagement-survey add-on (Rivally Pulse). [S02, S16, S04, S19, S12, S15, S06/S23]

PRICING (source + date; newer source wins; conflict noted)
- Current list: $7 per user/month, annual billing required, Recognition Starter tier — Rivally pricing page, 2026-08-12 [S17]. This is the newest pricing-page source and wins.
- Conflict: earlier pricing pages showed $5 per user/month, annual billing — 2026-01-20 [S03] and still $5 on 2026-04-01 [S08]. List rose $5 -> $7 = +$2 = +40% [arithmetic on S03/S08 vs S17].
- Deal-level quotes (deal-specific, not list): $6.50/user/mo quoted to a 500-seat prospect, annual term, 2026-06-02 [S13] — $0.50 below the later $7 list [arithmetic S13 vs S17]; and $7/user/mo list with a 15% discount for a 3-year term, reported by a prospect 2026-08-14 [S18] — effective $7.00 x 0.85 = $5.95/user/mo [arithmetic on S18].
- Note: S21 (a rep's belief that Rivally discounts aggressively) is rep opinion, not confirmed — not used as a pricing fact. No Bonusly price is provided in the data, so no price comparison is made.

WHERE THEY WIN
- EU / distributed EU teams: strong for distributed EU teams, multi-language support praised [S12]; EU data residency GA and Dublin office opened 2026-07-01 [S15]; hired ex-Workday VP EMEA to lead European expansion 2026-05-09 [S11].
- Fast rollout: mid-market setup took under a week [S04].
- Integrations: Slack integration worked out of the box [S04]; Microsoft Teams app v2 in public preview 2026-08-20 [S19]. (This contradicts the old card's "lacks a Slack integration" — see unverified/contradicted below.)
- Product feel and support: recognition feed praised as engaging [S02, S16]; support response time under 4 hours praised [S22].

WHERE WE WIN
- Analytics depth — the only directly sourced win reason: an 800-seat prospect picked Bonusly over Rivally citing analytics depth, 2026-09-03 [S25]. Rivally-side gaps that back this: limited analytics [S02]; reporting dashboards basic vs enterprise tools [S07]; analytics exports are CSV-only, which made migration off Rivally hard [S20].
- Enterprise admin gaps at Rivally: lacks SCIM provisioning, manual user management painful [S10]; admin console lacks bulk recognition editing [S24]; admin tooling lags peers [S16].
- Rewards catalog: EMEA catalog thinner than US catalog [S14].
- Caveat: no snippet confirms Bonusly's own SCIM/analytics/catalog features, so these are framed as sourced Rivally gaps plus the one sourced win reason (S25), not as verified Bonusly capabilities.

OBJECTIONS AND RESPONSES (responses use only sourced facts)
- Objection: "Rivally is cheaper." Response: their list price rose from $5 to $7/user/mo (+40%) between 2026-04-01 and 2026-08-12 [S08 -> S17]; discounting is deal-specific (15% off a 3-year term = $5.95 effective [S18]; a $6.50 quote to one 500-seat prospect [S13]). No Bonusly price is in the data, so do not claim we are cheaper.
- Objection: "Rivally is stronger for EU / we need EU data residency." Response: acknowledge — EU data residency GA, Dublin office [S15], multi-language praised [S12]. No snippet sources a Bonusly EU counter; do not improvise one — verify before positioning.
- Objection: "Rivally is more engaging and easier to roll out." Response: acknowledge — engaging feed [S02, S16], sub-week setup [S04], Slack out of the box [S04], support under 4 hours [S22]. Counter with analytics and admin gaps [S07, S10, S24] and switching friction: CSV-only analytics exports made leaving Rivally hard [S20].
- Objection: "Rivally's analytics are good enough." Response: limited analytics [S02], basic dashboards [S07], CSV-only exports [S20]; an 800-seat prospect chose Bonusly over Rivally on analytics depth [S25].

RECENT CHANGES (chronological)
- 2025-11-04: Series C of $40M led by Northgate Ventures [S01].
- 2026-03-05: launched Rivally Pulse, a lightweight engagement-survey add-on [S06]; 2026-09-01: Pulse exits beta, priced as an add-on, not bundled [S23].
- 2026-05-09: hired ex-Workday VP EMEA to lead European expansion [S11].
- 2026-07-01: opened Dublin office; EU data residency generally available [S15].
- 2026-08-12: Recognition Starter list price raised $5 -> $7 per user/month [S17]; conflicts with $5 on 2026-01-20 [S03] and 2026-04-01 [S08].
- 2026-08-20: Microsoft Teams app v2 announced in public preview [S19].

12-MONTH WIN/LOSS RECORD VS RIVALLY (deals_with_competitor.csv, 2025-09 through 2026-08 = 12 distinct months, 20 deals, all vs Rivally)
- Wins: 13. Losses: 7. Total: 13 + 7 = 20.
- Win rate: 13/20 = 65.0%. Loss rate: 7/20 = 35.0%.
- By month (W/L): 2025-09 1/1, 2025-10 2/0, 2025-11 1/1, 2025-12 1/1, 2026-01 2/0, 2026-02 2/0, 2026-03 1/1, 2026-04 0/2, 2026-05 0/1, 2026-06 1/0, 2026-07 1/0, 2026-08 1/0.
- Worst stretch: 2026-04 (0W/2L) and 2026-05 (0W/1L). Best: 2025-10, 2026-01, 2026-02 (2W/0L each).
- Deal aliases — wins: Deal-A9FD43, Deal-7AA785, Deal-44C524, Deal-0D0CD6, Deal-D5B790, Deal-5C636E, Deal-67BE14, Deal-1B6969, Deal-F03E7B, Deal-072E31, Deal-F65C8F, Deal-E46EAB, Deal-1D2392. Losses: Deal-7767F5, Deal-5645A5, Deal-C6FFAA, Deal-D263E0, Deal-935746, Deal-9066A6, Deal-72A02F.
- Outside the deal file: S25 (2026-09-03) is a further Bonusly win over Rivally in call notes, but it is not in deals_with_competitor.csv and falls outside the 12-month window, so it is excluded from the 13/20 count.

OLD-CARD CLAIMS: VERIFIED / CONTRADICTED / UNVERIFIED
- "Points-based recognition for mid-market" — re-sourced: points-based recognition feed [S02, S16]; mid-market supported by a mid-market reviewer's sub-week setup report [S04].
- "Pricing: starts at $5 per user/month, annual billing (as of 2026-01)" — OUTDATED: superseded by $7 list on 2026-08-12 [S17]; conflict with $5 noted above [S03, S08].
- "Rivally lacks a Slack integration" — CONTRADICTED: Slack integration worked out of the box [S04]. Removed.
- "Rivally was acquired by WorkHuman in 2025" — UNVERIFIED: no snippet sources an acquisition. S01 reports a Series C raise led by Northgate Ventures (2025-11-04), which is the only 2025 corporate event sourced; it does not confirm or describe an acquisition. Do not use until sourced.
- "Strong in EU enterprise with multi-language support" — re-sourced [S12], reinforced by EU data residency GA and Dublin office [S15] and the ex-Workday VP EMEA hire [S11].

EXCLUDED AS NON-FACTS (rep opinion, per your rule)
- S09 (2026-04-10, Bryce Harmon): "Rivally's UI is clunky" — AE opinion, not verified against the product. Not used.
- S21 (2026-08-28, Elena Sinclair): "Rivally is discounting aggressively" — AE opinion, not confirmed. Not used.

Data gaps (stated explicitly): no Bonusly pricing, no Bonusly feature confirmations (SCIM/analytics/catalog), no sourced reason for the 7 losses, and no source for the WorkHuman acquisition claim.
nurture-sequence-diagnosisPASS 1.00
Plain-English question: What is broken in each email sequence — rates, overlaps, tracking errors?
Prompt — what the model saw
Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0969 · 55s · in 65,491 / out 3,544 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

Plain-English answer: Per-sequence open, reply, and meeting rates with the weakest step (all three weakest at step 3), an opened-above-sent tracking error, and audience overlap flagged.
The answer in one line: fix_priority_first = Cold Outbound - HR Leaders
all_pairwise_overlaps['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
audience_overlap_pairCold Outbound - HR Leaders, Cold Outbound - People Ops
failure_modeCold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_firstCold Outbound - HR Leaders
per_sequence{"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_errorsequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
under_2pct_replyCold Outbound - HR Leaders
graded checkstracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
Per-sequence totals [sent = sum steps; rate = total / total sent]:

New Logo Nurture: sent 500+458+428=1386. Open 210+160+120=490; 490/1386=35.4%. Reply 42+30+18=90; 90/1386=6.5%. Meetings 12+9+6=27; 27/1386=1.9%. Weakest: Step 3, 120/428=28.0% open, 18/428=4.2% reply.

Expansion Nurture: sent 300+300+275=875. Open 130+340+95=565; 565/875=64.6%. Reply 22+25+12=59; 59/875=6.7%. Meetings 5+4+3=12; 12/875=1.4%. Weakest: Step 3, 95/275=34.5% open, 12/275=4.4% reply.

Cold Outbound - HR Leaders: sent 600+595+590=1785. Open 240+175+130=545; 545/1785=30.5%. Reply 5+2+1=8; 8/1785=0.45%. Meetings 0/1785=0%. Weakest: Step 3, 1/590=0.17% reply.

Cold Outbound - People Ops: sent 400+386+377=1163. Open 150+110+80=340; 340/1163=29.2%. Reply 14+9+6=29; 29/1163=2.5%. Meetings 3+2+1=6; 6/1163=0.5%. Weakest: Step 3, 80/377=21.2% open, 6/377=1.6% reply.

Tracking error: Expansion Nurture Step 2 opened 340 > sent 300. Impossible; fix open logging/dedup before reporting.

Audience overlap: contact_key appears in >1 sequence. New Logo Nurture x Expansion Nurture: CT-000301, CT-000624. Cold Outbound - HR Leaders x Cold Outbound - People Ops: CT-000849, CT-000884, CT-001033, CT-001101, CT-001103, CT-001153, CT-001159, CT-001217, CT-001227, CT-001255, CT-001277, CT-001285, CT-001345. No other cross-sequence overlap observed.

Under 2% reply failure mode: Cold Outbound - HR Leaders 0.45% overall; Steps 1: 5/600=0.83%, 2: 2/595=0.34%, 3: 0.17%. Cold Outbound - People Ops Step 3: 1.59%. Mode: open holds ~30% but reply collapses and HR Leaders yields 0 meetings = message/offer mismatch, not deliverability.

Fix: HR Leaders first. One change: pause Steps 2-3, rewrite Step 1 to one HR-Leader pain + proof + low-friction CTA, suppress People Ops overlap. People Ops: one change: replace Step 3 with plain-text breakup question and stop after no reply.
weekly-marketing-goals-rollupPASS 1.00
Plain-English question: Are we on pace on the quarter's marketing goals?
Prompt — what the model saw
Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0107 · 97s · in 3,107 / out 2,735 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

Plain-English answer: SQMs ahead of pace, SQOs on pace, DS2s behind (35 short of expected at day 66 of 92), with deltas and same-quarter closes.
The answer in one line: days_in_quarter = 92; days_elapsed = 66; coverage_ratio_active_pipeline = 0.75; elapsed_fraction = 0.7174 (+1 more below)
days_in_quarter92
coverage_ratio_active_pipeline0.75
days_elapsed66
elapsed_fraction0.7174
metrics{"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
pace_ruleexpected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required1
graded checksmetrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 weekly marketing goals update. Quarter: 2026-07-01 to 2026-09-30, 92 days, 66 elapsed (66/92 = 71.7% of quarter), 26 days remaining. Pace benchmark = target x 66/92.

Metric table (QTD actual vs target):

SQMs: 230 vs 300. Delta = 230 - 300 = -70. Pace benchmark = 300 x 66/92 = 215.2. Actual 230 is +14.8 over pace (76.7% of target at 71.7% elapsed). Pace: AHEAD.

SQOs: 84 vs 120. Delta = 84 - 120 = -36. Pace benchmark = 120 x 66/92 = 86.1. Actual 84 is -2.1 under pace (70.0% of target). Pace: BEHIND (slightly).

DS2s: 40 vs 75. Delta = 40 - 75 = -35. Pace benchmark = 75 x 66/92 = 53.8. Actual 40 is -13.8 under pace (53.3% of target). Pace: BEHIND.

Closed-lost MIA rate: 5 MIA / 25 closed-lost = 0.20 = 20.0% vs target 10% (lower is better). Delta = 20.0% - 10.0% = +10.0 pp (worse). Pace: BEHIND — already at 2x the ceiling with 26 days left.

Same-quarter close count: 10 vs 20. Delta = 10 - 20 = -10. Pace benchmark = 20 x 66/92 = 14.3. Actual 10 is -4.3 under pace (50.0% of target). Pace: BEHIND.

Active pipeline coverage: $3,000,000 vs $4,000,000 target. Delta = 3,000,000 - 4,000,000 = -$1,000,000. Coverage = 3,000,000 / 4,000,000 = 75.0%. Pace benchmark = 4,000,000 x 66/92 = $2,869,565. Actual is +$130,435 over pace. Pace: AHEAD.

What moved this week: no prior-week snapshot was provided in the data, so week-over-week movement cannot be computed and I won't invent it. What the QTD data does show: the quarter is 71.7% elapsed and the funnel is split — volume and pipeline are holding ahead of pace (SQMs +14.8 over benchmark, active pipeline +$130K over benchmark at 75% coverage), while the middle and bottom of funnel are behind (SQOs -2.1, DS2s -13.8, same-quarter closes -4.3 vs benchmark) and closed-lost MIA rate is running at 20%, double the 10% ceiling. The gap to close in the final 26 days: 70 SQMs, 36 SQOs, 35 DS2s, 10 same-quarter closes, $1.0M pipeline, and MIA rate needs to come down, not just hold.

Missing data noted: prior-week QTD values (for WoW movement) and a breakdown of the 25 closed-lost deals beyond the 5 MIA (for MIA-rate context).
communication 5 tests
ceo-slack-compression0.60
Plain-English question: Compress the forecast analysis into one Slack message the CEO can read in ten seconds.
Prompt — what the model saw
Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0225 · 116s · in 21,113 / out 3,899 tok
✗ max_3_sentences✗ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

Plain-English answer: Three sentences, under 80 words, decision first, every distinct number kept plus the one caveat — all checked by deterministic text rules.
The answer in one line: weighted_number = 115976; must_keep = data-quality caveat — deleting the bad news fails
weighted_number115,976
limitmax 3 sentences, max 80 words, no em-dashes
must_keepdata-quality caveat — deleting the bad news fails
graded checksmax_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Q3 weighted forecast is $115,976.75: $44,729 COMMIT (7 deals) plus 35% of $203,565 BEST_CASE (24 deals), with 23 PIPELINE deals at zero. It looks precise but is not: owners are blank on 85 of 86 deals, why-buys is empty on all 7 in-quarter COMMIT deals, and 32 deals totaling $227,575 slipped to October close dates, several still DS2/DS3 weeks before quarter end. Treat $115,976.75 as a ceiling, not a commitment, until close dates are re-qualified.

(3 sentences, 75 words. Arithmetic shown: 44,729 + 0.35 x 203,565 = 44,729 + 71,247.75 = 115,976.75. Omitted under the word cap: 54-of-86 in-quarter scope, excluded Deal-D348E1 at 13,770, the 9 BEST_CASE at 28,240 / 22 PIPELINE split of the excluded 227,575, and the top-5 BEST_CASE list; the aggregates carry the caveat.)
stalled-deal-followup-emailPASS 1.00
Plain-English question: Write a follow-up email for this stalled deal — under 120 words, one prior touch, one ask.
Prompt — what the model saw
Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0058 · 46s · in 483 / out 1,940 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

Plain-English answer: An email that references the July 28 milestone-awards demo, makes one clear ask, and invents nothing — Deal-0D2F7A, 31 days since last engagement.
The answer in one line: amount = 5100; word_cap = 120; days_since_last_engagement = 31; deal_alias = Deal-0D2F7A (+4 more below)
amount5,100
days_since_last_engagement31
word_cap120
allowed_facts14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_newENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
deal_aliasDeal-0D2F7A
must_referenceengagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
ownerAlex Franklin
single_cta1
stageDS3
tonewarm, concise, professional (matches prior thread)
graded checksword_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Deal-0D2F7A — following up on the 150-seat pricing

Hi,

Following up on my August 5 recap of the July 28 demo. The People team reacted strongly to the automated milestone awards and points catalog, and asked for pricing — I sent the 150-seat quote that same week.

One ask: can we get 15 minutes with the VP People this week to confirm the recognition budget and set a start date ahead of the November 26 close?

Happy to work around their calendar.

Alex Franklin

(76 words)
funnel-digest-paragraphPASS 1.00
Plain-English question: Write the weekly GTM digest — one paragraph per team, every number traceable.
Prompt — what the model saw
Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0388 · 259s · in 24,803 / out 5,143 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

Plain-English answer: Four paragraphs (Marketing, Sales, CS, PLG) using only the allowed numbers, no contact names, no performance commentary, under 400 words.
The answer in one line: word_cap = 400
word_cap400
allowed_numbers16 items — e.g. 2, 3, 9, 08
bannedcustomer contact names, employee performance commentary, compensation
required_paragraphsMarketing, Sales, CS, PLG
graded checksfour_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
**Marketing**

Marketing brought in 46 SQMs against a weekly target of 52, finishing 6 short (52 − 46 = 6), or 88.5% of target (46 ÷ 52 = 88.5%). The webinar channel delivered 18 of those SQMs, about 39% (18 ÷ 46 = 39.1%). Paid search cost per SQM held at $150. The 6-SQM gap is the number to close next week.

**Sales**

Sales converted 14 SQOs and set 9 DS2 meetings. New pipeline created totaled $310,000. Same-quarter closes for the week came in at 3. That averages to about $22,143 of new pipeline per converted SQO ($310,000 ÷ 14 = $22,142.86).

**CS**

Customer Success saved 2 renewals this week. Team NPS moved to 61. Three red-flag accounts are still open heading into next week. Those 3 accounts carry into next week.

**PLG**

PLG added 412 new signups with activation at 31%. That works out to about 128 activated signups (412 × 0.31 = 127.72). 38 companies reached the aha moment of 10 recognition gives. Hitting 10 gives is the point where new companies feel the product work.

(196 words total. Every figure traces to week_extracts.md; the only derived numbers are the 6-SQM gap, the two percentages, the per-SQO pipeline average, and the ~128 activated signups, each with its arithmetic shown. No company aliases appear in the extract, so none are cited.)
partner-digest-sectionPASS 1.00
Plain-English question: Write the partner digest so every partner in the roster appears exactly once.
Prompt — what the model saw
Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0033 · 32s · in 520 / out 730 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

Plain-English answer: Four active partners with status and numbers, four quiet partners with one line each, and partner-sourced pipeline that reconciles to the deal data.
The answer in one line: rule = every roster partner named exactly once; quiet partners get one line; no partners outside roster.
active_no_dealsWorkWell Group
partner_sourced_pipelinecount=5; amount=550000
per_partner_pipeline{"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
quiet_partnersRecogniq, KudosWave, PeopleFirst Advisors, TotalPerk
rosterpartner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
ruleevery roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checksall_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
# Partner Digest — 2026-08-24 to 2026-09-04

Partner-sourced pipeline: 5 deals, $550,000 total.

## Active partners

**Apex Rewards Co** — active
- Sourced 2 deals, $275,000: Deal-DDAAF2 ($180,000) + Deal-2CF33E ($95,000) = $275,000. 180,000 + 95,000 = 275,000.
- Both logged with UTM Source = Partner; both in DS1.
- Co-webinar with Apex locked for 09-15.

**HRCloud Partners** — active
- Sourced 1 deal, $140,000: Deal-F1CDA5 ($140,000).
- Integration referral closed the security review; opp moved to DS2.

**CultureBridge** — active
- Sourced 2 deals, $135,000: Deal-096E1D ($60,000) + Deal-067213 ($75,000) = $135,000. 60,000 + 75,000 = 135,000.
- Both early stage; produced via lunch-and-learn.

**WorkWell Group** — active
- No sourced deals this period ($0, 0 deals). Wants to restart the joint playbook in Q4; planning call booked for 09-09.

## Quiet partners

**Recogniq** — quiet. No activity reported this period.
**KudosWave** — quiet. No activity reported this period.
**PeopleFirst Advisors** — quiet. No activity reported this period.
**TotalPerk** — quiet. No activity reported this period.

## Reconciliation

Deal data vs. Slack: Apex 2/2 match, HRCloud 1/1 match, CultureBridge 2/2 match, WorkWell 0/0 match. Total: 2 + 1 + 2 + 0 = 5 deals; $275,000 + $140,000 + $135,000 + $0 = $550,000. No partner-sourced deals are missing from either source, and no deals in the CSV lack a roster partner.
rfp-answer-from-policyPASS 1.00
Plain-English question: Answer the 10 RFP items from the policy excerpts only — no guessing.
Prompt — what the model saw
Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0039 · 35s · in 524 / out 988 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

Plain-English answer: Seven answered with excerpt citations (Q1-Q7), three flagged unanswerable with the document that would answer them (Q8-Q10).
The answer in one line: answered_count = 7; rule = unanswerable items must be flagged and the document that would answer them named; do not guess.
answered_count7
answer_map{"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
ruleunanswerable items must be flagged and the document that would answer them named; do not guess.
unanswerableQ8, Q9, Q10
valid_citation_idsP1, P2, P3, P4, P5, P6, P7, P8
graded checksanswered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
**Vendor responses — Bonusly policy questionnaire (source: policy_excerpts.md only)**

Q1. Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2. Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3. Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. A copy is available to prospects under NDA. [P2]

Q4. Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. Arithmetic: retention = contract term + 90 days; deletion from production systems occurs at contract end + 90 days. [P3]

Q5. The current subprocessor list is published at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

Q6. Yes. SAML 2.0 single sign-on is supported, available on the Enterprise plan (SCIM user provisioning is also included on Enterprise). [P5]

Q7. Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8. Unanswerable from the provided excerpts. No excerpt addresses ISO 27001. The document that would answer this: Bonusly's ISO 27001 certificate (or the certifications page of the trust portal / trust.bonusly.com).

Q9. Unanswerable from the provided excerpts. No excerpt states an uptime SLA. The document that would answer this: the Master Subscription Agreement (MSA) or the service-level agreement (SLA) schedule attached to it.

Q10. Unanswerable from the provided excerpts. No excerpt addresses HIPAA or a Business Associate Agreement. The document that would answer this: the Data Processing Addendum (DPA), or Bonusly's legal/security team via the trust portal contact.

Summary: 7 of 10 answered from excerpts (Q1–Q7); 3 unanswerable (Q8, Q9, Q10) — each needs a document not provided (ISO 27001 certificate, MSA/SLA, DPA respectively).
ops-maintenance 5 tests
skill-file-reconciliationPASS 1.00
Plain-English question: Audit a skill set against its manifest and flag every conflict with a severity and an action.
Prompt — what the model saw
Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.0973 · 854s · in 66,132 / out 24,357 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

Plain-English answer: A severity-ranked list of TRIM_DESC / UPDATE_BODY / DELETE_SKILL / MERGE / REVIEW actions covering every rule violation found.
The answer in one line: files_audited = 14; rank = CRITICAL / WARNING / INFO with one fix action per issue
files_audited14
findingsduplicate trigger, circular delegation chain, dangling file reference, oversized description, hardcoded values, manifest drift
rankCRITICAL / WARNING / INFO with one fix action per issue
graded checksduplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Scope assumption: I treated the 14 manifest rows and the 14 provided SKILL.md files as the closed universe. Skills referenced but not provided are treated as not present for dangling-target checks.

Arithmetic summary:
- Manifest rows = 14. Provided files = 14. Matching stems = 14/14.
- Files with no manifest row = 14 - 14 = 0.
- Manifest rows with no file = 14 - 14 = 0.
- Manifest description_chars = 656, 897, 996, 792, 965, 676, 945, 1004, 1006, 962, 1006, 708, 762, 656. Max = 1006. Count greater than 1024 = 0, because 1006 <= 1024.

Findings:

1) CRITICAL - MERGE - Duplicate ALWAYS-trigger scope: comms-drafter and email-drafter. They share the same email-drafting triggers and coverage: "write me an email," "draft a follow-up," "help me reply" / "help me reply to this," "what should I say," "bump email," "contract nudge," plus outbound prospecting, follow-ups, post-demo recaps, pricing/contract follow-up, end-of-quarter pushes, renewal/expansion emails, QBR follow-ups, and onboarding check-ins. Proposal: merge email-drafter into comms-drafter; comms-drafter survives as the single external-communications drafting skill; move the Gmail signature extraction into comms-drafter; update deal-strategy-coach to point to comms-drafter instead of email-drafter.

2) WARNING - TRIM_DESC - Overlapping generic pipeline triggers: pipeline-intelligence-report and weekly-pipeline-report. Overlap includes "pipeline update" / "run the pipeline update" / "update the pipeline," "run the pipeline report" / "generate the pipeline report" / "do the pipeline report," and "what's the pipeline look like" / "what does pipeline look like." Proposal: trim descriptions so pipeline-intelligence-report requires scored/tiered/full-pipeline language and weekly-pipeline-report requires weekly SQM/SQO/DS2/bookings-MTD language; remove the generic pipeline-update/report/look-like phrases from at least one side.

3) WARNING - TRIM_DESC - Universal trigger collision: model-selection says ALWAYS run at the start of every task, without exception, before any planning, execution, or skill invocation. That collides with every domain ALWAYS trigger in closed-lost-analysis, partner-digest, pipeline-intelligence-report, sales-forecast, stale-pipeline-report, and weekly-pipeline-report. Proposal: scope model-selection to planning/model-annotation work or mark it as an orchestrator-level planning gate that is subordinate to domain skill triggers.

4) INFO - REVIEW - Final-gate trigger overlap is broad but sequenced: analysis-validator, signalforge-claim-compressor, and signalforge-feedback can all match the same SignalForge report/analysis output. signalforge-claim-compressor already says it runs after analysis-validator, and signalforge-feedback says it runs after both. Proposal: keep all three, but make the required order explicit in each description: analysis-validator -> signalforge-claim-compressor -> signalforge-feedback; do not merge.

5) CRITICAL - UPDATE_BODY - Circular delegation chain: deal-strategy-coach <-> email-drafter, with comms-drafter adjacent. deal-strategy-coach says to use the email-drafter skill when drafting manager-to-prospect emails; email-drafter says to point users to deal-strategy-coach for strategic deal coaching; comms-drafter also says deal-strategy-coach handles strategy while comms-drafter drafts. Proposal: make ownership one-way: deal-strategy-coach owns diagnosis/strategy; one drafting skill owns copy after the comms/email merge; drafting skills may recommend a separate strategy session but must not hand back into deal-strategy-coach mid-task.

6) CRITICAL - REVIEW - Dangling explicit delegation targets in the closed set: prospect-research-multithreading is invoked by comms-drafter, email-drafter, and deal-strategy-coach. analysis-validator section 12.4 says to delegate to bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, and bonusly-datadog-questions. None of these are in the 14-row manifest or the 14 provided files. Proposal: either add the missing skills to the manifest with files, or replace the explicit invoke/delegate wording with in-set steps; until then mark them external/unresolvable.

7) WARNING - REVIEW - External/reference-only skill names absent from the manifest: bonusly-brand is referenced by comms-drafter, email-drafter, sales-forecast, and signalforge-claim-compressor; skill-orchestrator is referenced by analysis-validator and signalforge-feedback; signalforge-reports org skill paths are referenced by pipeline-intelligence-report and weekly-pipeline-report. Proposal: label these as external dependencies or add them to the manifest; do not count them as successful in-set delegation.

8) WARNING - UPDATE_BODY - Internal version conflict: analysis-validator. The header and changelog say v3.6, May 9, 2026, but the section 7 trail template still says "Validator: analysis-validator v3.2," and sections 2 and 6 describe Gate 2 as G2-A through G2-E even though G2-F exists. Survivor: analysis-validator v3.6. Proposal: update the trail template and gate diagrams to v3.6 and G2-A through G2-F.

9) CRITICAL - UPDATE_BODY - Version/body conflict: sales-forecast. The v1.1 changelog and description say quarter-agnostic and Elena -> Alaina, but the body still contains "Open Q2 Deals," "Q2 Narrative," and a Q3 example title. Survivor: the v1.1 quarter-agnostic behavior. Proposal: remove Q2/Elena remnants and compute the active quarter everywhere.

10) WARNING - UPDATE_BODY - Cross-skill roster conflict: analysis-validator section 12.3 lists Core 6 AEs including Hugo Lindqvist, HubSpot owner ID 77260721. pipeline-intelligence-report Phase 1 lists only five AE owner IDs: Bryce Harmon 119337721, Dana Mercer 83155923, Cole Ingram 83155924, Alex Franklin 84342457, Gavin Porter 1520255671; Hugo Lindqvist is omitted. Survivor: dynamic owner resolution, as modeled by stale-pipeline-report, over either hardcoded list. Proposal: update pipeline-intelligence-report to resolve owners through HubSpot and include all current AEs; keep analysis-validator's roster as reference only.

11) INFO - TRIM_DESC, not triggered - Manifest descriptions over 1024 characters: 0. Arithmetic: max description_chars = 1006; 1006 <= 1024; count greater than 1024 = 0. No trim required.

12) CRITICAL - UPDATE_BODY - Hardcoded publish/log destinations and personal routing: partner-digest hardcodes Amani Phipps, Slack from:<@U03QLMBL7AR>, Confluence cloud ID 73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f, space ID 1958248479, folder ID 2286616609, and page IDs 2286321666, 2265382925, 2236940297, 2237825028, 2239365136, 2238283777. signalforge-feedback hardcodes page ID 2295136266, space ID 2232811524, parent ID 2234417154, and Build Log page ID 2247295002. sales-forecast hardcodes space ID 2232811524 and parent page ID 2232582148. weekly-pipeline-report hardcodes Ben Lavin and spreadsheet IDs 1CLZeOsElVDF_LF0ZG_t2nfwvhnZ6bpwqM_nX3WEYzcw and 1ENuaEcCuLjdKhMvp8FK3Ys1ek5Aw9ZuOZhsHJJFoB_k. stale-pipeline-report hardcodes Slack channel C0561C1JCPJ and HubSpot org ID 1973303. Proposal: resolve destinations by title/CQL/search at runtime, parameterize owner/recipient, and keep IDs only as verified fallbacks.

13) WARNING - UPDATE_BODY - Hardcoded dated reference data, rosters, and person names: analysis-validator hardcodes a May 4, 2026 roster with Alaina Loori, Shealagh Coughlin, Ben Castelli, Amani Phipps, John Thomas, Yasmin Wahid, Bryce Harmon, Hugo Lindqvist, Dana Mercer, Alex Franklin, Cole Ingram, Gavin Porter, Colleen Perry, Ellie Barton, Ashley Reyer, Megan Franz, Elena Sinclair, Youssef Elkhateeb, and Amanda Czenkus, plus owner IDs, Finance escalation names Manish/Amani, population anchors ~452,000 and ~110,097, and expected ranges as of May 2026. deal-strategy-coach hardcodes a 2026 pricing table and AE Excellence Playbook page ID 2257879045 dated April 2026. model-selection hardcodes last_checked 2026-05-19 and model knowledge cutoffs. partner-digest hardcodes example weeks May 19, 2026 and June 2, 2026 plus the May 16, 2026 canonical page. weekly-pipeline-report hardcodes Q2 April 1 - June 30, 2026 and Q1 2026 actuals: $365,152 vs $475,000 and $2,490,532 vs $3,288,000. stale-pipeline-report hardcodes example dates 5/15, 5/19, and 5/7 plus a hardcoded "97 deals" performance note. Proposal: move volatile rosters, anchors, IDs, and dated actuals to live lookups or clearly mark them as time-stamped examples; keep canonical SQL and thresholds.

14) INFO - REVIEW - Illustrative aliases/examples are hardcoded in narrative bodies: closed-lost-analysis cites Softheon, Estee Lauder, LIFTOFF, Nestlé, Ozinga, Aurora Innovation, GCash, Ethos Cannabis, StickerYou, and MinIO; signalforge-claim-compressor cites Panopto, Schneider Downs, and Felix Construction; signalforge-feedback's example cites Gavin Porter and Lowe's. Proposal: label these as illustrative, non-authoritative examples; do not treat them as live pipeline or customer facts.

15) INFO - REVIEW, standing check only - Manifest drift in both directions: 0. Arithmetic: files without manifest row = 0; manifest rows without file = 0. Proposal: keep generating the manifest from files and add a parity check for file/manifest drift and description length.
runbook-from-slack-threadPASS 1.00
Plain-English question: Turn the resolved incident thread into a runbook.
Prompt — what the model saw
Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0056 · 89s · in 697 / out 1,753 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

Plain-English answer: Ordered steps from the thread, each with the exact command, who ran it, how success was verified, and rollback where state changed.
The answer in one line: rule = no steps beyond what the thread contains; every step traces to a message id.
ambiguous_steporder=4; msg=M05; mark=needs-confirmation
ruleno steps beyond what the thread contains; every step traces to a message id.
stepsorder=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
graded checksordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
# Runbook: reward-worker queue depth incident (#incident-2026-08-29-reward-queue, resolved)

Source: incident_thread.md, messages M01–M10. Every step below traces to a specific message. Arithmetic shown where the thread contains numbers.

---

## Step 1 — Acknowledge alert, take Incident Commander
- **Action:** Acknowledge the PagerDuty alert for reward-worker queue depth > 10k; take IC role.
- **Who:** Bryce Harmon
- **Source:** M01 (2026-08-29 14:02:10Z)
- **Verification:** Not stated in thread — needs confirmation.
- **Rollback:** Not applicable (role assignment, no system state changed).

## Step 2 — Measure queue depth (diagnostic, read-only)
- **Command:** `bundle exec rake sidekiq:queue_depth`
- **Who:** Farid Osman
- **Result:** 48,213 pending jobs. Normal is under 500. Backlog multiple: 48,213 / 500 = 96.4x normal.
- **Source:** M02 (14:04:33Z)
- **Rollback:** Not applicable (read-only).

## Step 3 — Inspect dead set (diagnostic, read-only)
- **Action:** Check Sidekiq dead set.
- **Who:** Farid Osman
- **Result:** 112 jobs, all `Redis::TimeoutError` from around 13:58.
- **Source:** M03 (14:06:02Z)
- **Rollback:** Not applicable (read-only).

## Step 4 — Pause enqueue to stop the bleed (state-changing)
- **Command:** `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- **Who:** Farid Osman
- **Source:** M04 (14:08:45Z)
- **Verification:** Not stated at time of action; effectiveness confirmed indirectly by Step 7 (depth falling ~1,200/min after scale-up) — needs confirmation whether the flag disable was verified directly.
- **Rollback (stated in thread):** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`

## Step 5 — Clear the dead set (state-changing)
- **Action:** Cleared out the dead set "in the console."
- **Who:** Elena Sinclair
- **Source:** M05 (14:15:20Z)
- **Exact command:** NOT STATED — needs confirmation.
- **Verification:** Not stated — needs confirmation.
- **Rollback:** NOT STATED — needs confirmation. Dead-set jobs are typically not restorable once cleared; confirm whether any of the 112 `Redis::TimeoutError` jobs needed replay before clearing.

## Step 6 — Scale workers up (state-changing)
- **Command:** `kubectl scale deployment/reward-worker --replicas=6` (was 3 — a 2x increase)
- **Who:** Bryce Harmon
- **Source:** M06 (14:21:07Z)
- **Verification:** Indirect — Step 7 shows depth falling ~1,200/min. Direct verification of pod count not stated — needs confirmation.
- **Rollback (stated in thread):** `kubectl scale deployment/reward-worker --replicas=3`

## Step 7 — Monitor drain rate (read-only)
- **Action:** Observe queue depth trend.
- **Who:** Farid Osman
- **Result:** Depth down to 9,400 and falling ~1,200/min. Arithmetic check: from 48,213 (M02) to 9,400 is a drop of 38,813 jobs; at ~1,200/min that implies ~32 minutes of drain time, consistent with the M02 (14:04) to M07 (14:33) interval of ~29 minutes plus the pre-scale pause period.
- **Source:** M07 (14:33:41Z)
- **Rollback:** Not applicable (read-only).

## Step 8 — Verify queue drained and error rate recovered (read-only)
- **Command:** `bundle exec rake sidekiq:queue_depth` (returns 0); Datadog error rate checked.
- **Who:** Cole Ingram
- **Result:** Queue depth 0. Error rate in Datadog back to baseline.
- **Source:** M08 (14:47:55Z)
- **Rollback:** Not applicable (read-only).

## Step 9 — Re-enable enqueue (state-changing)
- **Command:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- **Who:** Bryce Harmon
- **Verification (stated in thread):** 40 new jobs processed cleanly in the next 3 minutes.
- **Source:** M09 (14:49:10Z)
- **Rollback:** Not explicitly stated for this step; the inverse is the Step 4 disable command from M04 — `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`. Marked as inferred from M04, needs confirmation.

## Step 10 — Scale workers back down (state-changing)
- **Command:** `kubectl scale deployment/reward-worker --replicas=3`
- **Who:** Bryce Harmon
- **Verification (stated in thread):** Queue stable at 0. Incident resolved.
- **Source:** M10 (14:55:00Z)
- **Rollback:** Not explicitly stated; the inverse is the Step 6 command from M06 — `kubectl scale deployment/reward-worker --replicas=6`. Marked as inferred from M06, needs confirmation.

---

## Open items needing confirmation (not stated in thread)
1. Step 1: how PagerDuty acknowledgment / IC handoff was verified.
2. Step 4: whether the feature-flag disable was verified directly at execution time.
3. Step 5: exact command/console action used to clear the dead set, whether the 112 `Redis::TimeoutError` jobs were replayed or lost, and whether a rollback path exists.
4. Step 6: whether the 6 replicas actually came up healthy (pod readiness not stated).
5. Steps 9–10: rollback commands are inferred from M04/M06, not stated at those steps.
6. Root cause: the thread documents response, not cause. The ~13:58 `Redis::TimeoutError` cluster (M03) is the only lead; no root-cause step exists in the thread.
datadog-incident-timelinePASS 1.00
Plain-English question: Sequence the failure from the logs — what broke first, and what cascaded after it?
Prompt — what the model saw
From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0140 · 263s · in 1,724 / out 5,452 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

Plain-English answer: A reward-service Redis timeout at 14:01:12Z cascades to sidekiq at 14:01:40 and api-gateway at 14:03:05, with the Datadog query that confirms the first error.
The answer in one line: job_class_involved = RewardGiveJob
cascade_orderservice=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
clean_servicespostgres
confirming_queryservice:reward-service status:error "@message:Redis::TimeoutError"
first_errortimestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
job_class_involvedRewardGiveJob
logs_do_not_showroot cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
graded checksfirst_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
FAILURE SEQUENCE — 2026-09-03 (all times UTC)

FIRST ERROR
2026-09-03T14:01:12Z — service: reward-service — ERROR — "Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s". Since the timeout is 5s, the failed connection attempt began ~14:01:07 (14:01:12 minus 5 seconds).

CASCADE IN ORDER (from datadog_logs.csv)
1. 14:01:12 — reward-service ERROR — Redis connect to redis-primary:6379 times out after 5s (first error)
2. 14:01:20 — reward-service ERROR — Redis::TimeoutError: retry exhausted for RewardGiveJob
3. 14:01:30 — reward-service ERROR — Redis::TimeoutError: retry exhausted for RewardGiveJob
4. 14:01:40 — reward-service ERROR — Redis::TimeoutError: retry exhausted for RewardGiveJob
5. 14:01:40 — sidekiq ERROR — RewardGiveJob failed: Redis::TimeoutError; retrying in 60s
6. 14:02:28 — sidekiq ERROR — RewardGiveJob failed: Redis::TimeoutError; retrying
7. 14:02:30 — sidekiq WARN — Queue reward depth above 10,000
8. 14:03:05 — api-gateway ERROR — 502 upstream timeout calling reward-service /gives
9. 14:03:30 — web-app ERROR — Give form submission failed: upstream 502 from api-gateway
10. 14:03:31 — sidekiq ERROR — RewardGiveJob failed: Redis::TimeoutError; retrying
11. 14:03:48 — api-gateway ERROR — 502 upstream timeout calling reward-service
12. 14:04:13 — api-gateway ERROR — 502 upstream timeout calling reward-service
13. 14:04:22 — sidekiq ERROR — RewardGiveJob failed: Redis::TimeoutError; retrying
14. 14:04:45 — web-app ERROR — Give form submission failed: upstream 502
15. 14:05:16 — api-gateway ERROR — 502 upstream timeout calling reward-service
16. 14:05:26 — sidekiq ERROR — RewardGiveJob failed: Redis::TimeoutError; retrying
17. 14:05:42 — web-app ERROR — Give form submission failed: upstream 502
18. 14:06:47 — sidekiq ERROR — RewardGiveJob failed: Redis::TimeoutError; retrying
19. 14:06:49 — web-app ERROR — Give form submission failed: upstream 502
20. 14:06:52 — api-gateway ERROR — 502 upstream timeout calling reward-service
Recovery:
21. 14:22:10 — reward-service INFO — Redis connection restored; resuming job processing
22. 14:24:45 — sidekiq INFO — Queue reward depth below 500

So the causal chain is: Redis connectivity loss at redis-primary:6379 → RewardGiveJob retries exhausted in reward-service → sidekiq job failures and retries → reward queue depth above 10,000 → reward-service cannot serve /gives → api-gateway 502s → web-app give form submission failures (user-facing).

SERVICE AND JOB INVOLVED
- Failing service: reward-service (Redis client to redis-primary:6379); failing worker: sidekiq.
- Job classes (from sidekiq_jobs.csv): RewardGiveJob — J-00001 through J-00012 = 12 jobs; RecognitionDigestJob — J-00013 through J-00016 = 4 jobs. All 16 failed with Redis::TimeoutError.
- Arithmetic: 12 + 4 = 16 failed jobs total. RewardGiveJob share = 12/16 = 75%. RecognitionDigestJob share = 4/16 = 25%.
- failed_at range in sidekiq_jobs.csv: earliest 14:01:46 (J-00005), latest 14:05:50 (J-00016); span = 4 min 4 s.
- Error-line counts in the log slice: reward-service 4, sidekiq 6, api-gateway 5, web-app 4 = 19 ERROR lines, plus 1 WARN. The log slice shows only 6 sidekiq ERROR lines against 16 failed jobs in sidekiq_jobs.csv, so the log slice is a sample, not an exhaustive record (16 - 6 = 10 failures not represented as sidekiq log lines, including all 4 RecognitionDigestJob failures).
- Duration arithmetic: first error 14:01:12 to Redis restore 14:22:10 = 20 min 58 s. First error to queue below 500 (14:24:45) = 23 min 33 s. Restore to queue below 500 = 2 min 35 s. User-facing 502 window: first 14:03:05 to last 14:06:52 = 3 min 47 s. First sidekiq failure (14:01:40) to queue-depth WARN (14:02:30) = 50 s.
- Note: jobs were enqueued before the first error (web-app "job enqueued" 13:58:49, reward-service "job enqueued" 13:59:30) and the last reward-service "health check ok" was 13:59:09, so Redis degraded sometime between 13:59:09 and 14:01:07.

DATADOG QUERY TO CONFIRM THE FIRST ERROR
service:reward-service status:error "Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out"
Time range 2026-09-03 13:55:00Z to 14:25:00Z, sorted by @timestamp ascending. The earliest hit should be 2026-09-03T14:01:12Z. A broader variant: service:reward-service status:error Redis::TimeoutError | sort @timestamp asc — earliest result confirms first occurrence.

WHAT THE LOGS DO NOT SHOW
1. Root cause of the Redis timeout. There are no redis-primary log lines at all, and no CPU/memory/network metrics for redis-primary:6379. Cannot tell whether Redis crashed, ran out of memory, hit a network partition, or was being deployed.
2. Whether the 16 failed jobs eventually succeeded after the 14:22:10 restore. sidekiq_jobs.csv records failures only; there are no per-job success or final-disposition records.
3. Whether any give was double-awarded or lost because retried jobs ran after recovery.
4. User impact counts. No user identifiers, no total number of affected give submissions; the web-app ERROR lines are samples.
5. Trace/correlation IDs linking web-app form submissions to specific job_ids, so individual user journeys cannot be reconstructed.
6. What happened between 14:06:52 and 14:22:10 — no log lines in that window; cannot distinguish a quiet period from missing data.
7. Redis health before the first error. Last "health check ok" was 13:59:09; degradation between 13:59:09 and 14:01:07 is invisible.
8. Queue depth over time — only two data points exist (above 10,000 at 14:02:30; below 500 at 14:24:45).
9. Whether anyone was alerted or paged; no alert/notification records.
10. Why jobs enqueued at 13:58:49 and 13:59:30 did not fail until 14:01:46 or later per sidekiq_jobs.csv — enqueue-to-failure latency is not explained by the logs.
11. The 4 RecognitionDigestJob failures appear in sidekiq_jobs.csv but have no corresponding lines in the log slice.
feature-flag-state-summaryPASS 1.00
Plain-English question: Summarize what each feature flag controls and who has it on.
Prompt — what the model saw
Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0047 · 65s · in 575 / out 1,395 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: Recognition streaks on for 42 beta companies, redeem-flow redesign off with a 12-company target list, plus any flag with no code reference or no targets.
The answer in one line: rule = company counts must equal the export; do not describe flags absent from the export.
code_referencedrecognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
flags{"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
flags_with_no_targetspaused_offboard_cleanup
orphan_flags_no_code_referencelegacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
rulecompany counts must equal the export; do not describe flags absent from the export.
graded checksorphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Flags in export: 9. Flags with code reference: 6. Flags with no code reference: 3 — legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup. Flag with no targets: paused_offboard_cleanup (targeting_rules blank, company_count 0). No company names are provided; only segment/rule labels and counts.

Per flag (state / targeting / count → what the code excerpt controls):

1. recognition_streaks_v2 — on / segment:beta_companies / 42
   Code: app/models/recognition.rb — when enabled for company, calls StreakTracker.record(give) on give.

2. points_budget_guardrails — on / all_companies / 220
   Code: app/services/budget_service.rb — when enabled for company, calls BudgetService.new(company).enforce!(giver, points).

3. slack_dm_nudges — on / segment:region_na / 87
   Code: app/jobs/nudge_job.rb — when enabled for company, job proceeds to SlackDm.send_nudge(user); otherwise returns early.

4. redeem_flow_redesign — off / targeted_list / 12
   Code: app/controllers/redeem_controller.rb — when enabled for company, renders RedeemV2Component.new(company); else renders RedeemV1Component.new(company). Flag is off, so the else branch (RedeemV1Component) is the live path for targeted companies unless the flag is turned on.

5. analytics_dashboard_v3 — on / segment:tier_three / 65
   Code: app/controllers/analytics_controller.rb — when enabled for company, sets @dashboard = AnalyticsV3.new(company).

6. ms_teams_app_v2 — off / targeted_list / 9
   Code: app/services/teams_installer.rb — when enabled for company, calls TeamsAppV2.install(company). Flag is off, so TeamsAppV2.install is not invoked.

7. legacy_give_modal — off / segment:legacy_plan / 14
   No code reference in flag_code.md; cannot state what it controls.

8. survey_boosters_q3 — on / segment:legacy_plan / 7
   No code reference in flag_code.md; cannot state what it controls.

9. paused_offboard_cleanup — off / (blank) / 0
   No code reference and no targeting rules/targets.

Arithmetic check: 42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0 = 456 total targeted-company entries across the export. On-state entries: 42 + 220 + 87 + 65 + 7 = 421. Off-state entries: 12 + 9 + 14 + 0 = 35. 421 + 35 = 456.
nda-triagePASS 1.00
Plain-English question: Triage each NDA green, yellow, or red — and quote the clause that decides it.
Prompt — what the model saw
Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0056 · 78s · in 614 / out 1,808 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

Plain-English answer: NDA-1 green (standard mutual terms), NDA-2 yellow for an embedded 18-month non-solicit, with the driving clause quoted for every classification.
The answer in one line: non_solicit_in = NDA-2
checksclassification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
classifications{"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_inNDA-2
graded checksgreen · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — [PARTY A] and Bonusly: GREEN (standard approval)
- Driving clauses: none adverse. Clause 3 carries all five standard carve-outs ("is or becomes publicly available through no breach... was known prior to disclosure... is received from a third party without duty of confidence... is independently developed... must be disclosed by law or court order"); clause 4 is Delaware law; clause 5 confirms "No license, no obligation to proceed, no exclusivity."
- Reasoning: Mutual NDA, full standard carve-outs, Delaware governing law, no non-solicit/non-compete; the 2-year term with 3-year survival (clause 2) is on the longer end but within market norms — approve as standard.

NDA-2 — [PARTY B] and Bonusly: YELLOW (counsel review)
- Driving clause: clause 4 — "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."
- Reasoning: An embedded non-solicitation covenant inside a confidentiality agreement is a restrictive covenant that counsel should review before signature, even though it is mutual and limited to employees contacted under the Agreement; the rest is standard (3-year term, full carve-outs in clause 2, Delaware law in clause 3).

NDA-3 — [PARTY C] and Bonusly: RED (full legal review)
- Driving clauses: clause 2 — "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business"; clause 4 — "No confidentiality carve-outs are stated in this Agreement"; clause 3 — "governed by and construed in accordance with the laws of the Republic of Ireland... exclusive jurisdiction of its courts"; clause 1 — one-way in favor of [PARTY C] with a 5-year term.
- Reasoning: A broad 3-year non-compete embedded in an NDA plus a total absence of standard carve-outs (no public-domain, prior-knowledge, independent-development, or legally-required-disclosure exceptions) plus unusual Irish governing law/exclusive jurisdiction plus a one-way structure and atypical 5-year term — four independent red flags requiring full legal review.