← back to all models ← RevenueBench home
nvidia

nvidia/nemotron-3.5-super-vl

Mean 0.945 · 31/40 perfect tests · $0.64 total · modalities in: text, image, video · out: text · each card: the prompt → the correct answer (gold) → this model's actual answer

What do the modalities mean?

Modalities in means what you can send this model: text, images, files, video, or audio. Modalities out means what it can send back. Bench tasks feed text extracts, so modalities do not change the score. They are here so you know what this model can handle in real work.

data-crm 5 tests
stage-hygiene-auditPASS 1.00
Plain-English question: How much of the open DS2+ pipeline is missing a substantive why-buys, and is anything missing DS1 timestamps or out of order?
Prompt — what the model saw
Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.0405 · 372s · in 120,736 / out 51,992 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

Plain-English answer: 81.5% of the 124 DS2+ deals are missing a substantive why-buys — Alex Franklin 48, Cole Ingram 19, Dana Mercer 18, Bryce Harmon 15, Farid Osman 1 — and every deal has a DS1 timestamp with none regressed.
The answer in one line: n_deals = 156; ds2plus = 124; noncomp_pct = 81.5; bryce_noncomp = 15; no_ds1 = 0; regressed = 0 (+1 more below)
n_deals156
bryce_noncomp15
ds2plus124
no_ds10
noncomp_pct81.5
regressed0
per_owner_noncompAlex Franklin=48 of 61 (78.7%); Cole Ingram=19 of 22 (86.4%); Dana Mercer=18 of 22 (81.8%); Bryce Harmon=15 of 18 (83.3%); Farid Osman=1 of 1 (100%)
graded checksnoncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions
- DS2+ deal: stage is DS2, DS3, DS4, or DS5.
- why-buy compliant: why_buys is non-empty, not a placeholder (tbd / n/a / see notes), and at least 15 characters. A LONG(n) value is treated as compliant. Blank, placeholder, or under-15-char values are non-compliant.
- Missing DS1: t_ds1 is blank.
- Regressed: among non-empty stage-entry timestamps t_ds1 → t_ds5 in stage order, an earlier stage's timestamp is later than a later stage's timestamp.

Results
- DS2+ deals: 124. Non-compliant on why-buys: 101 (101/124 = 81.45%). Compliant: 23 (all LONG(n); no substantive non-LONG values or placeholders exist in the data).
- Non-compliant by owner (DS2+ deals; non-compliant / total = percent):
  - Alex Franklin (84342457): 48/61 = 78.69%
  - Dana Mercer (83155923): 18/22 = 81.82%
  - Cole Ingram (83155924): 19/22 = 86.36%
  - Bryce Harmon (119337721): 15/18 = 83.33%
  - Farid Osman (716654662): 1/1 = 100.0%
- Deals with no DS1 timestamp: 0
- Deals with regressed stage-entry timestamps: 0

Note: Gavin Porter and Hugo Lindqvist are archived and have no open deals; Elena Sinclair's only open deal is at DS1, so neither appears in the DS2+ population.

Process fixes
1. Make why_buys a mandatory, gated field: block advancement past DS1 unless why_buys is present (>=15 chars, placeholder blacklist: tbd, n/a, see notes). LONG(n) is a display truncation, not a data gap — require and store the full text.
2. Enforce monotonic stage-entry timestamps at save (t_ds1 <= t_ds2 <= ... <= t_ds5) and run a daily stage-hygiene report that flags any inversion.
3. Auto-set t_ds1 on DS1 entry and assign stage-hygiene ownership per rep so the why-buy gap (81% non-compliant overall) is tracked and resolved by owner.
owner-resolutionPASS 1.00
Plain-English question: Every open deal has a blank owner name — resolve each one to a real person using the owners table.
Prompt — what the model saw
The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0182 · 129s · in 64,386 / out 21,970 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ bryce_amount✓ alex_amount✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: All 156 deals resolve to six named owners; jake_amount maps to Bryce Harmon ($1,054,144) and justin_amount to Alex Franklin ($624,310).
The answer in one line: jake_amount = 1,054,144; justin_amount = 624,310; resolved = 156
jake_amount1,054,144
justin_amount624,310
resolved156
deactivated
per_ownerBryce Harmon=1,054,144; Alex Franklin=624,310; Dana Mercer=341,195; Cole Ingram=288,161.43; Farid Osman=4,134; Elena Sinclair=2,100
graded checksresolved_count · unmatched_none · deactivated_flagged · bryce_amount · alex_amount · no_invented

Model's actual answer

show the model's full answer
Owner resolution complete. All 156 open deals resolved; every `owner_id` in `deals_open.csv` exists in `owners.csv`.

## 1. Deals resolved
**156 / 156** resolved to an owner name.

## 2. Deals whose owner id has no match in owners table
None. Count = 0. No unmatched ids to list (no names guessed).

## 3. Owner ids mapping to archived/deactivated owners
None appear on any deal. The owners table contains two archived owners — Gavin Porter (`1520255671`) and Hugo Lindqvist (`77260721`) — but zero deals in this extract carry either id, so the list of archived owners mapped to deals is empty.

## 4. Total pipeline amount per resolved owner

| Owner (id) | Deals | Total amount | Arithmetic (deal amounts summed) |
|---|---|---|---|
| Alex Franklin (`84342457`) | 67 | $624,310.00 | all amounts under id `84342457` |
| Bryce Harmon (`119337721`) | 35 | $1,054,144.00 | all amounts under id `119337721` |
| Cole Ingram (`83155924`) | 22 | $288,161.43 | includes decimal amounts (7225.40, 7781.20, 6947.50, 2480.40, 4779.88, 3334.80, 58529.25, etc.) |
| Dana Mercer (`83155923`) | 24 | $341,195.00 | all amounts under id `83155923` |
| Elena Sinclair (`701163055`) | 1 | $2,100.00 | Deal-57FF13 = 2100 |
| Farid Osman (`716654662`) | 7 | $4,134.00 | 1+1+1+2880+1+1 = 4134 |

**Grand total: 156 deals, $2,314,044.43**

Cross-check: sum of all deal amounts ($2,314,044.43) − unmatched amounts ($0.00) = sum of per-owner totals ($2,314,044.43). Deal counts also sum to 156 (67+35+22+24+1+7).
gong-hubspot-join-integrity0.75
Plain-English question: How many call records point at deal ids that do not exist in the open pipeline?
Prompt — what the model saw
Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.0363 · 190s · in 70,648 / out 32,018 tok
✓ orphan_count✗ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

Plain-English answer: 30 Gong call rows are orphans keyed to dead deal ids, none are duplicates, and DS3 call coverage is 29.4%.
The answer in one line: orphans = 30; ds3_cov_pct = 29.4; dup = 0
orphans30
ds3_cov_pct29.4
dup0
orphan_aliasesDeal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
graded checksorphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
Gong call-to-deal mapping integrity report

Data used: gong_calls_by_deal_90d.csv (67 rows) and deals_open.csv (156 open deals). No facts invented; every figure computed from the two files.

(a) Orphans — gong rows whose hs_deal_id has NO match in deals_open
Count: 30 of 67 gong rows (distinct deal ids = 30; no repeated deal id among them).
All 37 remaining gong rows match an open deal; the 30 orphans are the high-call block (calls_90d from 4 to 46).
Sample orphan aliases (deal id):
- Deal-8FA85D (60251290957)
- Deal-8FC3F9 (60251649055)
- Deal-3B7945 (60251639682)
- Deal-42B265 (61227242540)
- Deal-9CCC42 (61430316324)
- Deal-36EA09 (60251082126)
- Deal-9A43B4 (60250446726)
- Deal-605F3C (60251622924)
- Deal-E2D34B (60251652002)
- Deal-76821A (60251092117)

Arithmetic: 67 total gong rows − 37 matched to deals_open = 30 orphans.

(b) Duplicate conversation keys — rows where calls_90d > distinct_conversation_keys
Count: 0. Every gong row has calls_90d equal to distinct_conversation_keys, so there are no rows indicating duplicate/multiple conversations collapsed under one key.

(c) Open deals at stage DS3 or later — share with at least one logged call
Open deals at stage DS3/DS4/DS5: 85
- With ≥1 logged call (deal id present in gong with calls_90d ≥ 1): 25
- With no logged call: 60
Share = 25 / 85 = 29.4% (≈ 29%; 60/85 = 70.6% have no call).

Cross-check: gong rows mapping to open DS3+ deals = 25 (matches the "with call" count above). gong rows mapping to open deals overall = 37; 37 + 30 orphans = 67 total.

Note: gong_owner_id is a Gong user id and was not joined to HubSpot owner_id, per the caveat; no owner-id comparison was performed.

What's verified: orphan count, alias samples, duplicate-key check (zero), and the DS3+ call-coverage share. Nothing left for this request.

⚠️ File-mutation verifier: 1 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv`: `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` exists but this task has not seen its full c…
snowflake-sql-trapped-catalogPASS 1.00
Plain-English question: Write one Snowflake query for first-month usage per customer using only the provided, trap-laden catalog.
Prompt — what the model saw
Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0067 · 108s · in 4,240 / out 12,131 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

Plain-English answer: A single correct SQL query that uses the right catalog tables and avoids the trap columns — graded on the exact tables and columns.
The answer in one line: use_table = PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
avoid_stalePRODUCTION.HUBSPOT.DEALS (stale)
avoid_unpopulatedV2_LIVE.OBJECTS_DEALS (unpopulated)
forbidden_filterdeleted-giver exclusion must NOT be applied
outputone fenced SELECT with per-FROM/JOIN table-choice comment
required_filtersucceeded redemptions only
use_tablePRODUCTION.PLG.COMPANY_COHORT_SUMMARY
graded checksavoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- FROM: PRODUCTION.PLG.COMPANY_COHORT_SUMMARY — chosen as the company
-- dimension and first-month fact source: it is the only populated,
-- non-stale table in this catalog with per-company first-month
-- give/redemption rollups (M1_USERS, M1_GIVES, M1_REDEMPTIONS) and a
-- customer start date (SIGNUP_DATE). Avoided deal sources:
--   HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS — UNPOPULATED (do not use)
--   PRODUCTION.HUBSPOT.DEALS — STALE, last sync 2023-03 (do not use)
-- No JOIN is needed; every requested metric is a documented M1 rollup.
-- Scope notes: this catalog excerpt contains no event-level recognition or
-- giving table, so the documented "exclude deleted givers" filter is NOT
-- applied here (it understates history). The redemption rule
-- (STATE = 'succeeded') is documented for
-- PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2; that table's
-- company/date keys are not in this excerpt, so the first-month redemption
-- count is taken from the documented M1_REDEMPTIONS rollup.
SELECT
  COMPANY_ID                                                          -- company key implied by "one row per self-serve company"
    AS COMPANY_ID,
  DATE_TRUNC('month', SIGNUP_DATE)
    AS FIRST_CUSTOMER_MONTH,                                          -- first calendar month as a customer
  M1_USERS                                                            -- unique month-1 users; no event-level givers table exists in this excerpt
    AS UNIQUE_GIVERS,
  M1_GIVES                                                            -- recognitions given in month 1
    AS RECOGNITION_COUNT,
  M1_REDEMPTIONS                                                      -- redemptions in month 1
    AS SUCCESSFUL_REDEMPTION_COUNT
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
WHERE SIGNUP_DATE IS NOT NULL;
```
crm-completeness-enrichmentPASS 1.00
Plain-English question: Audit the CRM for missing and invalid fields, and propose the fixes.
Prompt — what the model saw
Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0145 · 148s · in 28,762 / out 22,017 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

Plain-English answer: 9 companies missing employee count, 6 missing HQ country, 4 invalid emails, 1 domain mismatch, 2 duplicate clusters, and 25 enrichment matches.
The answer in one line: n_contacts = 52; n_companies = 34; enrichment_matches = 25; invalid_email_count = 4; n_dup_clusters = 2; domain_mismatch_count = 1 (+4 more below)
n_contacts52
domain_mismatch_count1
enrichment_matches25
invalid_email_count4
n_companies34
n_dup_clusters2
company_missingindustry=0; employee_count=9; hq_country=6
domain_mismatchesCT-0011
duplicate_clusters['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
invalid_emailsCT-0010, CT-0080, CT-0081, CT-0192
graded checksdup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
CRM AUDIT — companies.csv (34 rows), contacts.csv (52 rows), zoominfo_enrichment.csv (25 rows)

NOTE: No deals file (owner, stage, amount, close date, why-buys) was provided. Deal-field completeness cannot be audited, and no pipeline amount at stake can be computed for any fix. All amounts below are stated as unknown, not invented.

1. COMPLETENESS PER FIELD

Companies (n = 34). Required fields: industry, employee_count, hq_country (company_alias and domain are 100% present).
- industry: 34/34 = 100.0% complete. Missing: none.
- employee_count: 25/34 = 73.5% complete. Missing: C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386, C-93C8BF.
- hq_country: 28/34 = 82.4% complete. Missing: C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB.

Contacts (n = 52). Required fields: email, title, persona (contact_key, company_alias, domain are 100% present).
- email: 52/52 = 100.0% present, but 4 are structurally invalid (see section 4). Valid-format: 48/52 = 92.3%.
- title: 39/52 = 75.0% complete. Missing: CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170.
- persona: 37/52 = 71.2% complete. Missing: CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181.

2. DUPLICATE COMPANY CLUSTERS

No company name column exists (aliases are opaque IDs), so clustering is by shared domain only. Two clusters found. Survivor rule: most complete fields; tie broken by first row in file. Both rows in each cluster are 5/5 complete, so the earlier alias survives.

Cluster 1 — domain acme-corp.com (2 rows)
- Survivor: C-0A092931 (industry=Technology, employee_count=500, hq_country=US)
- Retire: C-0A092932 (industry=tech, employee_count=510, hq_country=USA)
- Disagreements on retire row: industry tech vs Technology; employee_count 510 vs 500; hq_country USA vs US.

Cluster 2 — domain globex.io (2 rows)
- Survivor: C-0A092933 (industry=SaaS, employee_count=200, hq_country=US)
- Retire: C-0A092934 (industry=Technology, employee_count=200, hq_country=US)
- Disagreement on retire row: industry Technology vs SaaS. employee_count and hq_country agree.

3. INVALID EMAILS (4)

All four have a local part but no domain (missing text after "@"):
- CT-0010, company C-66D1FC, email "user0@", contact.domain 66d1fc.com
- CT-0080, company C-92D97D, email "user0@", contact.domain 92d97d.com
- CT-0081, company C-92D97D, email "user1@", contact.domain 92d97d.com
- CT-0192, company C-425E2A, email "user2@", contact.domain 425e2a.com

4. DOMAIN MISMATCHES (1)

- CT-0011, company C-66D1FC: email "user1@other-domain.com" (domain other-domain.com) does not match contact.domain 66d1fc.com or company.domain 66d1fc.com.

5. ENRICHMENT JOIN (ZoomInfo, matched by domain)

25 of 34 company rows have a ZoomInfo match. 9 have no match: C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20, C-0A092931, C-0A092932, C-0A092933, C-0A092934.

Fills applied only where CRM is empty and ZoomInfo has a value (8 fills, all employee_count = 400):
- C-EC3025 employee_count: blank → 400
- C-96039F employee_count: blank → 400
- C-44EA29 employee_count: blank → 400
- C-D04904 employee_count: blank → 400
- C-B23205 employee_count: blank → 400
- C-60C75F employee_count: blank → 400
- C-7BBDFA employee_count: blank → 400
- C-50D386 employee_count: blank → 400

Not fillable from provided data: hq_company for C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5 (ZoomInfo rows exist but hq is blank there too) and C-EE9FFB (no ZoomInfo row). industry is 100% complete in CRM, so no industry fills.

Conflicts where both CRM and ZoomInfo are populated and disagree — list both, recommend ZoomInfo as source (third-party enrichment):

Industry conflicts (10; CRM tech/Tech/Technology vs ZoomInfo Computer Software):
- C-66D1FC: CRM tech vs ZI Computer Software
- C-EC3025: CRM Technology vs ZI Computer Software
- C-44EA29: CRM tech vs ZI Computer Software
- C-92D97D: CRM Technology vs ZI Computer Software
- C-D04904: CRM Technology vs ZI Computer Software
- C-77A95A: CRM Technology vs ZI Computer Software
- C-AA8DDA: CRM Technology vs ZI Computer Software
- C-B25F40: CRM Technology vs ZI Computer Software
- C-60C75F: CRM tech vs ZI Computer Software
- C-425E2A: CRM "Tech " vs ZI Computer Software

hq_country conflicts (10; CRM US/USA vs ZoomInfo United States):
- C-66D1FC: CRM US vs ZI United States
- C-950043: CRM US vs ZI United States
- C-EC3025: CRM USA vs ZI United States
- C-96039F: CRM USA vs ZI United States
- C-77A95A: CRM US vs ZI United States
- C-B23205: CRM US vs ZI United States
- C-E51FB7: CRM USA vs ZI United States
- C-D0662E: CRM US vs ZI United States
- C-425E2A: CRM USA vs ZI United States
- C-2D7423: CRM USA vs ZI United States

employee_count: no CRM-vs-ZoomInfo conflicts where both are populated (all equal). Intra-CRM duplicate disagreement: C-0A092931 500 vs C-0A092932 510.

6. TOP 10 FIXES BY PIPELINE AMOUNT AT STAKE

Pipeline amounts cannot be scored: no deals data (owner, stage, amount, close date, why-buys) was provided. Amount at stake = unknown for every fix below. Ranking is by records affected and field criticality only.

1. Merge duplicate company cluster acme-corp.com → survivor C-0A092931, retire C-0A092932; reconcile industry Technology/tech, employee_count 500/510, hq_country US/USA. (2 company records; amount unknown)
2. Merge duplicate company cluster globex.io → survivor C-0A092933, retire C-0A092934; reconcile industry SaaS vs Technology. (2 company records; amount unknown)
3. Repair 4 invalid emails missing the domain part: CT-0010 (C-66D1FC), CT-0080 (C-92D97D), CT-0081 (C-92D97D), CT-0192 (C-425E2A). (4 contact records; amount unknown)
4. Fix domain mismatch CT-0011 (C-66D1FC): email user1@other-domain.com vs company domain 66d1fc.com. (1 contact record; amount unknown)
5. Fill 8 missing company employee_count values from ZoomInfo (all 400): C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386. (8 company records; amount unknown)
6. Resolve 10 hq_country US/USA → United States conflicts per ZoomInfo: C-66D1FC, C-950043, C-EC3025, C-96039F, C-77A95A, C-B23205, C-E51FB7, C-D0662E, C-425E2A, C-2D7423. (10 company records; amount unknown)
7. Resolve 10 industry tech/Tech/Technology → Computer Software conflicts per ZoomInfo: C-66D1FC, C-EC3025, C-44EA29, C-92D97D, C-D04904, C-77A95A, C-AA8DDA, C-B25F40, C-60C75F, C-425E2A. (10 company records; amount unknown)
8. Obtain missing contact titles for 13 records (no enrichment source available for contacts): CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170. (13 contact records; amount unknown)
9. Obtain missing contact personas for 15 records: CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181. (15 contact records; amount unknown)
10. Flag 6 companies with unresolvable missing hq_country from provided data (ZoomInfo hq also blank or no ZoomInfo row): C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB. Requires manual research. (6 company records; amount unknown)

No values were invented; all figures derived from the three provided files.
deal-intelligence 5 tests
closed-lost-classificationPASS 1.00
Plain-English question: Why did we lose the last six months of deals — one primary category per deal, and whose side is the reason on?
Prompt — what the model saw
Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0324 · 456s · in 65,253 / out 57,720 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

Plain-English answer: Every closed-lost deal gets exactly one category (pricing, competitor, no decision, timing, product gap, champion left, other) plus a side, per the computed classification.
The answer in one line: total_lost_6mo = 90; top_reason = competitor (26)
total_lost_6mo90
categoriespricing, competitor, no decision, timing, product gap, champion left, other
patternstwo most actionable, tag-vs-text disagreements counted
sideBonusly / buyer / unknown per deal
top_reasoncompetitor (26)
graded checkstotal_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Deal classification (closed_lost_6mo.csv, 90 deals, close_date 2026-07-29 to 2026-09-30)
Format: deal alias | primary category | side

timing (23):
Deal-DB0AAC   buyer
Deal-91A056   buyer
Deal-29326C   buyer
Deal-831B7B   buyer
Deal-39E25C   buyer
Deal-B3ABED   buyer
Deal-ED9AE7   buyer
Deal-B6AC09   buyer
Deal-E6E80A   buyer
Deal-B038F0   buyer
Deal-175756   buyer
Deal-BB78F3   buyer
Deal-15DA99   buyer
Deal-F4AF5D   buyer
Deal-79B7A1   buyer
Deal-9F176A   buyer
Deal-69CF3D   buyer
Deal-ECBF89   buyer
Deal-D1A623   buyer
Deal-55867E   buyer
Deal-DAFB82   buyer
Deal-2FEDDB   buyer
Deal-5E64CE   buyer

competitor (25):
Deal-F7F635   buyer
Deal-F97C37   buyer
Deal-422BA6   buyer
Deal-381C8C   buyer
Deal-F1E8A6   buyer
Deal-DDAB52   buyer
Deal-ACE061   buyer
Deal-2D2F8D   buyer
Deal-0F96AA   buyer
Deal-1BCA50   buyer
Deal-7CC678   unknown
Deal-242273   buyer
Deal-A2C349   buyer
Deal-C7156E   buyer
Deal-8A0992   buyer
Deal-D0C698   buyer
Deal-EECC02   buyer
Deal-47F1A1   buyer
Deal-BF2A98   buyer
Deal-1E7DA9   buyer
Deal-286F9C   buyer
Deal-369281   buyer
Deal-9FCD0D   buyer
Deal-64B19A   buyer
Deal-DC77FE   buyer

pricing (5):
Deal-7ED004   buyer
Deal-7B2236   buyer
Deal-C33D91   buyer
Deal-5AD03E   buyer
Deal-8A119B   buyer

no decision (9):
Deal-13E9CF   buyer
Deal-E74A73   buyer
Deal-FAC17C   buyer
Deal-50E5D8   buyer
Deal-413C56   buyer
Deal-2A292B   buyer
Deal-FEDBCB   buyer
Deal-7FBAC6   buyer
Deal-ABD14C   buyer

product gap (4):
Deal-8E27DA   Bonusly
Deal-9048EB   Bonusly
Deal-3618CC   Bonusly
Deal-981AD4   Bonusly

champion left (1):
Deal-F325A5   buyer

other (23):
Deal-AC944F   unknown
Deal-214060   unknown
Deal-5DB9B0   unknown
Deal-21B045   unknown
Deal-988493   unknown
Deal-F308CA   unknown
Deal-70F704   unknown
Deal-4664E1   unknown
Deal-D48E0B   unknown
Deal-583ADB   unknown
Deal-E0441F   unknown
Deal-7CB44D   unknown
Deal-AFA56C   unknown
Deal-D1AABF   unknown
Deal-2BBA21   unknown
Deal-386F6E   unknown
Deal-3F86A0   unknown
Deal-096750   unknown
Deal-79E61A   unknown
Deal-AE7C4E   unknown
Deal-DAB4F1   unknown
Deal-B4B50F   unknown
Deal-5885B9   unknown

Summary
Total deals: 90

Category counts:
timing          23
competitor      25
pricing          5
no decision      9
product gap      4
champion left    1
other           23

Side split:
buyer    62
Bonusly   4
unknown  24

Tag clearly disagrees with free-text reason: 3 deals
- Deal-5AD03E: tag Competitor, text "Wanted more defined budget access" (pricing, no competitor named)
- Deal-DAFB82: tag Lost- Budget/Price, text "said she loves Bonusly ... not budgeted in until 2028" (timing, no price objection)
- Deal-9048EB: tag MIA, text "bad fit based on their desired setup and multiple feature gaps" with leadership confirming close/lost (product gap)

Two patterns most worth acting on:
1. Timing is the largest recoverable pool (23 deals). Nearly all are deferred to 2027 or 2028, "on hold," or "pause" with stated reconnect windows (early 2027, Q2 2028, contract end Oct 2027, end of year). Run a structured re-engagement/nurture program tied to those dates.
2. Competitor is the single largest category (25 deals) but is mostly not price-driven. Reps repeatedly state price was not the factor (Deal-DC77FE price "wasn't a factor"; Deal-DDAB52 Rippl "same cost"; Deal-D0C698 returning to Kudos). Losses go to partnerships/integrations (Deal-422BA6 ADP TotalSource PEO partner), product fit (Deal-A2C349 Awardco surveys; Deal-981AD4 UI / not UK focused), or regional preference (Canadian/UK providers). Compete on positioning, integrations, and fit, not discounting.

Note: 24 deals carry an "unknown" side (MIA/unresponsive with no captured reason) — a data-capture gap, not a loss signal.
pipeline-tieringPASS 1.00
Plain-English question: Score all 156 open deals into exactly one of LOCK / ACTION / BUILD / REVIVE / WATCH / RISKY.
Prompt — what the model saw
Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.0345 · 247s · in 117,341 / out 28,086 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

Plain-English answer: All 156 open deals tiered exactly once across the six tiers using forecast category, stage, meetings, contacts, and recency.
The answer in one line: total = 156
total156
graded checksjson_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{
  "tier_counts": {"LOCK": 3, "ACTION": 22, "BUILD": 45, "REVIVE": 3, "WATCH": 82, "RISKY": 1},
  "tier_examples": {"LOCK": ["Deal-D348E1", "Deal-C26D20", "Deal-403845"], "ACTION": ["Deal-25F752", "Deal-E53952", "Deal-5408B0"], "BUILD": ["Deal-C6FE92", "Deal-D73B89", "Deal-012CB1"], "REVIVE": ["Deal-2D1F1B", "Deal-3EED2C", "Deal-57FF13"], "WATCH": ["Deal-6787C2", "Deal-66D1FC", "Deal-950043"], "RISKY": ["Deal-7BBDFA"]},
  "risky_deals": ["Deal-7BBDFA"],
  "lock_violations": 0,
  "pipeline_shape": "The open pipeline (156 deals) is weighted toward early-stage PIPELINE forecasts in DS1-DS3 held by a small rep set, with a thin late-stage DS4-DS5 COMMIT core. Because inbound email is empty and most rows show zero meetings in the last 30 days, only three late-stage COMMIT deals carry a recent meeting and enough contacts to tier LOCK; roughly half the pipeline sits in WATCH (no recent meeting, contact still within about 60 days) and BUILD (early/mid-stage with at least one meeting), while a small REVIVE group is stale beyond 60 days. One BEST_CASE deal, Deal-7BBDFA, tiers RISKY because its optimistic forecast conflicts with cold engagement (no meeting, last contact about 45 days)."
}
call-transcript-extractionPASS 1.00
Plain-English question: Extract the CRM write-back fields from each call transcript as JSON.
Prompt — what the model saw
For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0050 · 67s · in 5,240 / out 7,658 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

Plain-English answer: Per transcript: prospect-only why-buys, pain points, stakeholders, budget and timeline signals, and competitors — e.g. TX-001: '$40k earmarked', live before open enrollment.
The answer in one line: rep = Alex Franklin
allowed_newTX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
per_transcript{"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
repAlex Franklin
rep_sourced_nulls{"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
graded checksjson_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
{
  "transcripts": [
    {
      "transcript_id": "TX-001",
      "deal_alias": "Deal-CFE7F4",
      "why_buys": [
        "The big win for us would be automating anniversary and birthday awards — our HR team of three cannot keep up with it manually."
      ],
      "pain_points": [
        "HR team of three cannot keep up with anniversary/birthday awards manually",
        "Track everything in a spreadsheet; people slip through the cracks",
        "Need SSO and audit logs for IT to sign off"
      ],
      "stakeholders": [
        "Prospect (VP People)",
        "Prospect (HR Admin)"
      ],
      "budget_signal": "We have about $40k earmarked for engagement tools this fiscal year.",
      "timeline_signal": "Ideally we would have this live before open enrollment in November.",
      "competitor_mentioned": "Achievers",
      "next_step": "Yes — let's do the security review on September 12.",
      "objections": [
        "One concern: we need SSO and audit logs for IT to sign off."
      ],
      "confidence": "high"
    },
    {
      "transcript_id": "TX-002",
      "deal_alias": "Deal-70BB30",
      "why_buys": [
        "We want to tie recognition to retention for our hourly workforce — regretted turnover there is over 30%."
      ],
      "pain_points": [
        "Regretted turnover over 30% for hourly workforce",
        "Integration with Workday has to be rock solid — that's my one condition."
      ],
      "stakeholders": [
        "Prospect (Head of Total Rewards)",
        "Prospect (CFO)"
      ],
      "budget_signal": "Finance has approved a $25k pilot budget for this quarter.",
      "timeline_signal": "We want a decision by end of September.",
      "competitor_mentioned": null,
      "next_step": "Yes — send the pilot agreement and we'll route it to legal this week.",
      "objections": [
        "Integration with Workday has to be rock solid — that's my one condition."
      ],
      "confidence": "high"
    },
    {
      "transcript_id": "TX-003",
      "deal_alias": "Deal-530B50",
      "why_buys": [
        "We need to make recognition visible across our 12 retail locations."
      ],
      "pain_points": [
        "Recognition not visible across 12 retail locations",
        "Store managers have zero budget autonomy for on-the-spot recognition today",
        "The CEO has to be sold first — she decides anything people-related"
      ],
      "stakeholders": [
        "Prospect (People Ops Manager)"
      ],
      "budget_signal": null,
      "timeline_signal": "Honestly there's no rush on our side until Q1.",
      "competitor_mentioned": "Bucketlist",
      "next_step": "Yes, let's schedule a call with our CEO — I'll send two times.",
      "objections": [],
      "confidence": "high"
    },
    {
      "transcript_id": "TX-004",
      "deal_alias": "Deal-180D02",
      "why_buys": [
        "We want to consolidate three separate recognition tools into one.",
        "We're paying for three tools and none of them talk to our HRIS."
      ],
      "pain_points": [
        "Paying for three tools and none of them talk to our HRIS",
        "Procurement cycle runs six to eight weeks minimum",
        "The security review took three months for our last vendor — that's my hesitation."
      ],
      "stakeholders": [
        "Prospect (VP People)",
        "Prospect (IT Security Lead)"
      ],
      "budget_signal": "If it's under $15k annually, I can approve it without going to the board.",
      "timeline_signal": null,
      "competitor_mentioned": null,
      "next_step": null,
      "objections": [
        "The security review took three months for our last vendor — that's my hesitation."
      ],
      "confidence": "medium"
    },
    {
      "transcript_id": "TX-005",
      "deal_alias": "Deal-F8767A",
      "why_buys": [
        "Two things: automate service milestones, and give us analytics on recognition equity across departments."
      ],
      "pain_points": [
        "Our night-shift teams feel invisible — their engagement scores run 20 points lower.",
        "Our exec team is skeptical after a failed rollout two years ago."
      ],
      "stakeholders": [
        "Prospect (HR Director)",
        "Prospect (People Ops Coordinator)"
      ],
      "budget_signal": "We have $12k approved under our engagement line.",
      "timeline_signal": "We need this running before our January all-hands.",
      "competitor_mentioned": "Nectar",
      "next_step": "Yes — come present to our exec team on October 2.",
      "objections": [
        "Our exec team is skeptical after a failed rollout two years ago."
      ],
      "confidence": "high"
    },
    {
      "transcript_id": "TX-006",
      "deal_alias": "Deal-EE195F",
      "why_buys": [
        "I'd love to cut the admin time on service awards.",
        "I personally spend five hours a month ordering and shipping plaques."
      ],
      "pain_points": [
        "I personally spend five hours a month ordering and shipping plaques.",
        "Our COO usually prefers building things in-house."
      ],
      "stakeholders": [
        "Prospect (HR Manager)"
      ],
      "budget_signal": null,
      "timeline_signal": "Our fiscal year resets in February, so a Q1 start is realistic.",
      "competitor_mentioned": null,
      "next_step": "Yes — send the one-page overview and I'll forward it to our COO this week.",
      "objections": [
        "Fair warning, our COO usually prefers building things in-house."
      ],
      "confidence": "high"
    }
  ]
}

Notes: budget_signal and timeline_signal are null where the prospect stated no amount/date (TX-003, TX-006 budget; TX-004 timeline). competitor_mentioned is null unless the prospect named a vendor (internal comparison in TX-006 is not a competitor). next_step is null for TX-004 because the prospect said "maybe — no promises," which is not an explicit agreement. All fields sourced only from prospect statements; rep statements (e.g., Alex Franklin's $8/employee/month pricing in TX-003) were not used.
next-to-close0.75
Plain-English question: Which 3 deals are closest to signature right now?
Prompt — what the model saw
Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0027 · 20s · in 4,632 / out 2,120 tok
✓ exactly_three✗ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

Plain-English answer: Deal-547B2B ($11,200), Deal-B7EBD1 ($9,000), and Deal-403845 ($9,000) — all DS5 COMMIT deals closing September 10-11, with what is left on each.
blocked_excludeddeal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
exactly_three1
ranking_ruleSlack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
top3rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
graded checksexactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
1. Deal-547B2B — $11,200 (Alex Franklin)
   Why close: Slack 09-04 09:12 — redlines came back clean, signing page is out, VP People said they are signing tomorrow. Alex called it "signature-imminent."
   Left: customer signature.

2. Deal-403845 — $9,000 (Dana Mercer)
   Why close: Slack 09-04 10:02 — order form is with their finance team, moving fine.
   Left: finance sign-off, then execution.

3. Deal-D348E1 — $13,770 (Alex Franklin)
   Why close: Slack 09-04 09:44 — still warm, normal legal-review pace.
   Left: complete legal review.

Deal-2465CE ($5,400) is excluded: Slack 09-04 09:20 — champion left, procurement froze vendors, pulled out of commit, now Q4.

Arithmetic: $11,200 + $9,000 + $13,770 = $33,970 combined.

Note: Deal-D348E1 and Deal-A2B47C ($6,360) got identical Slack status ("normal legal-review pace"); I ranked D348E1 third (listed first by Alex, larger commit).
product-gap-detectionPASS 1.00
Plain-English question: Which prospect complaints are true product gaps versus rollout or enablement issues?
Prompt — what the model saw
Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0029 · 28s · in 3,877 / out 2,656 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

Plain-English answer: Two real gaps (SCIM provisioning, ADP Workforce Now integration) separated from plan gates and enablement issues, each with the prospect quote and deal.
allowed_newTG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
enablement_issuestranscript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voicetranscript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
plan_gatestranscript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
real_gapstranscript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
graded checksreal_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Real gaps raised by prospects (capability not supported per product docs):

1. Deal-EC3025 — SCIM user provisioning
   Prospect (TG-001, IT Security Lead): "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
   Docs basis: Supported integrations states "SCIM user provisioning and ADP Workforce Now integrations are NOT currently listed as supported capabilities." No tier (Core/Pro/Enterprise) lists SCIM.
   Amount: not provided in the data.

2. Deal-D0D6B5 — ADP Workforce Now integration
   Prospect (TG-002, HRIS Manager): "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
   Docs basis: HRIS integrations listed are Workday, BambooHR, Gusto, Rippling (Pro and above); the docs explicitly state ADP Workforce Now is NOT currently a supported capability.
   Amount: not provided in the data.

Classified but excluded from the summary (not real gaps):

- Deal-CFE7F4 — plan gate. Prospect (VP People): "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?" Custom report builder is Enterprise-tier only per the plans table; the capability exists and is gated by plan, not missing.
- Deal-84DBA6 — rollout/enablement issue. Prospect (People Ops Manager): "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it." Slack is supported on all plans; this is a training/adoption issue, not a product capability gap.
- Deal-36C33F — rep language, not prospect voice. Alex Franklin stated the native mobile app for hourly workers is not yet available; the prospect (HR Manager) replied the web version is fine for office staff. No prospect-raised gap.

Tally: 5 transcripts, 4 prospect-raised topics → 2 real gaps, 1 plan gate, 1 rollout/enablement issue, 0 prospect gaps in TG-005.

Amounts: transcripts_gaps.csv has no amount column, so no deal amounts can be named; none invented.
rep-performance 5 tests
stale-pipeline-by-rep0.83
Plain-English question: Which open deals have had no email, call, or meeting in the last 7 days?
Prompt — what the model saw
The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.0279 · 248s · in 70,207 / out 39,881 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ bryce_top_amount✓ cole_stale_amt✗ dana_stale_amt✓ stale_count

✓ Correct Answer (ground truth)

Plain-English answer: 65 open deals are stale; the largest is Deal-2D1F1B at $240,000, with Bryce Harmon carrying the most stale dollars at $626,243.
The answer in one line: jake_amt = 626,243; jenn_amt = 261,645; kevin_amt = 252,905.03; top_amount = 240,000; stale_count = 65; top_alias = Deal-2D1F1B
jake_amt626,243
jenn_amt261,645
kevin_amt252,905.03
stale_count65
top_amount240,000
top_aliasDeal-2D1F1B
graded checksstale_recall_ge_80 · grouped_by_owner · bryce_top_amount · cole_stale_amt · dana_stale_amt · stale_count

Model's actual answer

show the model's full answer
Stale-open-deal report (snapshot 2026-09-05; stale = no email/call/meeting on or after 2026-08-29, i.e. >7 days ago; recency from engagements.last_email/last_call/last_meeting only, deal last_contacted_field ignored; future-dated engagements excluded as not yet occurred).

== Alex Franklin | stale deals: 20 | total stale amount: 113,936.00 ==
Deal-CC08D1 | Alex Franklin | DS1 | 24000 | 16 days
Deal-E73427 | Alex Franklin | DS3 | 18000 | 10 days
Deal-885F45 | Alex Franklin | DS2 | 9300 | 12 days
Deal-C2FF3C | Alex Franklin | DS1 | 8316 | 10 days
Deal-3EED2C | Alex Franklin | DS2 | 7200 | no logged contact
Deal-0D2F7A | Alex Franklin | DS3 | 5100 | 12 days
Deal-6C60D4 | Alex Franklin | DS3 | 4800 | 12 days
Deal-13FEBD | Alex Franklin | DS2 | 4680 | 12 days
Deal-819506 | Alex Franklin | DS1 | 4400 | 8 days
Deal-9D0060 | Alex Franklin | DS3 | 3840 | 12 days
Deal-690476 | Alex Franklin | DS2 | 3600 | 18 days
Deal-C6D97A | Alex Franklin | DS4 | 3240 | 8 days
Deal-EE195F | Alex Franklin | DS3 | 3120 | 8 days
Deal-278DEC | Alex Franklin | DS3 | 2700 | 8 days
Deal-635B8E | Alex Franklin | DS3 | 2600 | 18 days
Deal-6883F3 | Alex Franklin | DS1 | 2400 | 16 days
Deal-4A13AD | Alex Franklin | DS3 | 2160 | 26 days
Deal-F67D31 | Alex Franklin | DS2 | 1800 | 8 days
Deal-5FDCE4 | Alex Franklin | DS3 | 1600 | 12 days
Deal-BA571A | Alex Franklin | DS4 | 1080 | 18 days

== Bryce Harmon | stale deals: 18 | total stale amount: 692,964.00 ==
Deal-2D1F1B | Bryce Harmon | DS1 | 240000 | 81 days
Deal-66D1FC | Bryce Harmon | DS1 | 99000 | 16 days
Deal-950043 | Bryce Harmon | DS1 | 70000 | 19 days
Deal-B23205 | Bryce Harmon | DS1 | 45000 | 16 days
Deal-7BBDFA | Bryce Harmon | DS3 | 37440 | 46 days
Deal-332637 | Bryce Harmon | DS2 | 36000 | 9 days
Deal-1BEEBF | Bryce Harmon | DS1 | 31500 | 19 days
Deal-A414F6 | Bryce Harmon | DS1 | 25200 | 19 days
Deal-C5658B | Bryce Harmon | DS1 | 23400 | 16 days
Deal-40522D | Bryce Harmon | DS3 | 21000 | 19 days
Deal-C1FA6D | Bryce Harmon | DS1 | 18000 | 16 days
Deal-01E193 | Bryce Harmon | DS1 | 12600 | 8 days
Deal-F0EBBB | Bryce Harmon | DS3 | 11400 | 24 days
Deal-927338 | Bryce Harmon | DS1 | 10920 | 18 days
Deal-E25A09 | Bryce Harmon | DS1 | 6000 | 9 days
Deal-C9C286 | Bryce Harmon | DS2 | 5502 | 9 days
Deal-012CB1 | Bryce Harmon | DS1 | 1 | 23 days
Deal-3795AD | Bryce Harmon | DS2 | 1 | 8 days

== Cole Ingram | stale deals: 18 | total stale amount: 252,905.03 ==
Deal-D04904 | Cole Ingram | DS2 | 58529.25 | 11 days
Deal-B25F40 | Cole Ingram | DS3 | 40000 | 8 days
Deal-813836 | Cole Ingram | DS2 | 32175 | 11 days
Deal-1BA595 | Cole Ingram | DS2 | 31750 | 11 days
Deal-CFE1E8 | Cole Ingram | DS3 | 18000 | 11 days
Deal-CD47A6 | Cole Ingram | DS2 | 12168 | 11 days
Deal-627646 | Cole Ingram | DS3 | 11193 | 11 days
Deal-FF809F | Cole Ingram | DS2 | 7781.2 | 11 days
Deal-AF932D | Cole Ingram | DS2 | 7225.4 | 11 days
Deal-A71728 | Cole Ingram | DS2 | 6947.5 | 11 days
Deal-8BC9F5 | Cole Ingram | DS2 | 5616 | 10 days
Deal-175395 | Cole Ingram | DS3 | 4779.88 | 11 days
Deal-481E24 | Cole Ingram | DS3 | 4140 | 10 days
Deal-C7F9BF | Cole Ingram | DS2 | 3360 | 11 days
Deal-2F3A66 | Cole Ingram | DS3 | 3334.8 | 11 days
Deal-342E96 | Cole Ingram | DS2 | 2700 | 24 days
Deal-E568D5 | Cole Ingram | DS3 | 1875 | 11 days
Deal-FD9F4E | Cole Ingram | DS5 | 1330 | 10 days

== Dana Mercer | stale deals: 16 | total stale amount: 279,495.00 ==
Deal-44EA29 | Dana Mercer | DS2 | 60000 | 10 days
Deal-E51FB7 | Dana Mercer | DS2 | 43875 | 12 days
Deal-B42F46 | Dana Mercer | DS1 | 27000 | 19 days
Deal-BA3DDC | Dana Mercer | DS3 | 23400 | 15 days
Deal-9DDE86 | Dana Mercer | DS2 | 20000 | 15 days
Deal-215CCA | Dana Mercer | DS3 | 18900 | 17 days
Deal-5EED42 | Dana Mercer | DS3 | 16250 | 11 days
Deal-57887A | Dana Mercer | DS2 | 15000 | 8 days
Deal-944310 | Dana Mercer | DS4 | 10500 | 33 days
Deal-3974EB | Dana Mercer | DS4 | 9000 | 8 days
Deal-B7EBD1 | Dana Mercer | DS5 | 9000 | 16 days
Deal-F40F04 | Dana Mercer | DS2 | 8100 | 15 days
Deal-7599B8 | Dana Mercer | DS3 | 7350 | 18 days
Deal-87DDD1 | Dana Mercer | DS1 | 5000 | 19 days
Deal-F336B6 | Dana Mercer | DS3 | 4200 | 15 days
Deal-0660B4 | Dana Mercer | DS4 | 1920 | 16 days

== Elena Sinclair | stale deals: 1 | total stale amount: 2,100.00 ==
Deal-57FF13 | Elena Sinclair | DS1 | 2100 | no logged contact

== Farid Osman | stale deals: 2 | total stale amount: 2,881.00 ==
Deal-8BA24E | Farid Osman | DS1 | 2880 | 8 days
Deal-8FDCD2 | Farid Osman | DS1 | 1 | 15 days

TOTAL: 75 stale deals | total stale amount 1,344,281.03 across 6 owners. (Gavin Porter and Hugo Lindqvist are archived and carry no open deals.)

Notes on edge handling: Deal-3EED2C and Deal-57FF13 have no row in engagements_by_deal_90d, so they have no logged email/call/meeting and are stale with "no logged contact." Future-dated engagements (e.g. last_meeting after 2026-09-05) were not counted as contact; the most recent on-or-before-snapshot email/call/meeting was used. Amounts summed exactly as given in deals_open.csv.
activity-mix-vs-outcomePASS 1.00
Plain-English question: Per rep, what did the last 30 days of emails, calls, and meetings look like next to the deals that advanced?
Prompt — what the model saw
The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.0395 · 278s · in 109,932 / out 46,596 tok
✓ alex_ds2_30d✓ alex_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

Plain-English answer: Alex Franklin is the standout signal with 18 deals entering DS2 in 30 days (next closest: Bryce Harmon with 4), alongside per-rep activity totals.
The answer in one line: alex_ds2_30d = 18; bryce_ds2_30d = 4; note = legacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
alex_ds2_30d18
bryce_ds2_30d4
notelegacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
graded checksalex_ds2_30d · alex_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
**Method:** snapshot = 2026-09-05; 30-day window = 2026-08-06 through 2026-09-05 (inclusive). Activity totals (emails_30d + calls_30d + meetings_30d) are summed across every deal owned by each rep that has an engagement row. DS2 entries = deals with a non-empty t_ds2 falling in the window. Activities per DS2 entry = total activities ÷ DS2 entries (lower = more efficient). Reps with zero DS2 entries have an undefined ratio and are excluded from the ranking. All arithmetic verified in code; per-rep sums re-check OK.

### Per-rep results

| Rep (owner_id) | Emails | Calls | Meetings | Total activities | Email % | Call % | Meeting % | DS2 entries (last 30d) | Activities per DS2 entry |
|---|---|---|---|---|---|---|---|---|---|
| Alex Franklin (84342457) | 307 | 36 | 41 | 384 | 79.9% | 9.4% | 10.7% | 18 | 21.33 |
| Bryce Harmon (119337721) | 162 | 0 | 43 | 205 | 79.0% | 0.0% | 21.0% | 4 | 51.25 |
| Cole Ingram (83155924) | 96 | 14 | 1 | 111 | 86.5% | 12.6% | 0.9% | 2 | 55.50 |
| Farid Osman (716654662) | 38 | 0 | 34 | 72 | 52.8% | 0.0% | 47.2% | 1 | 72.00 |
| Dana Mercer (83155923) | 84 | 18 | 11 | 113 | 74.3% | 15.9% | 9.7% | 1 | 113.00 |
| Elena Sinclair (701163055) | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0 | n/a |
| Gavin Porter (1520255671) | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0 | n/a | (archived)
| Hugo Lindqvist (77260721) | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0 | n/a | (archived)

### Efficiency ranking (lowest activities per DS2 entry = most efficient)

1. Alex Franklin — 384 activities / 18 DS2 entries = **21.33**
2. Bryce Harmon — 205 activities / 4 DS2 entries = 51.25
3. Cole Ingram — 111 activities / 2 DS2 entries = 55.50
4. Farid Osman — 72 activities / 1 DS2 entry = 72.00
5. Dana Mercer — 113 activities / 1 DS2 entry = 113.00

### Key calls
- Most efficient rep: **Alex Franklin** (21.33 activities per DS2 entry).
- Highest-volume rep (total activities): **Alex Franklin** (384).
- Do they differ? **No** — same rep.
- DS2 entries within the window, by alias:
  - Alex Franklin: Deal-403845 (09-02), Deal-1FC049 (09-03), Deal-3EED2C (09-03), Deal-7FA0C3 (08-07), Deal-E531A6 (08-07), Deal-5296C9 (08-28), Deal-36C33F (08-11), Deal-EE195F (08-06), Deal-F436DA (08-19), Deal-317E6F (08-12), Deal-D1E6C2 (08-11), Deal-D9A72E (08-06), Deal-CA5E44 (08-24), Deal-4F775F (08-17), Deal-898FC5 (08-28), Deal-46988D (08-26), Deal-E73427 (08-28), Deal-92D97D (09-02)
  - Bryce Harmon: Deal-25F752 (08-10), Deal-D73B89 (09-03), Deal-CA7DC0 (08-12), Deal-1CCE5C (08-06)
  - Cole Ingram: Deal-42326B (08-26), Deal-1BA595 (08-12)
  - Farid Osman: Deal-499BF6 (08-26)
  - Dana Mercer: Deal-57887A (08-07)
- No cause is attributed. Elena Sinclair owns Deal-57FF13 but has no engagement row and no DS2 entry, so she contributes 0 activities. Archived reps Porter and Lindqvist have no deals or engagements in the data.

⚠️ File-mutation verifier: 2 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv`: `/Users/amaniphipps/.hermes/cache/scratch/deals_open.csv` exists but this task has not seen its full c…
  • `/Users/amaniphipps/.hermes/cache/scratch/owners.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/owners.csv`: `/Users/amaniphipps/.hermes/cache/scratch/owners.csv` exists but this task has not seen its full current c…
ae-qtd-scorecardPASS 1.00
Plain-English question: Build Alex Franklin's QTD scorecard: bookings versus quota, pipeline, and conversion.
Prompt — what the model saw
Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0801 · 281s · in 136,998 / out 40,587 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

Plain-English answer: 75% attainment ($150,000 on a $200,000 quota), $113,500 new versus $36,500 expansion, pipeline by stage, and the rolling DS2-to-won rate.
The answer in one line: quota = 200000; bookings = 150000; new_bookings = 113500; expansion_bookings = 36500; open_deals_count = 125; attainment_pct = 75; owner = Alex Franklin (+12 more below)
quota200,000
attainment_pct75
bookings150,000
expansion_bookings36,500
expansion_count3
losses_qtd27
new_bookings113,500
new_count5
open_deals_count125
wins_qtd8
activity_30d_totalsemails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
allowed_new18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
ds2_to_wonwindow_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
ownerAlex Franklin
pipeline_by_stage{"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
prior_quarter_excludeddeal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
quarter2026-Q3
snapshot_date2026-09-05
top_loss_reasonreason=Lost- Timing (1 year or more); count=13
graded checksbookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD SCORECARD — Alex Franklin, as of 2026-09-05 (Q3 2026 = 2026-07-01 to 2026-09-05; quota snapshot 2026-09-05)

1) BOOKINGS VS QUOTA
- Quota: 200,000
- QTD closed-won bookings: 150,000 (8 deals)
- Attainment: 150,000 / 200,000 = 75.00%
- Gap to quota: 50,000
- Excluded pre-quarter closed-won (dated before quarter): Deal-B3E6F1, 24,000, closed 2026-06-20

QTD won deals (alias, type, amount, close date):
Deal-A1C3E5, new, 40,000, 2026-07-15
Deal-F2C7D8, expansion, 20,000, 2026-07-24
Deal-B7D2F4, new, 35,000, 2026-07-31
Deal-C9E1A6, new, 21,000, 2026-08-12
Deal-A8B4D6, expansion, 12,000, 2026-08-19
Deal-D4B8C2, new, 11,000, 2026-08-21
Deal-E6F3A9, new, 6,500, 2026-09-02
Deal-C5D9E2, expansion, 4,500, 2026-09-03

Arithmetic: 40,000 + 20,000 + 35,000 + 21,000 + 12,000 + 11,000 + 6,500 + 4,500 = 150,000.

2) NEW VS EXPANSION SPLIT
- New: 113,500 (40,000 + 35,000 + 21,000 + 11,000 + 6,500) = 75.7% of bookings
- Expansion: 36,500 (20,000 + 12,000 + 4,500) = 24.3% of bookings
- Check: 113,500 + 36,500 = 150,000

3) ACTIVE PIPELINE BY STAGE (open deals, 125 total, 1,260,390)
- DS1: 20 deals, 284,621
- DS2: 28 deals, 353,760
- DS3: 67 deals, 552,705
- DS4: 5 deals, 23,574
- DS5: 5 deals, 45,730
- Check: 284,621 + 353,760 + 552,705 + 23,574 + 45,730 = 1,260,390; 20 + 28 + 67 + 5 + 5 = 125

4) ROLLING 90-DAY DS2-TO-WON RATE (window 2026-06-07 to 2026-09-05)
- Deals entering DS2 in window: 111 (8 won, 27 lost, 76 open)
- Decided DS2 deals (won + lost): 35
- DS2-to-won: 8 / 35 = 22.86%
- Basis: won divided by decided (won + lost) entrants; 76 open DS2 entrants have no outcome yet and are excluded from the denominator.
- (All entrants basis: 8 / 111 = 7.21%)

5) WIN / LOSS COUNTS AND TOP LOSS REASON (QTD)
- Wins: 8
- Losses: 27
- QTD lost amount: 329,272 (2.2x bookings)
- Loss reason counts:
  - Lost- Timing (1 year or more): 13 (48.1% of losses) — TOP
  - MIA: 5
  - Competitor: 5
  - Lost DM: 2
  - Feature Request: 1
  - Lost- Does not fit ICP (write in notes): 1
- Check: 13 + 5 + 5 + 2 + 1 + 1 = 27

6) ACTIVITY VOLUME, LAST 30 DAYS (2026-08-06 to 2026-09-05; per-deal 30-day aggregates summed across 161 deals)
- Emails: 807
- Calls: 112
- Meetings: 128
- Notes: 50
- Total activity: 1,097
- Arithmetic: 807 + 112 + 128 + 50 = 1,097

COACHING OBSERVATIONS
1. Conversion is the leak, not pipeline volume. The AE logged 27 QTD losses against 8 wins (a 1:3.4 win:loss ratio) and a decided DS2-to-won rate of only 22.86% (8 of 35). Lost amount (329,272) is 2.2x booked amount (150,000). Coaching priority: tighten discovery and competitive positioning so fewer DS2 entrants end in loss.

2. Nearly half of losses are to timing, not a better offer. "Lost- Timing (1 year or more)" accounts for 13 of 27 losses (48.1%), ahead of MIA (5) and Competitor (5). This signals weak timing qualification at entry — too many opportunities with a 1-year-plus buying window are entering the pipe. Coach the AE to hard-qualify timing at DS1/DS2 and deprioritize non-near-term buyers.

3. Late-stage coverage is thin relative to the gap. DS4 + DS5 open pipeline is just 10 deals worth 69,304 against a 50,000 quota gap, while DS1 + DS2 holds 638,381 across 48 deals. Because coverage is concentrated in few late deals, slippage of any single DS4/DS5 deal would push the AE below quota again. Coach the AE to advance and multi-thread the mid-funnel (DS2/DS3) to build more late-stage coverage for next quarter.

Data notes: All figures computed from the three provided files only; no facts, deals, or numbers invented. The quarter is Q3 2026 (2026-07-01 to 2026-09-30); QTD is measured to the 2026-09-05 snapshot. Closed-won Deal-B3E6F1 (2026-06-20) predates the quarter and is excluded per instructions. No missing data affected these results.
multithreading-gap0.75
Plain-English question: Which open deals are single-threaded or under-threaded?
Prompt — what the model saw
Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0082 · 124s · in 5,501 / out 15,683 tok
✓ flagged_recall✓ persona_language✗ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

Plain-English answer: Deals with fewer than 2 active contacts (single-threaded) or fewer than 3, or all in one persona (under-threaded), each flagged with the missing persona.
The answer in one line: active_window_start = 2026-07-07
active_window_start2026-07-07
allowed_new59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
best_persona_ruleDS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flag_ruleflag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
flaggeddeal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flaggedDeal-4B0BEB, Deal-84DBA6, Deal-D348E1
persona_universeeconomic buyer, champion, HR admin, IT security, finance
graded checksflagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Reference date: 2026-10-19 (session start). Active = last_engaged_date on or after 2026-08-20 and is_former = false. Arithmetic: 2026-08-20 to 2026-10-19 = 60 days (Aug 20-31 = 11, September = 30, Oct 1-19 = 19); 2026-08-19 = 61 days, so it is excluded (this excludes CT-DE6D7C 08-19, CT-8742FD 08-18, CT-BEDF5E 08-19, CT-4FE556 08-15 and includes CT-3F23A4 08-20 and CT-B03555 08-20). Flag rule: active contacts < 3, or all active contacts in one persona. Amount, stage, and deal open/closed status are not in either file, so they are reported as "not on file"; therefore the stage-specific most-valuable-persona to add is not determinable for any flagged deal from the provided data.

Flagged deals:

Deal-EC3025 (61032318100), company C-FDD0C7
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: CT-6827DB, Chief People Officer, economic buyer (fits missing persona economic buyer); fit to stage-based pick not confirmable (stage not on file)

Deal-92D97D (59728118877), company C-E23238
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: HR admin
  Personas missing: economic buyer, champion, IT security, finance
  On-file unengaged contact matching a missing persona: none on file

Deal-50D386 (61055128146), company C-EB10E4
  Amount: not on file
  Stage: not on file
  Active contacts: 2
  Personas present: champion, HR admin
  Personas missing: economic buyer, IT security, finance
  On-file unengaged contact matching a missing persona: CT-A1C4B3, Chief People Officer, economic buyer (fits missing persona economic buyer); fit to stage-based pick not confirmable (stage not on file)

Deal-D0D6B5 (60081655042), company C-32918E
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: CT-1FA4DB, Chief People Officer, economic buyer (fits missing persona economic buyer); fit to stage-based pick not confirmable (stage not on file)

Deal-5BFE3B (51674270311), company C-535D36
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: none on file

Deal-36C33F (63739413805), company C-077A0E
  Amount: not on file
  Stage: not on file
  Active contacts: 0
  Personas present: none
  Personas missing: economic buyer, champion, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: CT-1DB73E, Chief People Officer, economic buyer (fits missing persona economic buyer); fit to stage-based pick not confirmable (stage not on file)

Deal-885F45 (60686135564), company C-5E8EFB
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: economic buyer
  Personas missing: champion, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: CT-B3F25D, IT Security Lead, IT security (fits missing persona IT security); fit to stage-based pick not confirmable (stage not on file)

Deal-FCBE5B (62639586615), company C-737030
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: none on file

Deal-5408B0 (60182332309), company C-2AE3AA
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: CT-07FA76, Chief People Officer, economic buyer (fits missing persona economic buyer); fit to stage-based pick not confirmable (stage not on file)

Deal-C6D97A (62121783047), company C-5A8FC2
  Amount: not on file
  Stage: not on file
  Active contacts: 2
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: none on file

Deal-F9A08A (49757401138), company C-0D15DF
  Amount: not on file
  Stage: not on file
  Active contacts: 1
  Personas present: champion
  Personas missing: economic buyer, HR admin, IT security, finance
  On-file unengaged contact matching a missing persona: CT-697541, Chief People Officer, economic buyer (fits missing persona economic buyer); fit to stage-based pick not confirmable (stage not on file)

Not flagged (do not meet the rule): Deal-84DBA6 (63929535929, C-BE9BE0) — 3 active contacts across 3 personas; Deal-4B0BEB (61038797752, C-1BDFE0) — 3 active contacts across 3 personas; Deal-D348E1 (61750885954, C-804F29) — 4 active contacts across 4 personas.
talk-track-from-transcriptsPASS 1.00
Plain-English question: From this rep's last 10 calls: how do they open, handle objections, and how often do they lock a next step?
Prompt — what the model saw
From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0042 · 46s · in 6,308 / out 5,452 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

Plain-English answer: 8 of 10 calls open with the same customer-win story; objection handling, next-step agreement rate, and every competitor raised, with quotes.
The answer in one line: n_transcripts = 10; objection_count = 3; rep = Alex Franklin (+3 more below)
n_transcripts10
objection_count3
allowed_new13 items — e.g. TT-001, TT-002, TT-003, TT-004
competitor_quotesAwardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
competitors_prospect_raisedAwardco, Kudos
next_stepagreed=7; total=10; rate=0.7
objectionsobjection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
opener_themetheme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
repAlex Franklin
rep_raised_competitors_excludedWorkhuman
graded checksobjection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
LEAD (first 5 minutes)
8 of 10 calls (TT-001, TT-002, TT-003, TT-005, TT-006, TT-007, TT-008, TT-010) open with the same customer story:

"Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."

The other two open differently, unquoted: TT-004 leads with an agenda (security review, then pricing) and TT-009 leads with straight pricing.

THREE MOST COMMON OBJECTIONS AND HANDLING
Every call contains exactly one of these three objections, all raised by the prospect:

1. Budget locked until next fiscal year — 4 calls (TT-001, TT-003, TT-006, TT-010). Alex's response, verbatim in all four:

"Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."

2. Revisit next quarter / open enrollment in October — 3 calls (TT-002, TT-005, TT-008). Response, verbatim in all three:

"Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"

3. Already do recognition with a spreadsheet and quarterly gift cards — why change? — 3 calls (TT-004, TT-007, TT-009). Response, verbatim in all three:

"Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

CONCRETE NEXT STEP AGREED RATE
Alex asks "Should we lock the next step — a working session with your team this week?" in 8 of 10 calls. He does not ask in TT-004 (ends with "Understood — I'll leave it with you.") or TT-010 (ends with "Understood, thanks for the candor.").

Agreed in 7 of 10 calls = 70% (7 of 8 asks = 87.5% when asked).

Accepted next step, verbatim in all seven (TT-001, TT-002, TT-003, TT-005, TT-006, TT-008, TT-009):

"Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager."

Not agreed: TT-004 (budget committee), TT-007 ("I need to think about it — there's no urgency on our side." / "Fair enough."), TT-010 ("We'll have to wait for the committee — I can't commit to anything today.").

COMPETITORS RAISED BY PROSPECTS
- Awardco — TT-003: "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
- Kudos — TT-007: "How are you different from Kudos? Our CEO used them at her last company."

Workhuman is mentioned by Alex (TT-005, minute 2), not raised by a prospect.

COACHING NOTES
1. Next-step discipline: Alex asks for a next step in only 8 of 10 calls, and when prospects cite a committee, budget cycle, or no urgency (TT-004, TT-007, TT-010), he accepts a soft close with no fallback. Coach him to always propose a specific, low-friction next step — send materials to the committee, book a 15-minute follow-up — before ending the call.
2. Objection responses are verbatim repeats: the same three replies are used identically every time, regardless of the prospect's context. Coach him to tailor the answer to the stated constraint and probe the real blocker behind "budget is locked" or "no urgency" instead of recycling the same line.
reporting-analytics 5 tests
q3-weighted-forecast0.80
Plain-English question: Build the Q3 2026 weighted forecast from the open pipeline, and show the math deal by deal.
Prompt — what the model saw
Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0093 · 104s · in 25,026 / out 16,911 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✗ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✗ flags_missing_why_buys

✓ Correct Answer (ground truth)

Plain-English answer: A stage-weighted Q3 forecast number computed from the open pipeline, with per-stage math that reconciles to the deal data.
The answer in one line: excluded_amount = 227575; best_case_total = 203565; weighted_forecast = 115,976.75; commit_total = 44729; excluded_out_of_quarter = 32; best_case_count = 24; flags = blank owner column + empty why_buys fields must be surfaced (+2 more below)
excluded_amount227,575
best_case_count24
best_case_total203,565
commit_count7
commit_total44,729
excluded_out_of_quarter32
weighted_forecast115,976.75
flagsblank owner column + empty why_buys fields must be surfaced
top5_best_caseDeal-2D7423, Deal-25F752, Deal-E53952, Deal-5EED42, Deal-FA32A0
graded checkscommit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
COMMIT inside Q3 (7 deals, close date 2026-07-01 to 2026-09-30):
Deal-547B2B 11,200 (2026-09-11)
Deal-B7EBD1 9,000 (2026-09-10)
Deal-403845 9,000 (2026-09-11)
Deal-A2B47C 6,360 (2026-09-11)
Deal-2465CE 5,400 (2026-09-10)
Deal-A5E80A 2,520 (2026-09-11)
Deal-499BF6 1,249 (2026-09-30)
COMMIT total = 11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = 44,729

BEST_CASE inside Q3 (24 deals):
Deal-2D7423 38,935; Deal-25F752 24,000; Deal-E53952 19,656; Deal-5EED42 16,250; Deal-FA32A0 11,116; Deal-FC22A3 10,800; Deal-944310 10,500; Deal-5195DB 9,890; Deal-180D02 9,720; Deal-3974EB 9,000; Deal-5D8CEE 7,200; Deal-9D0060 3,840; Deal-46988D 3,780; Deal-357C30 3,600; Deal-C6D97A 3,240; Deal-DAF1D9 3,150; Deal-EE195F 3,120; Deal-55164C 3,060; Deal-001FF4 2,916; Deal-7B3B0F 2,760; Deal-F9A08A 2,484; Deal-8952F0 2,100; Deal-1FC049 1,920; Deal-87412C 528
BEST_CASE total = 203,565

Weighted forecast = 100% COMMIT + 35% BEST_CASE = 44,729 + (0.35 × 203,565) = 44,729 + 71,247.75 = 115,976.75

Deal counts inside the quarter: COMMIT 7, BEST_CASE 24, PIPELINE 24 (weighted at 0).

Excluded for being outside Q3 (close date after 2026-09-30): 31 deals, total amount 221,875.
Deal-E51FB7 43,875 (2026-10-01); Deal-B936FE 18,000 (2026-10-09); Deal-D9A12F 17,000 (2026-10-15); Deal-D348E1 13,770 (2026-10-15); Deal-4062CF 10,800 (2026-10-15); Deal-293AF3 9,000 (2026-10-09); Deal-034D49 9,000 (2026-10-15); Deal-E0ADD8 7,920 (2026-10-15); Deal-9F2E43 7,690 (2026-10-08); Deal-FCBE5B 7,500 (2026-10-07); Deal-712010 7,200 (2026-10-15); Deal-C61CF7 5,400 (2026-10-09); Deal-600CD9 5,400 (2026-10-02); Deal-A92065 5,400 (2026-10-15); Deal-1D532E 5,400 (2026-10-15); Deal-48B656 5,160 (2026-10-15); Deal-E531A6 4,800 (2026-10-15); Deal-D1E6C2 4,400 (2026-10-09); Deal-D9E112 4,300 (2026-10-09); Deal-5AD94B 4,000 (2026-10-15); Deal-901332 3,600 (2026-10-15); Deal-47AE31 3,600 (2026-10-09); Deal-15D24F 3,600 (2026-10-09); Deal-766C74 3,300 (2026-10-14); Deal-ED725A 2,400 (2026-10-08); Deal-8AD4A5 1,800 (2026-10-07); Deal-D7E999 1,800 (2026-10-15); Deal-ED13B0 1,680 (2026-10-09); Deal-5FDCE4 1,600 (2026-10-01); Deal-7FA0C3 1,400 (2026-10-01); Deal-F5A622 1,080 (2026-10-08).

Top 5 BEST_CASE deals inside Q3 by amount:
1. Deal-2D7423 — 38,935 (2026-09-30)
2. Deal-25F752 — 24,000 (2026-09-25)
3. Deal-E53952 — 19,656 (2026-09-30)
4. Deal-5EED42 — 16,250 (2026-09-30)
5. Deal-FA32A0 — 11,116 (2026-09-25)

## Data quality
forecast_category is not consistent with stage: COMMIT appears at DS1 (Deal-A5E80A, 2,520) and DS2 (Deal-499BF6, 1,249), PIPELINE appears at DS4/DS5, and BEST_CASE appears at DS5 (Deal-C61CF7, 5,400), so an unattended run trusting category over stage misweights the rollup.
Owner is blank on every row except Deal-C9C286 (Bryce Harmon), so forecast ownership and rep-level attribution cannot be verified.
Amount precision is inconsistent: Deal-42326B is 2,480.4 while every other amount is a whole dollar.
Three large deals close on 2026-10-01, one day after quarter end (Deal-E51FB7 43,875; Deal-5FDCE4 1,600; Deal-7FA0C3 1,400), so an off-by-one or timezone date error would swing the forecast materially.
aha-moment-2x2PASS 1.00
Plain-English question: Do companies that hit 5+ givers and one redemption in month one actually retain better at 24 months?
Prompt — what the model saw
Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0359 · 195s · in 109,839 / out 29,550 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

Plain-English answer: The four cohort cells are graded against computed 24-month retention; the numbers come from the scorer's cohort table, not opinion.
The answer in one line: both_5plus_givers_and_redemption = 66.0% (31 of 47)
both_5plus_givers_and_redemption66.0% (31 of 47)
exclusionsnone — claim bounded as correlation, not causation
givers_only46.9% (23 of 49)
largest_single_signal_liftunique givers
neither40.0% (38 of 95)
redemption_only31.0% (9 of 29)
graded checksboth_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
Activation hypothesis test — 220 companies, all 25+ months old; retained ⇔ current_status = 'active'. No missing m1_users/m1_redemptions/current_status values, so **0 companies excluded from the denominator** (denominator = all 220).

### 2x2 cohort table (24-month retention)

| Signal combination | Cohort n | Retained (active) | 24-month retention rate | Arithmetic |
|---|---|---|---|---|
| Both signals (m1_users ≥ 5 AND m1_redemptions ≥ 1) | 47 | 31 | **66.0%** | 31/47 |
| Givers only (m1_users ≥ 5, m1_redemptions = 0) | 49 | 23 | **46.9%** | 23/49 |
| Redemption only (m1_users < 5, m1_redemptions ≥ 1) | 29 | 9 | **31.0%** | 9/29 |
| Neither (m1_users < 5, m1_redemptions = 0) | 95 | 38 | **40.0%** | 38/95 |
| **Total** | 220 | 101 | 45.9% | 101/220 |

Check: 47+49+29+95 = 220; 31+23+9+38 = 101.

### Single signal with the largest retention lift (vs. neither = 40.0%)
- **Givers signal**: givers-only 46.9% − neither 40.0% = **+6.9 pp** (marginal givers lift vs. no-givers: 56.2% − 37.9% = +18.3 pp)
- **Redemption signal**: redemption-only 31.0% − neither 40.0% = **−9.0 pp** (marginal redemption lift vs. no-redemption: 52.6% − 42.4% = +10.3 pp, but that marginal is driven by the "both" cell)

**The givers signal (m1_users ≥ 5) has the largest single-signal retention lift (+6.9 pp over neither).** Redemption-only actually retains *worse* than neither.

### Hypothesis check
Both-signal retention (66.0%) vs. one-signal-or-neither (40.5% = 70/173): lift = **+25.5 pp**. Directionally consistent with the hypothesis in this extract.

### What this does and does not prove
**Does:** In this extract only, companies with both activation signals in month 1 show higher 24-month retention (66.0%) than any single-signal or neither group, and the givers signal carries a positive lift while redemption-only does not.
**Does not prove:** causation — this is a descriptive, observational association on one cohort; no controls for company size, tier, industry, sales touch, or product fit. Small cells (redemption-only n=29) make rates noisy. It does not establish that driving more givers/redemptions will raise retention, and it does not generalize beyond this extract. Retention is proxied by current_status='active' at 25+ months (non_renewing/cancelled counted as not retained).

Cell member aliases (exact): both = C-0D284E42, C-0BC46AD9, C-0F6C0F34, C-0B5A9593, C-0EFFAC85, C-0B05971D, C-0B0F1BAB, C-0D3B97B1, C-0AAA9434, C-0FF675C1, C-0B6AA651, C-101B8AF6, C-0D035262, C-0DB03492, C-0FA7E799, C-0B0D95EA, C-0B540A52, C-0EE91FED, C-0BC34584, C-10B5B236, C-0E1464F7, C-0EE1203E, C-0BA71F12, C-0E508AB1, C-10B2C38C, C-0F9F8F13, C-0D0B047C, C-0BA0465E, C-0F4910FE, C-0B817AC4, C-0D2AB865, C-0BBE3E60, C-0D70AF00, C-0BBC4E7A, C-0B20DB64, C-0D890324, C-0B8A7AEE, C-0CA21961, C-0ECCDC0A, C-0D23E154, C-0BE96399, C-0E8B0E96, C-0D66DF9E, C-1C50207B, C-14D70CE0, C-22170CA1, C-21FEBCBB; givers-only = C-0DD506E9, C-0BF20542, C-0D96A643, C-0FB0D628, C-0A6545BC, C-0DDFCE99, C-0FB16288, C-0BDCF1B6, C-0B38F7E7, C-0F60A6D7, C-0B511454, C-0DDFC9A7, C-0FA50C82, C-0B4963B2, C-0E3DC060, C-0D71342B, C-0D8E448A, C-0A536725, C-0F2B5144, C-0B843542, C-0D9FEB16, C-0F86EEA9, C-0B369871, C-0CEF69FD, C-0BCDB8C2, C-0DAABE31, C-0FC7A215, C-0F5E249E, C-0B7A7546, C-0B08F89E, C-0F6FAB3A, C-0B2895EF, C-0B01DD42, C-0FBB4C18, C-0DE935C4, C-0CBDB8E1, C-0B061E0B, C-0FE32233, C-0F79B789, C-0DE4B242, C-0FCCD2DF, C-0FE88CBD, C-0DB48281, C-0B87D748, C-0F7269D7, C-172EEFBC, C-21629AA4, C-0C8323BF, C-1736DD56; redemption-only = C-0A96134F, C-0F4F825B, C-0FC4DBB8, C-106654D1, C-0DD0626C, C-0B360C78, C-0B81917C, C-0B8435BE, C-0B2213A9, C-0B076F3C, C-0BDB9750, C-0B7D2C30, C-0FD34543, C-0B057828, C-0B540E02, C-0DF83066, C-0B6FD7C0, C-0D78BCB2, C-0FA8A938, C-108C5CE4, C-0FEB5AAF, C-0B807E07, C-0B144C78, C-0BEAF685, C-0F619016, C-F0A562FF, C-0F58EBD7, C-0D1104A2, C-0BCFBE8E; neither = the remaining 95 (C-0B2078FB, C-0B573D81, C-0F4F2729, C-0DC4FB8C, C-0D2ABDB0, C-0DC99C39, C-0DB874B2, C-0F876796, C-0D2C15BB, C-0BE0D5AA, C-0D09BA49, C-0CFA551D, C-0B827671, C-0BB4016D, C-0FF2DB67, C-0B5D1A7C, C-0E665A51, C-0C3848D2, C-0F5D2323, C-0E0D726D, C-0B291CD9, C-0BE48054, C-0DDC0001, C-0FAE9868, C-0D99527C, C-0D5BBE3A, C-0B026228, C-0D23D644, C-0F50DC91, C-0BD1AF2B, C-0BF2B542, C-0B165ED9, C-0F018E41, C-0B89AD2D, C-0FB9D5AF, C-0B4752CB, C-0FD7F358, C-0CAB6FDC, C-0B37FC52, C-0D7C9208, C-0CB2C1B4, C-0FFDBFCF, C-0B9621D0, C-0B18EEB7, C-0BDA785D, C-0BEAC6F3, C-0F933AFD, C-8C2E8F00, C-0D7F2627, C-0DCBE45C, C-0D4A0845, C-0D58F07F, C-0D6CC8E3, C-0D2A005B, C-0D2D5156, C-0D3278C7, C-0D8DE7E7, C-0FD551AB, C-0D9CA315, C-0F6694C3, C-0DD6727F, C-0D2B05C6, C-0EF78601, C-0B5A9287, C-0A7A6CEF, C-0F315622, C-0A092930, C-0B0C92C9, C-0ACCDBC3, C-0DDF0E3E, C-0E9C27D1, C-0DA98AC7, C-0B344485, C-0DE88C76, C-0BC71BDD, C-0DC317F3, C-0B1C1710, C-0D79BB9A, C-0EC6999D, C-10A56B0F, C-928E0B45, C-328E08B5, C-0D699CCF, C-0FF330FE, C-0DD49168, C-21F73C07, C-0CDCEDF6, C-0C117F54, C-0C94A701, C-1918D7DD, C-14264ABD, C-0CB05BE8, C-0C8D169D, C-0DF056D8, C-1794A52C).

⚠️ File-mutation verifier: 1 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/cohort.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/cohort.csv`: `/Users/amaniphipps/.hermes/cache/scratch/cohort.csv` exists but this task has not seen its full current c…
arr-rollforward-reconciliation0.60
Plain-English question: Does CRM ARR match billing ARR — and if not, exactly where is the variance?
Prompt — what the model saw
Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0131 · 137s · in 28,008 / out 18,748 tok
✓ billing_total✓ crm_total✗ variance✗ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

Plain-English answer: Billing $604,739.28 vs CRM $603,581.76: a $1,157.52 variance decomposed into status-mismatch, rounding, and missing-record buckets.
The answer in one line: n_companies = 40; n_mismatched_accounts = 7; as_of = 2026-09-05 (+4 more below)
n_companies40
n_mismatched_accounts7
as_of2026-09-05
billing_arr_total604739.28
bucket_sum_equals_variance1
buckets{"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
crm_arr_total603581.76
ruleterm_months != 12 requires cf_agreement_end_date
variance1157.52
variance_definitionbilling_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
violationssubscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
graded checksbilling_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
Reconciliation as of 2026-09-05. Rule applied exactly as stated: Billing ARR = MRR × 12 over all 42 Chargebee subscriptions (no status exclusion specified). Owner data is not present in either file, so suggested owners are roles to confirm, not named individuals.

## Totals

- Billing ARR (42 subs, MRR × 12): $617,897.76
- CRM HubSpot ARR (39 company records): $603,581.76
- Variance (billing − CRM): +$14,316.00 (billing higher)

## Bucket decomposition (sums exactly to variance)

| Bucket | Amount | Accounts |
|---|---|---|
| Status mismatch | $0.00 | C-0C8323BF (billing cancelled, CRM active, $4,905.24), C-0DC4FB8C (billing cancelled, CRM active, $8,253.24). Amounts agree, so $0 variance impact. |
| Rounding | $0.00 | None. Every MRR × 12 is exact to the cent; no rounding artifact exists. |
| Missing records | +$11,952.00 | C-21629AA4 +$28,449.24 (billing SUB-0004, no CRM record); C-0D5BBE3A −$16,497.24 (CRM record, no billing subscription) |
| Other | +$2,364.00 | C-0F7269D7 +$2,400.00 (billing $26,796.00 vs CRM $24,396.00); C-0D66DF9E −$16.00 (billing $23,184.00 vs CRM $23,200.00); C-14D70CE0 −$20.00 (billing $18,180.00 vs CRM $18,200.00) |
| **Total** | **+$14,316.00** | |

Check: 0.00 + 0.00 + 11,952.00 + 2,364.00 = 14,316.00 = variance. ✓

## Mismatched accounts with suggested owner

- C-0C8323BF — status mismatch (SUB-0000E cancelled in billing, active in CRM; $4,905.24). Suggested owner: account CSM / renewal owner (not in source files — confirm in CRM).
- C-0DC4FB8C — status mismatch (SUB-000F cancelled in billing, active in CRM; $8,253.24). Suggested owner: account CSM / renewal owner.
- C-21629AA4 — missing in CRM (SUB-0004, billing ARR $28,449.24). Suggested owner: RevOps to create CRM company record; account owner to verify.
- C-0D5BBE3A — missing in billing (CRM $16,497.24, no subscription). Suggested owner: billing ops to locate subscription or void CRM record.
- C-0D66DF9E — amount mismatch (SUB-0005: billing $23,184.00 vs CRM $23,200.00, −$16.00). Suggested owner: account owner + RevOps.
- C-14D70CE0 — amount mismatch (SUB-0008: billing $18,180.00 vs CRM $18,200.00, −$20.00). Suggested owner: account owner + RevOps.
- C-0F7269D7 — amount mismatch (SUB-0006: billing $26,796.00 vs CRM $24,396.00, +$2,400.00). Suggested owner: account owner + RevOps.

## Business rule violations (term ≠ 12 months with empty cf_agreement_end_date)

- SUB-0002, C-1794A52C — term 24 months, end date empty
- SUB-0019, C-22170CA1 — term 36 months, end date empty

Compliant non-12-month terms (end date populated): SUB-000C C-0DB48281 (24 mo, 2027-11-30); SUB-001A C-0FC4DBB8 (36 mo, 2027-11-30).
kvm-month-over-month0.80
Plain-English question: How did the four core KVMs move in August versus July?
Prompt — what the model saw
Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0065 · 33s · in 16,067 / out 4,639 tok
✗ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

Plain-English answer: Giving rate 60.27% (up 0.07%), redemptions per user 1.7302 (up 0.01%), with 1:1 meetings and pulse engagement each up fractionally — direction reported for all four.
The answer in one line: n_company_rows_per_month = 30; aggregation = simple mean of company-level values per month
n_company_rows_per_month30
aggregationsimple mean of company-level values per month
biggest_moverpulse_engagement
driving_segmentsize_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
kvms{"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
segment_detail{"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
graded checksfour_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
KVMs are simple averages across the 30 company_alias rows in kvm_monthly.csv (all tier_three).

| KVM | 2026-08 | 2026-07 | Abs change | Rel change | Direction |
|---|---|---|---|---|---|
| giving_rate | 0.6027 | 0.6023 | +0.0004 | +0.07% | up |
| redemptions_per_user | 1.7302 | 1.7300 | +0.0002 | +0.01% | up |
| one_to_one_engagement | 0.4472 | 0.4469 | +0.0003 | +0.06% | up |
| pulse_engagement | 0.5086 | 0.6006 | −0.0920 | −15.31% | down |

Largest relative move: pulse_engagement, −15.31%. Segment driving it: size_band = enterprise (0.5500 → 0.2743, −50.13%); mid_market (+0.21%) and smb (−0.22%) were flat. plan_tier is uniform (tier_three) across all rows, so it cannot explain the move.
rewards-ytd-sectionPASS 1.00
Plain-English question: Produce the redemption section of the weekly rewards report through August.
Prompt — what the model saw
Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0517 · 291s · in 108,772 / out 48,299 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

Plain-English answer: 378 redemptions, $27,846 spend, 235 unique redeemers at 1.61 each, custom rewards at 39% of spend, US leading all countries.
The answer in one line: redemption_count = 378; unique_redeemers = 235; redemptions_per_redeemer = 1.61; total_spend_usd = 27846.00
redemption_count378
redemptions_per_redeemer1.61
unique_redeemers235
provider_mix_pct_of_spendTangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_countcountry=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
total_spend_usd27846.00
graded checkscount · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Last completed month: 2026-08 (August 2026). Data spans 2026-01-01 to 2026-08-31 (max redeemed_at = 2026-08-31), so all 378 rows in redemptions_ytd.csv are within the period through the last completed month.

Redemption section (through 2026-08):

Redemption count: 378
Spend: $27,846.00
Unique redeemers: 235
Redemptions per redeemer: 378 / 235 = 1.61

Provider mix (percent of spend):
- TangoCard: $3,230.00 = 3,230 / 27,846 × 100 = 11.60% (90 redemptions)
- Tremendous: $8,505.00 = 8,505 / 27,846 × 100 = 30.54% (192 redemptions)
- custom: $10,873.00 = 10,873 / 27,846 × 100 = 39.05% (37 redemptions)
- Snappy: $5,238.00 = 5,238 / 27,846 × 100 = 18.81% (59 redemptions)
- Total: $27,846.00 = 100.00%

Top 5 countries by redemptions:
1. US: 244
2. CA: 24
3. AU: 21
4. GB: 17
5. NL: 17

(GB and NL tie at 17; the next country, SG, has 12, so the top 5 is unambiguous.)

⚠️ File-mutation verifier: 1 file edit(s) FAILED this turn despite any wording above that may suggest otherwise. Run `git status` or `read_file` to confirm what actually landed.
  • `/Users/amaniphipps/.hermes/cache/scratch/redemptions_ytd.csv` — [write_file] Refusing to overwrite `/Users/amaniphipps/.hermes/cache/scratch/redemptions_ytd.csv`: `/Users/amaniphipps/.hermes/cache/scratch/redemptions_ytd.csv` exists but this task has not seen …
customer-success 5 tests
churn-save-eligibilityPASS 1.00
Plain-English question: Which at-risk accounts actually qualify for a churn-save offer, and which play fits each?
Prompt — what the model saw
Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0118 · 129s · in 21,579 / out 16,530 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

Plain-English answer: Only accounts meeting all three rules qualify — e.g. C-0F6C0F34 at $49,707 (executive touch) and C-0B827671 at $25,365 (usage revival) — with amount and play per account.
The answer in one line: n_accounts = 30; snapshot_date = 2026-09-05
n_accounts30
eligibleaccount_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
non_eligible_at_riskaccount_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
ruleshealth_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
snapshot_date2026-09-05
total_amount_at_stake224601.00
graded checkseligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Eligibility applied: R1 health_score < 60, R2 churn_save_eligible_amount > 0, R3 renewal_date within 120 days of snapshot 2026-09-05 (cutoff 2027-01-03). The rules file defines eligibility only; it does not define plays, so I mapped plays to the signals actually in the data: usage revival = usage_trend_3m declining; executive touch = champion_active false; commercial concession = champion active and usage not declining (the funded eligible amount is the concession budget). All seat-utilization percentages are computed from seats/seats_used.

QUALIFYING FOR CHURN-SAVE OFFER (all of R1, R2, R3 pass)

C-0F6C0F34 | health 51 | renewal 2026-10-03 (28 days out) | amount at stake $49,707 | play: executive touch | signal: champion_active=false (seat utilization 308/395 = 78%); usage is growing, so revival is not the fit
C-0B827671 | health 56 | renewal 2026-11-14 (70 days out) | $25,365 | play: usage revival | signal: usage_trend_3m=declining (utilization 113/202 = 56%); champion is active
C-0B360C78 | health 57 | renewal 2026-10-28 (53 days out) | $35,748 | play: commercial concession | signal: champion_active=true, usage growing (no revival or exec-touch signal); funded eligible amount $35,748
C-0B0F1BAB | health 38 | renewal 2026-09-23 (18 days out) | $5,494 | play: executive touch | signal: champion_active=false; lowest health score in the file (38); most imminent renewal
C-0CA21961 | health 58 | renewal 2026-12-28 (114 days out) | $16,829 | play: commercial concession | signal: champion_active=true, usage flat (not declining); utilization 84/325 = 26%
C-0E9C27D1 | health 39 | renewal 2026-09-24 (19 days out) | $41,235 | play: commercial concession | signal: champion_active=true, usage flat; health 39 (second lowest in file)
C-0CEF69FD | health 53 | renewal 2026-11-21 (77 days out) | $32,621 | play: executive touch | signal: champion_active=false (utilization 97/136 = 71%); usage is growing
C-0D3278C7 | health 54 | renewal 2026-11-12 (68 days out) | $17,602 | play: usage revival | signal: usage_trend_3m=declining (utilization 126/380 = 33%); champion is active

Total amount at stake: $224,601
Arithmetic: 49,707 + 25,365 + 35,748 + 5,494 + 16,829 + 41,235 + 32,621 + 17,602 = 224,601

Play distribution: executive touch x3 (C-0F6C0F34, C-0B0F1BAB, C-0CEF69FD), commercial concession x3 (C-0B360C78, C-0CA21961, C-0E9C27D1), usage revival x2 (C-0B827671, C-0D3278C7). No qualifier has a missing/invalid field.

AT RISK (health_score < 60) BUT NOT QUALIFYING

C-0BC71BDD | health 55 | eligible $0 | renewal 2026-10-27 (52 days, passes R3) | fails R2 only: churn_save_eligible_amount = 0, not > 0
C-0BA71F12 | health 52 | eligible $6,824 | renewal 2027-04-11 (218 days out) | fails R3 only: renewal outside the 120-day window (passes R1, R2)
C-0F6694C3 | health 43 | eligible $0 | renewal 2027-03-21 (197 days out) | fails R2 ($0 not > 0) and R3 (197 days > 120)
C-0BE96399 | health 54 | eligible $0 | renewal 2026-10-29 (54 days, passes R3) | fails R2 only: churn_save_eligible_amount = 0
C-0F876796 | health 47 | eligible $19,958 | renewal 2027-02-06 (154 days out) | fails R3 only: 154 days out > 120 (passes R1, R2)
C-0FCCD2DF | health 43 | eligible $0 | renewal 2027-04-23 (230 days out) | fails R2 ($0 not > 0) and R3 (230 days > 120)
C-10A56B0F | health 54 | eligible $0 | renewal 2026-12-12 (98 days, passes R3) | fails R2 only: churn_save_eligible_amount = 0

All remaining 15 accounts have health_score >= 60, so they are not at risk under R1 and were not evaluated further.
expansion-kitPASS 1.00
Plain-English question: Build the expansion kit for the customer who just replied to our outreach.
Prompt — what the model saw
A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0028 · 25s · in 3,539 / out 2,622 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

Plain-English answer: 37.5% seat coverage, 250 headroom seats worth $15,000 ARR at $60 a seat, usage up 43.2% in six months — and the replier cannot buy; Dana R. (VP People) is the right buyer.
The answer in one line: headroom_seats = 250; coverage_pct = 37.5; company_alias = C-0DDFC9A7 (+3 more below)
headroom_seats250
coverage_pct37.5
company_aliasC-0DDFC9A7
headroom_arr15000.00
per_seat_annual_rate60.00
replierMaria S.
replier_can_buy0
right_buyername=Dana R.; title=VP People
usage_factsactive_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84
usage_health_linesMonthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checkscoverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
SEAT COVERAGE
150 licensed seats ÷ 400 headcount = 37.5% coverage.

USAGE HEALTH
Monthly active users rose 88 → 126 from March to August 2026 (+38 users, ~7.6/month), growing every month in that window. 126 of 150 licensed seats are active (84% utilization). Maria also confirms the recognition feed is busy weekly.

HEADROOM (expansion to full headcount at current per-seat rate)
Per-seat rate: $9,000 ARR ÷ 150 seats = $60/seat/year.
Seat headroom: 400 − 150 = 250 seats.
ARR headroom: 250 × $60 = $15,000/year.
(24 of 150 licensed seats are unused today.)

WHO REPLIED / CAN THEY BUY
Maria S., People Operations Coordinator (replied 2026-09-02). No — she states budget and seat expansion sit with Dana R.

RIGHT BUYER
Dana R., VP People — last engaged 2026-05-18. Maria names her as the budget owner for seat expansion and says she has been asking about usage numbers.

REPLY EMAIL
Subject: Re: Growing your team's recognition program

Hi Maria,

Thanks for the note — glad the team loves Bonusly and that the feed stays busy weekly. Our data backs your read: MAU climbed from 88 in March to 126 in August.

You're right that seat expansion sits with Dana R., VP People. I'll reach out to Dana directly about usage and room to grow — no forward needed from you.

Best,
Cole Ingram
onboarding-statusPASS 1.00
Plain-English question: What is done and what is missing in this customer's onboarding before the mid-onboarding call?
Prompt — what the model saw
Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0026 · 21s · in 3,617 / out 2,100 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

Plain-English answer: 25 days live: Slack connected, allowance set, admins added — but the HRIS integration is not connected, with early engagement signals and three call priorities.
The answer in one line: days_live = 25; company_alias = C-0D284E42
days_live25
call_itemsConnect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
checklist{"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
company_aliasC-0D284E42
engagement_signalsavg_active_givers_week1=4.3; avg_active_givers_last_week=13; latest_active_givers=15; days_with_activity=25
graded checkscomplete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
Account: C-0D284E42 (signup 2026-08-11)

**Complete — data field present**
- Slack integration connected: 2026-08-12 (integration_slack populated)
- Allowance set: 2026-08-13 (allowance_set populated)
- Admins added: 2 (admins_added = 2)
- First recognition given: 2026-08-15 14:22 (first_recognition_at populated; 4 days after signup)

**Not complete — field empty**
- HRIS integration: not connected (integration_hris empty; Slack is the only integration)
- First redemption: not given (first_redemption_at empty)

**Early engagement signals (onboarding_usage.csv, 2026-08-11 → 2026-09-04, 25 days)**
- active_givers: 3 on signup day, peak 15 on 2026-09-03 and 2026-09-04
- 25-day sum = 223 active-giver days; daily average = 223 / 25 ≈ 8.92
- First-week average (Aug 11–17): 30 / 7 ≈ 4.29; last-seven-day average (Aug 29–Sep 4): 91 / 7 = 13.0 — roughly 3x
- Minor dips amid the rise: Aug 16 (5→4), Aug 18 (7→5), Aug 20 (7→6), Aug 22 (9→8)

**Three things to cover on the call**
1. Connect the HRIS integration — only Slack is connected; integration_hris has no date.
2. Drive the first redemption — allowance has been set since 2026-08-13, but first_redemption_at is empty; no reward has been redeemed yet.
3. Sustain engagement momentum — active_givers rose from 3 to a peak of 15; confirm what's driving adoption and address the intermittent dips (e.g., Aug 16, Aug 18).
renewal-risk-conflicting-datesPASS 1.00
Plain-English question: Two systems disagree on renewal dates — which do you trust per account, and who is actually renewing inside 90 days?
Prompt — what the model saw
Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0154 · 224s · in 9,829 / out 32,580 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

Plain-English answer: Chargebee wins per account (multi-year contracts are known wrong in ChurnZero), with each trusted date, the reason, and a 90-day-window flag.
The answer in one line: n_accounts = 20; n_disagreements = 5; snapshot_date = 2026-09-05 (+2 more below)
n_accounts20
n_disagreements5
accounts20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
arr_at_risk_90d359409.00
disagreementsaccount_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
snapshot_date2026-09-05
total_arr_renewing_90d1048715.00
trust_rulemulti-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
window2026-09-05 to 2026-12-04
graded checkstotal_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-DAY RENEWAL RISK BRIEF — as of 2026-10-19

Window: renewals due 2026-10-19 through 2027-01-17, plus renewals already past due (open churn risk). All 20 accounts' trusted renewal dates fall between 2026-09-15 and 2026-11-24; 10 are overdue and 10 fall within the next 90 days.

Trust rule: on multi-year contracts (is_multi_year = true) the Chargebee date is used, because ChurnZero renewal dates are known to be wrong on multi-year contracts. On single-year contracts both systems agree, so the common date is used.

DATE DISAGREEMENTS (5 accounts, all multi-year — Chargebee trusted):
1. C-0B7D2C30 — ChurnZero 2026-09-10 vs Chargebee 2026-09-15 (36-month term)
2. C-0BCDB8C2 — ChurnZero 2027-09-18 vs Chargebee 2026-09-18 (36-month term; ChurnZero is +1 year)
3. C-0D2AB865 — ChurnZero 2026-09-10 vs Chargebee 2026-09-22 (24-month term)
4. C-0BBE3E60 — ChurnZero 2027-09-26 vs Chargebee 2026-09-26 (24-month term; ChurnZero is +1 year)
5. C-0F5D2323 — ChurnZero 2026-09-10 vs Chargebee 2026-09-29 (24-month term)
Remaining 15 accounts: both systems show the same date; no discrepancy.

RENEWAL REGISTER — seat utilization = seats_used / seats; trend = avg active users Jun-Aug 2026 vs Mar-May 2026. Risk rule: HIGH = renewal date already overdue as of 2026-10-19, or 3-month usage decline of 3% or more; LOW = not overdue with stable or growing usage.

1. C-0B7D2C30 | Dana Mercer | $65,901 | 2026-09-15 [Chargebee; FLAG vs ChurnZero 2026-09-10] | OVERDUE | Util 57.6% (274/476) | Trend -18.2% declining (91.7 vs 112.0) | RISK: HIGH
   Evidence: The 2026-09-15 renewal date passed with no renewal on record and usage fell about 18% over the last three months.

2. C-0BCDB8C2 | Cole Ingram | $54,427 | 2026-09-18 [Chargebee; FLAG vs ChurnZero 2027-09-18] | OVERDUE | Util 54.7% (232/424) | Trend -17.6% declining (118.3 vs 143.7) | RISK: HIGH
   Evidence: The 2026-09-18 renewal date passed with usage down about 17.6% over the last three months.

3. C-0D2AB865 | Elena Sinclair | $38,022 | 2026-09-22 [Chargebee; FLAG vs ChurnZero 2026-09-10] | OVERDUE | Util 61.4% (250/407) | Trend -18.9% declining (117.0 vs 144.3) | RISK: HIGH
   Evidence: The 2026-09-22 renewal date passed with usage down about 18.9% over the last three months.

4. C-0BBE3E60 | Dana Mercer | $30,993 | 2026-09-26 [Chargebee; FLAG vs ChurnZero 2027-09-26] | OVERDUE | Util 64.9% (74/114) | Trend -19.5% declining (35.7 vs 44.3) | RISK: HIGH
   Evidence: The 2026-09-26 renewal date passed with usage down about 19.5% over the last three months.

5. C-0F5D2323 | Cole Ingram | $90,647 | 2026-09-29 [Chargebee; FLAG vs ChurnZero 2026-09-10] | OVERDUE | Util 28.5% (111/390) | Trend +3.5% growing (19.7 vs 19.0) | RISK: HIGH
   Evidence: The 2026-09-29 renewal date passed and seat utilization is only 28.5% (111 of 390 seats), signaling weak adoption.

6. C-0EC6999D | Elena Sinclair | $79,419 | 2026-10-03 [both systems agree] | OVERDUE | Util 27.7% (31/112) | Trend +6.7% growing (16.0 vs 15.0) | RISK: HIGH
   Evidence: The 2026-10-03 renewal date passed with utilization at just 27.7% (31 of 112 seats) despite flat usage.

7. C-0B20DB64 | Dana Mercer | $21,770 | 2026-10-07 [both systems agree] | OVERDUE | Util 56.6% (214/378) | Trend +0.1% stable (295.3 vs 295.0) | RISK: HIGH
   Evidence: The 2026-10-07 renewal date passed with no renewal on record and usage flat at about 295 active users.

8. C-0BBC4E7A | Cole Ingram | $56,374 | 2026-10-10 [both systems agree] | OVERDUE | Util 67.7% (228/337) | Trend -0.9% stable (140.7 vs 142.0) | RISK: HIGH
   Evidence: The 2026-10-10 renewal date passed; usage is stable at about 141 active users but no renewal is on record.

9. C-0FD551AB | Elena Sinclair | $48,815 | 2026-10-14 [both systems agree] | OVERDUE | Util 55.9% (210/376) | Trend -1.6% stable (123.7 vs 125.7) | RISK: HIGH
   Evidence: The 2026-10-14 renewal date passed with no renewal recorded and usage essentially flat at about 124 active users.

10. C-0F9F8F13 | Dana Mercer | $46,230 | 2026-10-18 [both systems agree] | OVERDUE | Util 56.5% (199/352) | Trend +0.2% stable (184.0 vs 183.7) | RISK: HIGH
    Evidence: The 2026-10-18 renewal date passed one day ago with no renewal on record and flat usage.

11. C-0BC34584 | Cole Ingram | $16,740 | 2026-10-22 [both systems agree] | DUE | Util 66.2% (327/494) | Trend +1.0% stable (104.7 vs 103.7) | RISK: LOW
    Evidence: Renewal on 2026-10-22 with stable usage and 66.2% seat utilization.

12. C-0B7A7546 | Elena Sinclair | $35,062 | 2026-10-25 [both systems agree] | DUE | Util 88.8% (182/205) | Trend +4.3% growing (64.0 vs 61.3) | RISK: LOW
    Evidence: Renewal on 2026-10-25 with usage up about 4.3% and 88.8% seat utilization.

13. C-0B369871 | Dana Mercer | $85,128 | 2026-10-29 [both systems agree] | DUE | Util 75.1% (317/422) | Trend +4.3% growing (329.7 vs 316.0) | RISK: LOW
    Evidence: Renewal on 2026-10-29 with usage up about 4.3% and 75.1% seat utilization.

14. C-0B144C78 | Cole Ingram | $30,899 | 2026-11-02 [both systems agree] | DUE | Util 75.4% (169/224) | Trend +3.0% growing (102.7 vs 99.7) | RISK: LOW
    Evidence: Renewal on 2026-11-02 with usage up about 3.0% and 75.4% seat utilization.

15. C-0FC4DBB8 | Elena Sinclair | $94,732 | 2026-11-05 [both systems agree] | DUE | Util 76.7% (356/464) | Trend +4.2% growing (191.0 vs 183.3) | RISK: LOW
    Evidence: Renewal on 2026-11-05 with usage up about 4.2% and 76.7% seat utilization.

16. C-0D5BBE3A | Dana Mercer | $39,740 | 2026-11-09 [both systems agree] | DUE | Util 83.3% (85/102) | Trend +3.9% growing (89.7 vs 86.3) | RISK: LOW
    Evidence: Renewal on 2026-11-09 with usage up about 3.9% and 83.3% seat utilization.

17. C-0FB9D5AF | Cole Ingram | $63,158 | 2026-11-13 [both systems agree] | DUE | Util 72.4% (144/199) | Trend +4.6% growing (174.0 vs 166.3) | RISK: LOW
    Evidence: Renewal on 2026-11-13 with usage up about 4.6% and 72.4% seat utilization.

18. C-0B344485 | Elena Sinclair | $64,384 | 2026-11-16 [both systems agree] | DUE | Util 78.0% (224/287) | Trend +4.2% growing (240.7 vs 231.0) | RISK: LOW
    Evidence: Renewal on 2026-11-16 with usage up about 4.2% and 78.0% seat utilization.

19. C-0CB2C1B4 | Dana Mercer | $40,628 | 2026-11-20 [both systems agree] | DUE | Util 81.6% (386/473) | Trend -2.0% stable (48.0 vs 49.0) | RISK: LOW
    Evidence: Renewal on 2026-11-20 with usage roughly flat (-2.0%) and 81.6% seat utilization.

20. C-22170CA1 | Cole Ingram | $45,646 | 2026-11-24 [both systems agree] | DUE | Util 85.4% (251/294) | Trend +2.6% stable (145.7 vs 142.0) | RISK: LOW
    Evidence: Renewal on 2026-11-24 with usage up about 2.6% and 85.4% seat utilization.

TOTALS
Total ARR renewing: $1,048,715 across 20 accounts (sum of all 20 ARR values).
ARR at risk (HIGH): $532,598 across the 10 overdue accounts = 50.8% of renewing ARR ($532,598 / $1,048,715).
ARR rated LOW: $516,117 across 10 accounts = 49.2% of renewing ARR.
ticket-theme-synthesisPASS 1.00
Plain-English question: What are the real themes in this quarter's support tickets, ignoring the existing tags?
Prompt — what the model saw
Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0098 · 127s · in 8,193 / out 18,886 tok
✓ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

Plain-English answer: 80 tickets in 4-7 themes — points-not-posting leads (20 tickets, 25%, $31,100 ARR affected) — each with counts, distinct accounts, ARR, and a recommendation.
The answer in one line: total_tickets = 80; quarter_window = 2026-06-01 through 2026-08-31
total_tickets80
noteexisting tags are noisy/unreliable; classify by text
quarter_window2026-06-01 through 2026-08-31
ranking_by_arr_exposurehris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
sample_ticket_ids80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
single_account_noise_themebilling_dispute
themes{"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
graded checkstheme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Quarter support-ticket synthesis: 80 tickets, 5 themes, ranked by ARR affected. Tags ignored; tickets grouped by body text.

1. HRIS provisioning failures (broad pattern)
   Count: 12 tickets. Share: 12/80 = 15.0%.
   Distinct accounts: 3 — C-0B2213A9 (36,000), C-0F6C0F34 (30,000), C-0DDFC9A7 (48,000).
   ARR affected: 36,000 + 30,000 + 48,000 = 114,000.
   Ticket ids: IC-460059, IC-460062.
   Recommendation: Fix the HRIS provisioning connector so new-hire account creation succeeds and sync errors surface in the provisioning log.

2. Checkout, redemption, and gift card failures (broad pattern)
   Count: 18 tickets. Share: 18/80 = 22.5%.
   Distinct accounts: 7 — C-0CEF69FD (8,900), C-0B827671 (10,700), C-0FCCD2DF (9,600), C-0F876796 (8,700), C-0D9CA315 (9,600), C-0B0F1BAB (10,300), C-14264ABD (11,000).
   ARR affected: 8,900 + 10,700 + 9,600 + 8,700 + 9,600 + 10,300 + 11,000 = 68,800.
   Ticket ids: IC-460025, IC-460024.
   Recommendation: Fix the checkout redemption flow and gift card delivery so failed redemptions neither deduct points nor drop codes or emails.

3. Invoice seat-count and tier billing errors (single-account noise)
   Count: 16 tickets. Share: 16/80 = 20.0%.
   Distinct accounts: 1 — C-0E9C27D1 (52,000). All 16 tickets come from this one account; not a broad pattern.
   ARR affected: 52,000.
   Ticket ids: IC-460071, IC-460069.
   Recommendation: Correct C-0E9C27D1's invoices and tier pricing directly, then audit billing tier logic so the error cannot recur.

4. Recognition points not posting or missing from balance (broad pattern)
   Count: 20 tickets. Share: 20/80 = 25.0%.
   Distinct accounts: 9 — C-0D3278C7 (3,500), C-0BE96399 (2,700), C-0DD0626C (2,500), C-0B2895EF (2,900), C-0D284E42 (3,400), C-0BF20542 (4,500), C-0D0B047C (4,500), C-0D6CC8E3 (4,200), C-21FEBCBB (2,900).
   ARR affected: 3,500 + 2,700 + 2,500 + 2,900 + 3,400 + 4,500 + 4,500 + 4,200 + 2,900 = 31,100.
   Ticket ids: IC-460004, IC-460016.
   Recommendation: Fix recognition-to-points delivery so posted recognitions credit balances promptly and reliably.

5. Slack integration sync and auth failures (broad pattern)
   Count: 14 tickets. Share: 14/80 = 17.5%.
   Distinct accounts: 4 — C-0B843542 (4,400), C-0BA71F12 (3,900), C-10A56B0F (5,400), C-8C2E8F00 (5,200).
   ARR affected: 4,400 + 3,900 + 5,400 + 5,200 = 18,900.
   Ticket ids: IC-460041, IC-460047.
   Recommendation: Stabilize the Slack sync toggle and slash command re-auth so recognitions post to channels without disconnecting.

All 80 tickets are accounted for across the 5 themes; no data is missing. The highest-ARR theme (billing, 52,000) is single-account noise from C-0E9C27D1, while the four broad patterns carry 232,800 in combined distinct-account ARR.
marketing 5 tests
case-study-matchPASS 1.00
Plain-English question: Which three existing customers are the best case-study matches for this prospect?
Prompt — what the model saw
Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0030 · 29s · in 3,834 / out 3,127 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

Plain-English answer: Three ranked matches on industry, size, use case, and region (e.g. C-11C31562), with the higher-scoring decoy explicitly excluded for a mismatched field.
The answer in one line: prospect_alias = C-82AF3719
constraintsno_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
decoys_excludedcompany_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
prospect_aliasC-82AF3719
prospect_fields_usedindustry, size_band, use_case, region
top3company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
graded checkstop3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Prospect: C-82AF3719 | Technology | Mid-Market | employee_recognition | NA-West

Eligible set: customers with has_case_study=true. Scored by exact field matches, 1 point per field, equal weight.

1. C-64171065 | Technology | Mid-Market | employee_recognition | NA-East
   Matches: industry ✓, size_band ✓, use_case ✓, region ✗ (NA-East ≠ NA-West) = 3/4
   Fields driving match: industry, size band, use case. Mismatch: region.

2. C-A13C193D | Technology | Mid-Market | retention | NA-West
   Matches: industry ✓, size_band ✓, use_case ✗ (retention ≠ employee_recognition), region ✓ = 3/4
   Fields driving match: industry, size band, region. Mismatch: use case.

3. C-11C31562 | Manufacturing | Mid-Market | employee_recognition | NA-West
   Matches: industry ✗ (Manufacturing ≠ Technology), size_band ✓, use_case ✓, region ✓ = 3/4
   Fields driving match: size band, use case, region. Mismatch: industry.

Tiebreak: all three score 3/4. Ties resolved in the field order given (industry → size_band → use_case → region), favoring the candidate with a match at the first differing field. C-64171065 differs only on region; C-A13C193D matches industry but misses use case; C-11C31562 misses industry. No other case-study customers score above 2/4 (next best: C-CD4829A7 and C-9D69D147 at 2/4).
channel-attribution-mathPASS 1.00
Plain-English question: Compute cost per SQM, cost per SQO, and pipeline per dollar for every channel over six months.
Prompt — what the model saw
Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0083 · 100s · in 7,446 / out 15,291 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

Plain-English answer: LinkedIn Ads at $960/SQM and $3,000/SQO, organic search at $90,000 pipeline on zero spend, paid social flagged at $18,000 with zero SQMs, and a channel winner named.
The answer in one line: best_pipeline_per_dollar_channel = paid_search
best_pipeline_per_dollar_channelpaid_search
confidence_notewebinars and referral have small SQM samples (<20); treat rates as directional.
flagged_rows_sqo_before_sqmCT-000041, CT-000044
months2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel{"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
reallocation_recommendationfrom=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
undefined_channelspaid_social
graded checksundefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Channel performance — trailing 6 months (2026-03 through 2026-08). All contact dates fall inside this window; spend file covers exactly these 6 months.

PAID CHANNELS

paid_search
  Spend: 6,000 x 6 = $36,000
  SQMs: 38   SQOs: 17
  Cost per SQM: 36,000 / 38 = $947.37
  Cost per SQO: 36,000 / 17 = $2,117.65
  SQM-to-SQO rate: 17 / 38 = 44.7%
  Pipeline amount: 17 x $40,000 = $680,000
  Pipeline per dollar: 680,000 / 36,000 = $18.89

linkedin_ads
  Spend: 4,000 x 6 = $24,000
  SQMs: 24   SQOs: 8
  Cost per SQM: 24,000 / 24 = $1,000.00
  Cost per SQO: 24,000 / 8 = $3,000.00
  SQM-to-SQO rate: 8 / 24 = 33.3%
  Pipeline amount: 8 x $12,000 = $96,000
  Pipeline per dollar: 96,000 / 24,000 = $4.00

paid_social
  Spend: 3,000 x 6 = $18,000
  SQMs: 0   SQOs: 0
  Cost per SQM: undefined (spend with zero SQMs)
  Cost per SQO: undefined
  SQM-to-SQO rate: undefined
  Pipeline amount: $0
  Pipeline per dollar: $0.00 (0 / 18,000)

webinars
  Spend: 1,500 x 6 = $9,000
  SQMs: 12   SQOs: 5
  Cost per SQM: 9,000 / 12 = $750.00
  Cost per SQO: 9,000 / 5 = $1,800.00
  SQM-to-SQO rate: 5 / 12 = 41.7%
  Pipeline amount: 5 x $12,000 = $60,000
  Pipeline per dollar: 60,000 / 9,000 = $6.67

ORGANIC CHANNEL

organic_search
  Volume (SQMs): 27   SQOs: 7
  SQO rate: 7 / 27 = 25.9%
  Pipeline: 7 x $9,000 = $63,000
  (No spend row for organic_search, so cost metrics are not computed.)

DATE-ORDER FLAGS (SQO date precedes SQM date)
  CT-000044, linkedin_ads: SQM 2026-07-23, SQO 2026-07-18 (SQO 5 days before SQM)
  CT-000041, linkedin_ads: SQM 2026-06-14, SQO 2026-06-09 (SQO 5 days before SQM)

REALLOCATION RECOMMENDATION
  Stop paid_social: $18,000 spend produced zero SQMs, so every derived metric is undefined.
  Reduce linkedin_ads: weakest paid efficiency ($4.00 pipeline per dollar, $3,000 cost per SQO, 33.3% SQM-to-SQO).
  Shift to paid_search: strongest paid efficiency ($18.89 pipeline per dollar, $2,117.65 cost per SQO, 44.7% conversion) and the largest SQM base (38).
  Hold webinars at current level / scale cautiously: $6.67 pipeline per dollar and $1,800 cost per SQO look strong, but only 12 SQMs / 5 SQOs.

CONFIDENCE (sample sizes)
  paid_search: 38 SQMs / 17 SQOs — moderate confidence; largest paid sample and clear efficiency leader.
  linkedin_ads: 24 SQMs / 8 SQOs — low-to-moderate; direction is clear but the sample is small.
  webinars: 12 SQMs / 5 SQOs — low confidence; do not scale up on this sample.
  paid_social: 0 SQMs on $18,000 — high confidence to cut (zero signal despite spend).

Data scope notes: paid_social is in spend with zero contacts (reported undefined, not zero). organic_search is in contacts with no spend row (volume/rate/pipeline only). Referral contacts exist in the file but were not requested and have no spend row, so they are excluded.
competitor-battlecard-updatePASS 1.00
Plain-English question: Update the Rivally battlecard with sourced pricing and where each side wins.
Prompt — what the model saw
Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0069 · 93s · in 4,688 / out 12,579 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

Plain-English answer: Rivally at $7 per user per month (the August source wins over the stale $5, conflict noted), a 13-7 win/loss record, and two unverified old claims flagged.
The answer in one line: competitor = Rivally
citation_ids_available25 items — e.g. S01, S02, S03, S04
competitorRivally
pricingvalue_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
rep_opinion_snippet_ids_not_factsS09, S21
unverified_old_claimsRivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
win_loss_12mowins=13; losses=7
graded checkscompetitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Battlecard: Rivally (updated)

## One-line positioning
Points-based employee recognition feed for mid-market and distributed EU teams, with multi-language support, EU data residency (generally available), a Dublin office, and a paid Pulse engagement-survey add-on — S02, S04, S12, S15, S06, S23.

## Pricing (source and date)
Current list price: Recognition Starter at $7 per user/month, annual billing required — S17 (2026-08-12).
Conflict: older pricing pages show $5 per user/month, annual billing required — S03 (2026-01-20), S08 (2026-04-01). Newer source (S17) wins; the $5 figure is superseded.
Deal-specific quotes (not list price): $6.50/user/month quoted to a 500-seat prospect, annual term — S13 (2026-06-02); $7/user/month list with a 15% discount for a 3-year term — S18 (2026-08-14).
Pulse add-on is priced separately, not bundled — S23 (2026-09-01).

## Where Rivally wins
Engaging, points-based recognition feed — S02, S16.
Setup under a week; Slack integration works out of the box — S04.
Strong with distributed EU teams; multi-language praised; EU data residency GA; Dublin office — S12, S15 (EU residency also pitched to Prospect C — S05).
Support response under 4 hours — S22.
Microsoft Teams app v2 in public preview — S19.
$40M Series C led by Northgate Ventures — S01.

## Where we win (Rivally gaps)
Limited analytics; reporting dashboards basic vs enterprise tools; analytics exports are CSV-only, making migration off Rivally hard — S02, S07, S20. An 800-seat prospect picked Bonusly over Rivally citing analytics depth — S25.
No SCIM provisioning; manual user management is painful — S10.
Admin console lacks bulk recognition editing — S24.
EMEA rewards catalog thinner than the US catalog — S14.
Admin tooling lags peers — S16.

## Objections and responses
Objection: "Rivally's recognition feed is engaging and points-based." — S02, S16.
Response: Their analytics are limited and dashboards are basic vs enterprise tools; CSV-only exports make migration hard — S02, S07, S20.
Objection: "Rivally sets up in under a week with Slack out of the box." — S04.
Response: No SCIM (manual user management is painful) and no bulk recognition editing in the admin console — S10, S24.
Objection: "Rivally offers EU data residency and multi-language support for distributed EU teams." — S05, S12, S15.
Response: Their EMEA rewards catalog is thinner than the US catalog — S14. Bonusly EMEA catalog data not provided.
Support-SLA objection (S22): response data not in provided files.

## Recent changes
2026-09-01: Rivally Pulse exits beta; priced as an add-on, not bundled — S23.
2026-08-20: Microsoft Teams app v2 enters public preview — S19.
2026-08-12: Recognition Starter price raised to $7/user/month, annual billing required (was $5 on S03/S08) — S17.
2026-07-01: Dublin office opened; EU data residency generally available — S15.
2026-06-02: Quoted $6.50/user/month to a 500-seat prospect, annual term — S13.
2026-05-09: Hired ex-Workday VP EMEA to lead European expansion — S11.
2026-03-05: Launched Rivally Pulse engagement-survey add-on — S06.

## 12-month win/loss vs Rivally
Window: 2025-09 to 2026-08 (all 20 deal rows provided).
Arithmetic: 13 wins / 20 deals = 65% win rate; 7 losses.
Wins: Deal-A9FD43 (2025-10), Deal-7AA785 (2025-11), Deal-44C524 (2025-12), Deal-0D0CD6 (2026-01), Deal-D5B790 (2026-02), Deal-5C636E (2026-03), Deal-67BE14 (2026-06), Deal-1B6969 (2026-07), Deal-F03E7B (2026-08), Deal-072E31 (2025-09), Deal-F65C8F (2025-10), Deal-E46EAB (2026-01), Deal-1D2392 (2026-02).
Losses: Deal-7767F5 (2025-09), Deal-5645A5 (2026-04), Deal-C6FFAA (2026-05), Deal-D263E0 (2025-11), Deal-935746 (2025-12), Deal-9066A6 (2026-03), Deal-72A02F (2026-04).
Corroboration, not in deals file: 800-seat prospect picked Bonusly over Rivally citing analytics depth — S25 (2026-09-03).

## Old-card claims — re-sourced or unverified
"points-based recognition for mid-market": sourced — points-based feed S02; mid-market reviewer S04.
"starts at $5/user/month, annual billing (as of 2026-01)": superseded — $5 shown in S03 (2026-01-20) and S08 (2026-04-01), but newer S17 (2026-08-12) lists $7.
"Rivally lacks a Slack integration": unverified — contradicted by S04 (Slack worked out of the box).
"Rivally was acquired by WorkHuman in 2025": unverified — no supporting snippet.
"Strong in EU enterprise with multi-language support": sourced — S12, S15.

## Not facts (rep opinion, excluded from claims)
UI feels clunky — AE opinion (Bryce Harmon), S09, not verified against the product.
Rivally discounting aggressively — AE opinion (Elena Sinclair), S21, not confirmed.
nurture-sequence-diagnosisPASS 1.00
Plain-English question: What is broken in each email sequence — rates, overlaps, tracking errors?
Prompt — what the model saw
Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0168 · 121s · in 58,277 / out 20,404 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

Plain-English answer: Per-sequence open, reply, and meeting rates with the weakest step (all three weakest at step 3), an opened-above-sent tracking error, and audience overlap flagged.
The answer in one line: fix_priority_first = Cold Outbound - HR Leaders
all_pairwise_overlaps['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
audience_overlap_pairCold Outbound - HR Leaders, Cold Outbound - People Ops
failure_modeCold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_firstCold Outbound - HR Leaders
per_sequence{"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_errorsequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
under_2pct_replyCold Outbound - HR Leaders
graded checkstracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
Per sequence (totals across 3 steps; rates = ÷ total sent):

- New Logo Nurture — sent 1,386; opened 490 → open rate 35.4% (490/1,386); replied 90 → reply rate 6.5% (90/1,386); meetings 27 → meeting rate 2.0% (27/1,386). Weakest step: 3 (reply 4.2%, 18/428).
- Expansion Nurture — sent 875; opened 565 → 64.6%; replied 59 → 6.7%; meetings 12 → 1.4%. Weakest step: 3 (reply 4.4%, 12/275).
- Cold Outbound - HR Leaders — sent 1,785; opened 545 → 30.5%; replied 8 → reply rate 0.5% (8/1,785); meetings 0 → 0.0%. Weakest step: 3 (reply 0.2%, 1/590).
- Cold Outbound - People Ops — sent 1,163; opened 340 → 29.2%; replied 29 → 2.5%; meetings 6 → 0.5%. Weakest step: 3 (reply 1.6%, 6/377).

Tracking error: Expansion Nurture step 2 — opened 340 > sent 300. Its 64.6% open rate is inflated (likely pixel double-count).

Audience overlap: 23 contacts appear in more than one sequence (963 rows, 940 unique). Cold Outbound - HR Leaders ↔ Cold Outbound - People Ops: 21 shared contacts. Expansion Nurture ↔ New Logo Nurture: 2 shared.

Failure mode under 2% reply: Cold Outbound - HR Leaders (0.5%). Replies decay 0.8% → 0.3% → 0.2% across steps with zero meetings from 1,785 sent — list/offer mismatch for HR leaders; replies never convert to conversations.

One change per weak sequence; fix Cold Outbound - HR Leaders first:

1. Cold Outbound - HR Leaders (fix first): rebuild sequence 1 with a segmented, role-specific HR value prop (step 1 now 0.8% reply, 5/600, 0 meetings).
2. Expansion Nurture: fix step-2 tracking duplication, then retest the step-3 CTA (4.4%).
3. New Logo Nurture: strengthen the step-3 meeting ask (4.2%).
4. Cold Outbound - People Ops: rework the step-3 follow-up (1.6%).
weekly-marketing-goals-rollupPASS 1.00
Plain-English question: Are we on pace on the quarter's marketing goals?
Prompt — what the model saw
Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0033 · 32s · in 3,397 / out 3,848 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

Plain-English answer: SQMs ahead of pace, SQOs on pace, DS2s behind (35 short of expected at day 66 of 92), with deltas and same-quarter closes.
The answer in one line: days_in_quarter = 92; days_elapsed = 66; coverage_ratio_active_pipeline = 0.75; elapsed_fraction = 0.7174 (+1 more below)
days_in_quarter92
coverage_ratio_active_pipeline0.75
days_elapsed66
elapsed_fraction0.7174
metrics{"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
pace_ruleexpected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required1
graded checksmetrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 weekly marketing goals update (quarter: 2026-07-01 to 2026-09-30; 66 of 92 days elapsed = 71.74% of quarter)

SQMs: QTD actual 230 | target 300 | delta -70 | pace 76.67% of target (linear expectation at 66 days: 300 x 66/92 = ~215) | AHEAD
SQOs: QTD actual 84 | target 120 | delta -36 | pace 70.00% of target (expect 120 x 66/92 = ~86) | BEHIND
DS2s: QTD actual 40 | target 75 | delta -35 | pace 53.33% of target (expect 75 x 66/92 = ~54) | BEHIND
closed_lost_mia_rate: 20.00% (5 MIA of 25 closed lost) | target 10.00% | delta +10.00 pts | lower is better; actual is 2x target | BEHIND
same_quarter_closes: QTD actual 10 | target 20 | delta -10 | pace 50.00% of target (expect 20 x 66/92 = ~14) | BEHIND
active_pipeline: QTD actual $3,000,000 | target $4,000,000 | delta -$1,000,000 | pace 75.00% of target (expect 4,000,000 x 66/92 = ~$2,869,565) | AHEAD

Arithmetic: time elapsed 66 / 92 = 71.74%. Pace compares metric actual/target to 71.74% elapsed; for closed_lost_mia_rate (lower better), actual 5/25 = 0.20 vs target 0.10, delta +0.10.

Metric names cited exactly as given: SQMs, SQOs, DS2s, closed_lost_mia_rate, same_quarter_closes, active_pipeline. No deal or company aliases appear in the provided data.

What moved this week: cannot be determined from the files provided — they contain QTD actuals and targets only, with no prior-week or week-over-week figures. No movement claim is made; the update reflects QTD position against target only.
communication 5 tests
ceo-slack-compression0.80
Plain-English question: Compress the forecast analysis into one Slack message the CEO can read in ten seconds.
Prompt — what the model saw
Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0035 · 34s · in 3,762 / out 4,335 tok
✗ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

Plain-English answer: Three sentences, under 80 words, decision first, every distinct number kept plus the one caveat — all checked by deterministic text rules.
The answer in one line: weighted_number = 115976; must_keep = data-quality caveat — deleting the bad news fails
weighted_number115,976
limitmax 3 sentences, max 80 words, no em-dashes
must_keepdata-quality caveat — deleting the bad news fails
graded checksmax_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Q3 weighted forecast is 115,976.75: 44,729 COMMIT plus 71,247.75 (35 percent of 203,565 BEST_CASE). Treat it as directional, not a number to commit to. Owner is blank on 85 of 86 deals, why-buys is empty on all 7 in-quarter COMMIT deals, and 32 of 86 close in October, so this lacks rep rollup, buyer reasons, and clean close dates. Re-qualify October commits, including Deal-D348E1 at 13,770 closing 2026-10-15, before locking the number.
stalled-deal-followup-emailPASS 1.00
Plain-English question: Write a follow-up email for this stalled deal — under 120 words, one prior touch, one ask.
Prompt — what the model saw
Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0026 · 21s · in 3,541 / out 2,133 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

Plain-English answer: An email that references the July 28 milestone-awards demo, makes one clear ask, and invents nothing — Deal-0D2F7A, 31 days since last engagement.
The answer in one line: amount = 5100; word_cap = 120; days_since_last_engagement = 31; deal_alias = Deal-0D2F7A (+4 more below)
amount5,100
days_since_last_engagement31
word_cap120
allowed_facts14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_newENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
deal_aliasDeal-0D2F7A
must_referenceengagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
ownerAlex Franklin
single_cta1
stageDS3
tonewarm, concise, professional (matches prior thread)
graded checksword_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Re: Deal-0D2F7A — follow-up on the July 28 demo

Hi,

I'm following up on my August 5 email recapping the July 28 demo with your People team, including pricing for 150 seats. You asked about pricing after the demo — do you have questions on automated milestone awards or the points catalog?

Would you have 15 minutes next week to revisit next steps?

Best,
Alex Franklin

Word count arithmetic: body = 56 words; subject = 10 words; total = 66 words, under the 120-word cap.

Prior touch referenced: ENG-00001 (2026-08-05) — recap email of the July 28 demo with pricing for 150 seats. All details taken from stalled_deal.csv and stalled_engagements.csv only; no facts invented.
funnel-digest-paragraphPASS 1.00
Plain-English question: Write the weekly GTM digest — one paragraph per team, every number traceable.
Prompt — what the model saw
Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0020 · 8s · in 3,351 / out 653 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

Plain-English answer: Four paragraphs (Marketing, Sales, CS, PLG) using only the allowed numbers, no contact names, no performance commentary, under 400 words.
The answer in one line: word_cap = 400
word_cap400
allowed_numbers16 items — e.g. 2, 3, 9, 08
bannedcustomer contact names, employee performance commentary, compensation
required_paragraphsMarketing, Sales, CS, PLG
graded checksfour_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing

The team landed 46 SQMs against a weekly target of 52. The webinar channel contributed 18 of those SQMs. Cost per SQM on paid search held at $150.

Sales

Sales converted 14 SQOs and set 9 DS2 meetings. The team created $310,000 in new pipeline and closed 3 same-quarter deals.

CS

CS saved 2 renewals this week. Team NPS moved to 61. Three open red-flag accounts head into next week.

PLG

PLG added 412 new signups with activation at 31%. Thirty-eight companies reached the aha moment of 10 recognition gives.
partner-digest-sectionPASS 1.00
Plain-English question: Write the partner digest so every partner in the roster appears exactly once.
Prompt — what the model saw
Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0027 · 19s · in 3,578 / out 2,296 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

Plain-English answer: Four active partners with status and numbers, four quiet partners with one line each, and partner-sourced pipeline that reconciles to the deal data.
The answer in one line: rule = every roster partner named exactly once; quiet partners get one line; no partners outside roster.
active_no_dealsWorkWell Group
partner_sourced_pipelinecount=5; amount=550000
per_partner_pipeline{"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
quiet_partnersRecogniq, KudosWave, PeopleFirst Advisors, TotalPerk
rosterpartner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
ruleevery roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checksall_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner digest — 2026-08-24 to 2026-09-04

Active partners:

- Apex Rewards Co (active): 2 partner-sourced opps, both in DS1; pipeline $275,000 (Deal-DDAAF2 $180,000 + Deal-2CF33E $95,000). Co-webinar locked for 09-15. UTM Source = Partner confirmed on Slack 08-25.
- HRCloud Partners (active): 1 partner-sourced opp, moved to DS2 after security review; pipeline $140,000 (Deal-F1CDA5). Slack 08-27.
- CultureBridge (active): 2 partner-sourced opps, both early stage; pipeline $135,000 (Deal-096E1D $60,000 + Deal-067213 $75,000). Lunch-and-learn 08-29.
- WorkWell Group (active): 0 partner-sourced opps; pipeline $0. No sourced deals this period (Slack 09-02); Q4 joint playbook restart, planning call booked 09-09.

Quiet partners:

- Recogniq (quiet): no activity this period.
- KudosWave (quiet): no activity this period.
- PeopleFirst Advisors (quiet): no activity this period.
- TotalPerk (quiet): no activity this period.

Totals: 5 partner-sourced opps, $550,000 pipeline.

Arithmetic: $180,000 + $95,000 = $275,000 (Apex); $140,000 (HRCloud); $60,000 + $75,000 = $135,000 (CultureBridge); $0 (WorkWell). $275,000 + $140,000 + $135,000 + $0 = $550,000. Count: 2 + 1 + 2 + 0 = 5. Matches partner_deals.csv: 5 rows, all utm_source=Partner, all four active partners represented; WorkWell and all four quiet partners have no deal rows.

No data exists for Recogniq, KudosWave, PeopleFirst Advisors, or TotalPerk in the Slack export or deal file; nothing invented.
rfp-answer-from-policyPASS 1.00
Plain-English question: Answer the 10 RFP items from the policy excerpts only — no guessing.
Prompt — what the model saw
Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0023 · 15s · in 3,539 / out 1,423 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

Plain-English answer: Seven answered with excerpt citations (Q1-Q7), three flagged unanswerable with the document that would answer them (Q8-Q10).
The answer in one line: answered_count = 7; rule = unanswerable items must be flagged and the document that would answer them named; do not guess.
answered_count7
answer_map{"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
ruleunanswerable items must be flagged and the document that would answer them named; do not guess.
unanswerableQ8, Q9, Q10
valid_citation_idsP1, P2, P3, P4, P5, P6, P7, P8
graded checksanswered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1: Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2: Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3: Yes. Bonusly maintains a SOC 2 Type II report, renewed annually; the report is available to prospects under NDA. [P2]

Q4: Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. Retention after contract termination: 90 days. [P3]

Q5: The subprocessor list is published at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

Q6: Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7: Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8: Unanswerable. No excerpt addresses ISO 27001 certification. An ISO 27001 certificate or certification register would answer this.

Q9: Unanswerable. No excerpt states a contractual uptime SLA percentage. A service-level agreement (SLA) document would answer this.

Q10: Unanswerable. No excerpt addresses HIPAA or a Business Associate Agreement. A HIPAA compliance policy or BAA template would answer this.
ops-maintenance 5 tests
skill-file-reconciliation0.71
Plain-English question: Audit a skill set against its manifest and flag every conflict with a severity and an action.
Prompt — what the model saw
Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.0619 · 330s · in 179,923 / out 39,637 tok
✗ duplicate_trigger_found✓ circular_chain✗ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

Plain-English answer: A severity-ranked list of TRIM_DESC / UPDATE_BODY / DELETE_SKILL / MERGE / REVIEW actions covering every rule violation found.
The answer in one line: files_audited = 14; rank = CRITICAL / WARNING / INFO with one fix action per issue
files_audited14
findingsduplicate trigger, circular delegation chain, dangling file reference, oversized description, hardcoded values, manifest drift
rankCRITICAL / WARNING / INFO with one fix action per issue
graded checksduplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
FINDING 1 — Two skills whose ALWAYS-trigger phrases overlap/duplicate

1A. pipeline-intelligence-report <-> weekly-pipeline-report — SEVERITY: CRITICAL — ACTION: TRIM_DESC
Both skills ALWAYS-trigger on the same phrases: "pipeline report", "pipeline update", and "what does/look like pipeline" (weekly-pipeline-report: "run the pipeline update", "weekly pipeline report", "generate the pipeline report", "what does pipeline look like", "do the pipeline report", "update the pipeline"; pipeline-intelligence-report: "run the pipeline report", "pipeline update", "what's the pipeline look like"). A router cannot decide which skill owns "pipeline report/update".
Proposal: Trim weekly-pipeline-report's ALWAYS list to weekly-specific phrasing only ("weekly pipeline report", "pipeline summary", "mid-month pipeline check", "give me this week's numbers", "SQM/SQO/DS2 metrics", "bookings MTD"). pipeline-intelligence-report keeps "pipeline report", "pipeline update", "score the pipeline", "pipeline intelligence", "full pipeline".

1B. comms-drafter <-> email-drafter — SEVERITY: WARNING — ACTION: TRIM_DESC
Both trigger on "write me an email", "draft a follow-up", "help me reply", "what should I say", "bump email", "contract nudge", and feedback/rewrite of an existing message. comms-drafter is channel-agnostic external copy; email-drafter is email-specific, yet the trigger phrases duplicate.
Proposal: Dedupe the triggers — email-drafter keeps email-specific phrasing; comms-drafter keeps non-email channels (Intercom/support tickets, partner outreach, tickets) and drops the email-specific duplicates.

FINDING 2 — Circular delegation chain — SEVERITY: CRITICAL — ACTION: TRIM_DESC
Chain name: deal-strategy-coach <-> email-drafter
- deal-strategy-coach body: "When drafting manager-to-prospect emails, use the email-drafter skill which automatically retrieves your Gmail signature..." (delegates TO email-drafter)
- email-drafter body: "For deal strategy, diagnosis, or coaching (not email drafting), use deal-strategy-coach instead." (delegates TO deal-strategy-coach)
- comms-drafter feeds the cycle: "For deep deal strategy, use deal-strategy-coach — this skill drafts, that skill diagnoses."
Proposal: Remove the mutual cross-delegation sentences from both bodies so neither skill points at the other and the cycle is broken; strategy-vs-drafting routing is handled at the request/orchestrator layer, not inside the skills.

FINDING 3 — Delegation targets that do not exist in this skill set — SEVERITY: CRITICAL — ACTION: REVIEW
Targets referenced in skill bodies that are not a declared_name or file in the manifest/14 files (11 unique):
- analysis-validator §12.4: bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, bonusly-datadog-questions
- analysis-validator §11: skill-orchestrator
- comms-drafter Step 0: bonusly-brand; comms-drafter: prospect-research-multithreading
- deal-strategy-coach: prospect-research-multithreading
- email-drafter: bonusly-brand, prospect-research-multithreading
- sales-forecast: bonusly-brand
- signalforge-claim-compressor: bonusly-brand
Proposal: Confirm each target exists outside this set; if not, remove or repoint the delegation. Do not invent targets.

FINDING 4 — Version conflict — SEVERITY: CRITICAL — ACTION: DELETE_SKILL + UPDATE_BODY
The on-disk skill_manifest.csv (15 rows) declares stale-pipeline-report twice with conflicting values:
- stale-pipeline-report,stale-pipeline-report,956,645 (stale duplicate)
- stale-pipeline-report,stale-pipeline-report,762,295 (matches the actual file)
Surviving row: 762/295. Delete the 956/645 row.
Also: pipeline-intelligence-report's manifest row has "v6" (its version) leaked into the description_chars column; it should be 1006/536 (UPDATE_BODY).
Note: the prompt-rendered manifest (14 rows) does not contain the duplicate; the two manifest copies disagree.

FINDING 5 — Manifest descriptions exceeding 1,024 characters — SEVERITY: INFO — ACTION: none
Count: 0. Max description_chars in the manifest is 1,006 (pipeline-intelligence-report, signalforge-claim-compressor). None exceed 1,024.

FINDING 6 — Hardcoded page ids, dates, or person names in skill bodies — SEVERITY: WARNING — ACTION: REVIEW
- analysis-validator: Person names (§12.3 GTM roster: Bryce Harmon, Hugo Lindqvist, Dana Mercer, Alex Franklin, Cole Ingram, Gavin Porter, Colleen Perry, Ellie Barton, Ashley Reyer, Megan Franz, Elena Sinclair, Youssef Elkhateeb, Amanda Czenkus, Alaina Loori, Shealagh Coughlin, Ben Castelli, Amani Phipps, John Thomas, Yasmin Wahid, each with a HubSpot owner ID); escalation names Manish, Amani. Dates: Created April 26, 2026; Last Updated May 9, 2026 (v3.6); CALL_SPOTLIGHT_BRIEF removed May 4, 2026; HubSpot DEALS stale as of March 28, 2023; "May 2026" anchors.
- closed-lost-analysis: Date: "30-deal AI-field sample from May 2026." (company examples only; no person names, no page ids)
- comms-drafter: none
- deal-strategy-coach: Page id: AE Excellence Playbook Confluence page 2257879045 (cloud bonusly1612893911). Person name: Farid (.edu routed to Farid).
- email-drafter: none
- model-selection: Dates: last_checked 2026-05-19; knowledge cutoffs Feb 2025 / Aug 2025 / Jan 2026; changelog 2026-05-19.
- next-to-close: none (1973303 is a HubSpot org id, not a page id; no dates or person names)
- partner-digest: Page/space ids: cloudId 73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f, spaceId 1958248479, folder id 2286616609, canonical reference page 2286321666. Dates: May 16, 2026 (canonical issue); "Week of May 19, 2026", "Week of June 2, 2026"; changelog 2026-05-17. Person names: Amani (Amani's threads; from:<@U03QLMBL7AR>; Owner: Amani Phipps).
- pipeline-intelligence-report: Person names (AE roster with owner IDs: Bryce Harmon 119337721, Dana Mercer 83155923, Cole Ingram 83155924, Alex Franklin 84342457, Gavin Porter 1520255671). Id: HubSpot org 1973303 (deal URL template). Dates: v6 · May 2026; "verified May 2026."
- sales-forecast: Page/space ids: spaceId 2232811524, cloudId 73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f, parent page id 2232582148. Dates: changelog April 27, 2026; title format Q[N] [YEAR]. Person names: Alaina (VP Sales); Elena (prior, replaced).
- signalforge-claim-compressor: Date: changelog 2026-05-09.
- signalforge-feedback: Page/space ids: page id 2295136266, spaceId 2232811524, cloudId 73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f, parent 2234417154, Build Log page id 2247295002.
- stale-pipeline-report: Hardcoded ids (Slack channel C0561C1JCPJ; Bonusly Support owner id 55483190) but no page ids, dates, or person names.
- weekly-pipeline-report: Spreadsheet ids: 1CLZeOsElVDF_LF0ZG_t2nfwvhnZ6bpwqM_nX3WEYzcw, 1ENuaEcCuLjdKhMvp8FK3Ys1ek5Aw9ZuOZhsHJJFoB_k. Dates: Q2 April 1 – June 30, 2026; Q1 2026 actuals ($365,152 vs $475,000 plan; $2,490,532 vs $3,288,000 forecast). Person names: Ben Lavin (title), "Ben" (reviewer).

FINDING 7 — Manifest drift in both directions — SEVERITY: INFO — ACTION: none
- Files with no manifest row: 0 (all 14 files have a manifest row).
- Manifest rows with no file: 0 (all 14 manifest rows have a file).
The 14 files and the 14 prompt-rendered manifest rows map 1:1 with no drift in either direction. The on-disk CSV's extra stale-pipeline-report row is internal duplication (Finding 4), not a file/row mismatch.

Summary of actions proposed: TRIM_DESC for findings 1A, 1B, and 2; REVIEW for findings 3 and 6; DELETE_SKILL (duplicate stale-pipeline-report 956/645 row) + UPDATE_BODY (pipeline-intelligence-report v6 leak) for finding 4; no action for findings 5 and 7. Nothing was rewritten.
runbook-from-slack-threadPASS 1.00
Plain-English question: Turn the resolved incident thread into a runbook.
Prompt — what the model saw
Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0043 · 49s · in 3,804 / out 6,321 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

Plain-English answer: Ordered steps from the thread, each with the exact command, who ran it, how success was verified, and rollback where state changed.
The answer in one line: rule = no steps beyond what the thread contains; every step traces to a message id.
ambiguous_steporder=4; msg=M05; mark=needs-confirmation
ruleno steps beyond what the thread contains; every step traces to a message id.
stepsorder=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
graded checksordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
INCIDENT RUNBOOK — reward-worker queue depth > 10k
Source: incident_thread.md, #incident-2026-08-29-reward-queue (resolved)
All steps trace to thread messages M01-M10. No facts, deals, companies, or numbers added beyond the thread. Unclear items are marked NEEDS CONFIRMATION. No deal or company aliases appear in the thread.

ARITHMETIC (thread values only)
- Peak queue depth: 48,213 pending jobs (M02). Normal is under 500. Using 500 as the top of the normal range, excess above normal: 48,213 - 500 = 47,713 jobs.
- Depth at 14:33:41Z: 9,400 (M07). Cleared between M02 and M07: 48,213 - 9,400 = 38,813 jobs.
- Reported drain rate: about 1,200 jobs/minute (M07). At that rate, 9,400 remaining is about 7.8 minutes to drain.
- M09 verification: 40 new jobs processed cleanly in 3 minutes.

STEP 1 — Acknowledge alert, take incident commander role
Source: M01 (2026-08-29 14:02:10Z, Bryce Harmon)
Action: PagerDuty alert fired for reward-worker queue depth > 10k. Bryce Harmon acknowledging, taking IC.
Command: none recorded.
Success verified by: alert acknowledged; IC assigned (per M01 text).
Rollback: none (notification/action, no system state change).
NEEDS CONFIRMATION: none.

STEP 2 — Measure reward queue depth
Source: M02 (2026-08-29 14:04:33Z, Farid Osman)
Action: Checked reward queue depth.
Command: bundle exec rake sidekiq:queue_depth
Result: 48,213 pending jobs. Normal is under 500.
Success verified by: command returned 48,213, confirming the queue depth > 10k alert.
Rollback: none (read-only diagnostic).
NEEDS CONFIRMATION: none.

STEP 3 — Inspect Sidekiq dead set
Source: M03 (2026-08-29 14:06:02Z, Farid Osman)
Action: Inspected the dead set.
Command: not recorded in the thread.
Result: 112 jobs, all Redis::TimeoutError from around 13:58.
Success verified by: dead set contents observed (112 jobs).
Rollback: none (read-only diagnostic).
NEEDS CONFIRMATION: exact command used to inspect the dead set.

STEP 4 — Pause enqueue to stop the bleed
Source: M04 (2026-08-29 14:08:45Z, Farid Osman)
Action: Disabled the auto_recognition_enqueue feature flag.
Command: bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'
Rollback command: bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'
Success verified by: enqueue paused (stated intent: "Pausing enqueue to stop the bleed"); no post-command metric is quoted in the thread.
NEEDS CONFIRMATION: thread states intent but does not quote a verification metric after this command.

STEP 5 — Clear dead set
Source: M05 (2026-08-29 14:15:20Z, Elena Sinclair)
Action: Cleared out the dead set while in the console (112 jobs, all Redis::TimeoutError, per M03).
Command: NOT recorded in the thread.
Rollback: NOT recorded / not recoverable — cleared jobs are not shown as restorable in the thread.
Success verified by: Elena Sinclair states the dead set was cleared; no post-clear count is quoted.
NEEDS CONFIRMATION: exact console command used to clear the dead set; whether any rollback or recovery was considered for the 112 cleared jobs.

STEP 6 — Scale workers up
Source: M06 (2026-08-29 14:21:07Z, Bryce Harmon)
Action: Scaled reward-worker deployment from 3 to 6 replicas.
Command: kubectl scale deployment/reward-worker --replicas=6
Rollback command: kubectl scale deployment/reward-worker --replicas=3
Success verified by: subsequent M07 shows queue depth falling; no immediate post-scale metric is quoted in M06 itself.
NEEDS CONFIRMATION: none for command/rollback; immediate post-scale verification metric not quoted.

STEP 7 — Confirm drain in progress
Source: M07 (2026-08-29 14:33:41Z, Farid Osman)
Action: Checked queue depth during drain.
Command: not re-quoted in M07 (same check as M02 implied, not restated).
Result: 9,400 pending, falling about 1,200/minute.
Success verified by: 48,213 to 9,400 (38,813 cleared) at reported about 1,200/minute.
Rollback: none (observation).
NEEDS CONFIRMATION: command used for this check is not re-quoted in M07.

STEP 8 — Verify queue at zero and error rate baseline
Source: M08 (2026-08-29 14:47:55Z, Cole Ingram)
Action: Verified queue depth and Datadog error rate.
Command: bundle exec rake sidekiq:queue_depth
Result: 0 pending jobs. Error rate in Datadog back to baseline.
Success verified by: command returned 0; Datadog error rate at baseline.
Rollback: none (verification).
NEEDS CONFIRMATION: none.

STEP 9 — Re-enable enqueue
Source: M09 (2026-08-29 14:49:10Z, Bryce Harmon)
Action: Re-enabled the auto_recognition_enqueue feature flag (rollback of Step 4).
Command: bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'
Rollback command: bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)' (the Step 4 command)
Success verified by: 40 new jobs processed cleanly in the next 3 minutes.
NEEDS CONFIRMATION: none.

STEP 10 — Scale workers down, close incident
Source: M10 (2026-08-29 14:55:00Z, Bryce Harmon)
Action: Scaled reward-worker deployment back to 3 replicas (rollback of Step 6).
Command: kubectl scale deployment/reward-worker --replicas=3
Rollback command: kubectl scale deployment/reward-worker --replicas=6 (the Step 6 command)
Result: Queue stable at 0.
Success verified by: queue stable at 0 after scale-down; incident resolved (per M10).
Rollback: kubectl scale deployment/reward-worker --replicas=6 if queue rises again.
NEEDS CONFIRMATION: none.
datadog-incident-timelinePASS 1.00
Plain-English question: Sequence the failure from the logs — what broke first, and what cascaded after it?
Prompt — what the model saw
From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0055 · 62s · in 5,204 / out 8,988 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

Plain-English answer: A reward-service Redis timeout at 14:01:12Z cascades to sidekiq at 14:01:40 and api-gateway at 14:03:05, with the Datadog query that confirms the first error.
The answer in one line: job_class_involved = RewardGiveJob
cascade_orderservice=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
clean_servicespostgres
confirming_queryservice:reward-service status:error "@message:Redis::TimeoutError"
first_errortimestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
job_class_involvedRewardGiveJob
logs_do_not_showroot cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
graded checksfirst_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
FIRST ERROR

2026-09-03T14:01:12Z | service: reward-service | level: ERROR | message: Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s

CASCADE IN ORDER (datadog_logs.csv, timestamp order)

1. 2026-09-03T14:01:12Z reward-service ERROR Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s (root cause)
2. 2026-09-03T14:01:20Z reward-service ERROR Redis::TimeoutError: retry exhausted for RewardGiveJob
3. 2026-09-03T14:01:30Z reward-service ERROR Redis::TimeoutError: retry exhausted for RewardGiveJob
4. 2026-09-03T14:01:40Z reward-service ERROR Redis::TimeoutError: retry exhausted for RewardGiveJob
5. 2026-09-03T14:01:40Z sidekiq ERROR RewardGiveJob failed: Redis::TimeoutError; retrying in 60s
   sidekiq_jobs.csv corroboration: J-00005 RewardGiveJob failed_at 2026-09-03T14:01:46Z (earliest failed job in the jobs file)
6. 2026-09-03T14:02:28Z sidekiq ERROR RewardGiveJob failed: Redis::TimeoutError; retrying
7. 2026-09-03T14:02:30Z sidekiq WARN "Queue reward depth above 10,000"
   sidekiq_jobs.csv corroboration: J-00013 RecognitionDigestJob failed_at 2026-09-03T14:02:36Z (first RecognitionDigestJob failure)
8. 2026-09-03T14:03:05Z api-gateway ERROR 502 upstream timeout calling reward-service /gives (first downstream 502)
9. 2026-09-03T14:03:30Z web-app ERROR Give form submission failed: upstream 502 from api-gateway
10. 2026-09-03T14:03:31Z sidekiq ERROR RewardGiveJob failed: Redis::TimeoutError; retrying
11. 2026-09-03T14:03:48Z api-gateway ERROR 502 upstream timeout calling reward-service
12. 2026-09-03T14:04:13Z api-gateway ERROR 502 upstream timeout calling reward-service
13. 2026-09-03T14:04:22Z sidekiq ERROR RewardGiveJob failed: Redis::TimeoutError; retrying
14. 2026-09-03T14:04:45Z web-app ERROR Give form submission failed: upstream 502
   sidekiq_jobs.csv corroboration: J-00015 RecognitionDigestJob failed_at 2026-09-03T14:04:55Z
15. 2026-09-03T14:05:16Z api-gateway ERROR 502 upstream timeout calling reward-service
16. 2026-09-03T14:05:26Z sidekiq ERROR RewardGiveJob failed: Redis::TimeoutError; retrying
17. 2026-09-03T14:05:42Z web-app ERROR Give form submission failed: upstream 502
   sidekiq_jobs.csv corroboration: J-00016 RecognitionDigestJob failed_at 2026-09-03T14:05:50Z
18. 2026-09-03T14:06:47Z sidekiq ERROR RewardGiveJob failed: Redis::TimeoutError; retrying
19. 2026-09-03T14:06:49Z web-app ERROR Give form submission failed: upstream 502
20. 2026-09-03T14:06:52Z api-gateway ERROR 502 upstream timeout calling reward-service

RECOVERY

21. 2026-09-03T14:22:10Z reward-service INFO Redis connection restored; resuming job processing
22. 2026-09-03T14:24:45Z sidekiq INFO Queue reward depth below 500

SERVICE AND JOB INVOLVED

Service: reward-service is the origin of the failure. sidekiq processes the job and reports failures/retries. api-gateway and web-app are downstream, hit by 502s.
Job: RewardGiveJob (named in every reward-service and sidekiq ERROR line). RecognitionDigestJob also fails per sidekiq_jobs.csv: J-00013, J-00014, J-00015, J-00016.

DATADOG QUERY TO CONFIRM THE FIRST ERROR

service:reward-service level:error "redis-primary:6379" | sort by @timestamp asc | limit 1

Expected first match: 2026-09-03T14:01:12Z, "Connection to Redis at redis-primary:6379 timed out after 5s".

ARITHMETIC

Outage window: first error 14:01:12Z to Redis restored 14:22:10Z = 1,258 seconds = 20 minutes 58 seconds.
First error to first api-gateway 502 (14:03:05Z): 113 seconds = 1 minute 53 seconds.
First error to first web-app failure (14:03:30Z): 138 seconds = 2 minutes 18 seconds.
Redis restored to queue below 500 (14:22:10Z to 14:24:45Z): 155 seconds = 2 minutes 35 seconds.
Error volume in datadog_logs.csv: 19 ERROR lines total (reward-service 4, sidekiq 6, api-gateway 5, web-app 4) plus 1 WARN (sidekiq, queue depth above 10,000).
sidekiq_jobs.csv: 16 failed jobs total — 12 RewardGiveJob (J-00001 to J-00012) and 4 RecognitionDigestJob (J-00013 to J-00016). Earliest failed_at: J-00005 at 14:01:46Z.

WHAT THE LOGS DO NOT SHOW

Root cause of the Redis timeout (network, Redis saturation, client pool exhaustion, etc.); only the symptom at redis-primary:6379.
The Redis key, command, or client configuration involved.
Whether each failed job was eventually retried to success or permanently dropped; sidekiq_jobs.csv records failures, not outcomes.
Peak queue depth; only "above 10,000" at 14:02:30Z is shown, with recovery marked "below 500" at 14:24:45Z.
Customer, deal, or company impact; no deal or company aliases appear in either file.
Completeness of the failure set; 16 failed jobs are listed, but the logs cannot confirm that is the total.
Whether every api-gateway 502 targeted /gives; only the 14:03:05Z 502 names that path.
feature-flag-state-summaryPASS 1.00
Plain-English question: Summarize what each feature flag controls and who has it on.
Prompt — what the model saw
Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0023 · 14s · in 3,587 / out 1,397 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: Recognition streaks on for 42 beta companies, redeem-flow redesign off with a 12-company target list, plus any flag with no code reference or no targets.
The answer in one line: rule = company counts must equal the export; do not describe flags absent from the export.
code_referencedrecognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
flags{"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
flags_with_no_targetspaused_offboard_cleanup
orphan_flags_no_code_referencelegacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
rulecompany counts must equal the export; do not describe flags absent from the export.
graded checksorphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
**Feature flag state summary**

1. **recognition_streaks_v2** — state: on | targeting: segment:beta_companies | companies: 42
   - Controls: `app/models/recognition.rb` — when enabled, `StreakTracker.record(give)` runs on a give; when off, streak tracking is skipped.
   - Targets: beta_companies segment (42 companies).

2. **points_budget_guardrails** — state: on | targeting: all_companies | companies: 220
   - Controls: `app/services/budget_service.rb` — when enabled, `BudgetService.new(company).enforce!(giver, points)` runs; when off, budget enforcement is skipped.
   - Targets: all companies (220).

3. **slack_dm_nudges** — state: on | targeting: segment:region_na | companies: 87
   - Controls: `app/jobs/nudge_job.rb` — when enabled, `SlackDm.send_nudge(user)` sends; when off, the job returns early and no nudge is sent.
   - Targets: region_na segment (87 companies).

4. **redeem_flow_redesign** — state: off | targeting: targeted_list | companies: 12
   - Controls: `app/controllers/redeem_controller.rb` — when enabled, renders `RedeemV2Component.new(company)`; when off, renders `RedeemV1Component.new(company)`.
   - Targets: targeted_list (12 companies); the export lists no specific segment or member list for "targeted_list".

5. **analytics_dashboard_v3** — state: on | targeting: segment:tier_three | companies: 65
   - Controls: `app/controllers/analytics_controller.rb` — when enabled, `@dashboard = AnalyticsV3.new(company)`; when off, no V3 dashboard is assigned (older dashboard implied by absence of an else branch in the excerpt).
   - Targets: tier_three segment (65 companies).

6. **ms_teams_app_v2** — state: off | targeting: targeted_list | companies: 9
   - Controls: `app/services/teams_installer.rb` — when enabled, `TeamsAppV2.install(company)` runs; when off, no Teams V2 install is attempted.
   - Targets: targeted_list (9 companies); the export lists no specific segment or member list for "targeted_list".

7. **legacy_give_modal** — state: off | targeting: segment:legacy_plan | companies: 14
   - Controls: no code reference in the provided excerpt.
   - Targets: legacy_plan segment (14 companies).

8. **survey_boosters_q3** — state: on | targeting: segment:legacy_plan | companies: 7
   - Controls: no code reference in the provided excerpt.
   - Targets: legacy_plan segment (7 companies).

9. **paused_offboard_cleanup** — state: off | targeting: (none listed) | companies: 0
   - Controls: no code reference in the provided excerpt.
   - Targets: none (0 companies).

**Flags with no code reference:** legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup.
**Flags with no targets:** paused_offboard_cleanup (empty targeting rules, 0 companies).
**Flags using "targeted_list" without a named segment or member list:** redeem_flow_redesign, ms_teams_app_v2.

Total companies across the export: 42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0 = 456.
nda-triagePASS 1.00
Plain-English question: Triage each NDA green, yellow, or red — and quote the clause that decides it.
Prompt — what the model saw
Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0026 · 19s · in 3,603 / out 2,177 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

Plain-English answer: NDA-1 green (standard mutual terms), NDA-2 yellow for an embedded 18-month non-solicit, with the driving clause quoted for every classification.
The answer in one line: non_solicit_in = NDA-2
checksclassification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
classifications{"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_inNDA-2
graded checksgreen · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — [PARTY A]: GREEN (standard approval). Mutual NDA, standard five-part carve-outs, Delaware governing law, no non-solicit or non-compete; two-year term plus three-year survival equals a five-year maximum obligation window (2 + 3 = 5). Driving clause: "Term: two (2) years from the Effective Date; obligations survive for three (3) years after termination."

NDA-2 — [PARTY B]: YELLOW (counsel review). Otherwise standard (Delaware, standard carve-outs, three-year term), but it embeds an eighteen-month post-term employee non-solicit that extends the restriction to four and a half years from the Effective Date (3 + 1.5 = 4.5), which requires counsel review. Driving clause: "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."

NDA-3 — [PARTY C]: RED (full legal review). One-way confidentiality favoring [PARTY C], five-year term, no confidentiality carve-outs, a three-year non-compete overlapping years 1–3 of the term, and Republic of Ireland governing law with exclusive jurisdiction — each is unusual, and together they require full legal review. Driving clauses: "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business."; "No confidentiality carve-outs are stated in this Agreement."; "This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts."