← back to all models ← RevenueBench home
openai

openai/gpt-6.1-sol

Mean 0.945 · 34/40 perfect tests · $3.33 total · modalities in: text, image, file · out: text · each card: the prompt → the correct answer (gold) → this model's actual answer

What do the modalities mean?

Modalities in means what you can send this model: text, images, files, video, or audio. Modalities out means what it can send back. Bench tasks feed text extracts, so modalities do not change the score. They are here so you know what this model can handle in real work.

data-crm 5 tests
stage-hygiene-auditPASS 1.00
Plain-English question: How much of the open DS2+ pipeline is missing a substantive why-buys, and is anything missing DS1 timestamps or out of order?
Prompt — what the model saw
Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.0953 · 36s · in 6 / out 1,304 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

Plain-English answer: 81.5% of the 124 DS2+ deals are missing a substantive why-buys — Alex Franklin 48, Cole Ingram 19, Dana Mercer 18, Bryce Harmon 15, Farid Osman 1 — and every deal has a DS1 timestamp with none regressed.
The answer in one line: n_deals = 156; ds2plus = 124; noncomp_pct = 81.5; bryce_noncomp = 15; no_ds1 = 0; regressed = 0 (+1 more below)
n_deals156
bryce_noncomp15
ds2plus124
no_ds10
noncomp_pct81.5
regressed0
per_owner_noncompAlex Franklin=48 of 61 (78.7%); Cole Ingram=19 of 22 (86.4%); Dana Mercer=18 of 22 (81.8%); Bryce Harmon=15 of 18 (83.3%); Farid Osman=1 of 1 (100%)
graded checksnoncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions
- Why-buys non-compliance: a DS2–DS5 deal with blank, placeholder, or fewer than 15 characters of why-buys text. LONG(n) counts as substantive. Rate = non-compliant deals ÷ DS2+ deals × 100.
- Missing DS1: blank t_ds1 on any open deal.
- Regression: any populated earlier-stage timestamp later than a populated later-stage timestamp. Equal timestamps are allowed; missing timestamps do not prove regression.

Why-buys non-compliance
Overall: 101 ÷ 124 × 100 = 81.45%.

Owner                 Non-compliant / DS2+     Rate
Bryce Harmon          15 / 18                  83.33%
Dana Mercer           18 / 22                  81.82%
Alex Franklin         48 / 61                  78.69%
Cole Ingram           19 / 22                  86.36%
Farid Osman            1 / 1                  100.00%
Elena Sinclair         0 / 0                     N/A

Gavin Porter and Hugo Lindqvist have no supplied open deals; rates are N/A.

Arithmetic check: (15 + 18 + 48 + 19 + 1) ÷ (18 + 22 + 61 + 22 + 1) = 101 ÷ 124.

Timestamp hygiene
- No DS1 timestamp: 0 of 156 deals.
- Regressed under the timestamp-order rule: 0 of 156 deals. Missing stage timestamps limit detection; this does not establish that no historical backward moves occurred.

Three process fixes
1. Block entry into DS2 or later unless why-buys has at least 15 characters and is not a placeholder; require manager review of substance.
2. Give owners a cleanup queue for the 101 non-compliant DS2+ deals and review exceptions weekly.
3. Automatically capture DS1 and every subsequent stage transition in an immutable history; flag missing timestamps and out-of-order entries.
owner-resolutionPASS 1.00
Plain-English question: Every open deal has a blank owner name — resolve each one to a real person using the owners table.
Prompt — what the model saw
The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0983 · 32s · in 9 / out 1,082 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ bryce_amount✓ alex_amount✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: All 156 deals resolve to six named owners; jake_amount maps to Bryce Harmon ($1,054,144) and justin_amount to Alex Franklin ($624,310).
The answer in one line: jake_amount = 1,054,144; justin_amount = 624,310; resolved = 156
jake_amount1,054,144
justin_amount624,310
resolved156
deactivated
per_ownerBryce Harmon=1,054,144; Alex Franklin=624,310; Dana Mercer=341,195; Cole Ingram=288,161.43; Farid Osman=4,134; Elena Sinclair=2,100
graded checksresolved_count · unmatched_none · deactivated_flagged · bryce_amount · alex_amount · no_invented

Model's actual answer

show the model's full answer
Resolved: 156 of 156 open deals.
Arithmetic: 35 + 67 + 24 + 22 + 7 + 1 = 156.

Unmatched owner IDs: none. No unmatched deals to list.

Archived owners in owners.csv:
  1520255671 → Gavin Porter
  77260721 → Hugo Lindqvist
Neither ID appears on an open deal. No open deals map to archived owners. A separate deactivated status was not provided.

Total pipeline per resolved owner, calculated by summing amount for each matching owner_id across all forecast categories:

Owner ID    Owner name        Deals   Total amount
119337721   Bryce Harmon         35   1,054,144.00
84342457    Alex Franklin        67     624,310.00
83155923    Dana Mercer          24     341,195.00
83155924    Cole Ingram          22     288,161.43
716654662   Farid Osman           7       4,134.00
701163055   Elena Sinclair        1       2,100.00

Pipeline reconciliation:
1,054,144.00 + 624,310.00 + 341,195.00 + 288,161.43 + 4,134.00 + 2,100.00 = 2,314,044.43.

Currency was not provided.
gong-hubspot-join-integrityPASS 1.00
Plain-English question: How many call records point at deal ids that do not exist in the open pipeline?
Prompt — what the model saw
Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.1380 · 68s · in 9 / out 3,642 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

Plain-English answer: 30 Gong call rows are orphans keyed to dead deal ids, none are duplicates, and DS3 call coverage is 29.4%.
The answer in one line: orphans = 30; ds3_cov_pct = 29.4; dup = 0
orphans30
ds3_cov_pct29.4
dup0
orphan_aliasesDeal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
graded checksorphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
(a) Orphans: 30 Gong rows have no matching deal_id in deals_open.
Arithmetic: 67 Gong rows − 37 matched rows = 30.
Sample aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42.
These are absent from the supplied open-deal table; that alone does not prove the mappings are incorrect.

(b) Duplicate conversation keys: 0 rows where calls_90d > distinct_conversation_keys. All 67 rows have equal counts; total excess = Σ(calls_90d − distinct_conversation_keys) = 0. Raw conversation keys were not provided, so cross-deal duplication cannot be checked.

(c) Logged-call coverage among current DS3-or-later open deals: 29.4%.
Eligible deals: 61 DS3 + 14 DS4 + 10 DS5 = 85.
With at least one logged call: 12 DS3 + 6 DS4 + 7 DS5 = 25.
Share: 25 ÷ 85 × 100 = 29.4%.

Matching uses deal IDs, not gong_owner_id.
snowflake-sql-trapped-catalog0.50
Plain-English question: Write one Snowflake query for first-month usage per customer using only the provided, trap-laden catalog.
Prompt — what the model saw
Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0505 · 15s · in 3 / out 256 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✗ uses_cohort_summary✗ succeeded_filter✓ no_giver_deletion_filter✗ returns_sql

✓ Correct Answer (ground truth)

Plain-English answer: A single correct SQL query that uses the right catalog tables and avoids the trap columns — graded on the exact tables and columns.
The answer in one line: use_table = PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
avoid_stalePRODUCTION.HUBSPOT.DEALS (stale)
avoid_unpopulatedV2_LIVE.OBJECTS_DEALS (unpopulated)
forbidden_filterdeleted-giver exclusion must NOT be applied
outputone fenced SELECT with per-FROM/JOIN table-choice comment
required_filtersucceeded redemptions only
use_tablePRODUCTION.PLG.COMPANY_COHORT_SUMMARY
graded checksavoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Cannot write the requested query using only the provided catalog:
-- 1. No source documents customer company identifiers and customer start dates.
-- 2. No recognition-event source documents company, giver, and event timestamp columns.
-- 3. PRODUCTION.PLG.COMPANY_COHORT_SUMMARY covers only self-serve companies;
--    its company identifier is undocumented, and M1_* is not defined as the
--    first calendar month as a customer. M1_USERS is not documented as unique givers.
-- 4. PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2 is the documented
--    redemption source, but its company key and event timestamp are undocumented.
-- Required rules once those columns are supplied:
-- Count distinct givers and recognition events without excluding deleted givers.
-- Count redemption events only where STATE = 'succeeded'.
-- Use the customer-start calendar month, bounded by its start and the next month's start.
```
crm-completeness-enrichmentPASS 1.00
Plain-English question: Audit the CRM for missing and invalid fields, and propose the fixes.
Prompt — what the model saw
Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0875 · 80s · in 6 / out 2,898 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

Plain-English answer: 9 companies missing employee count, 6 missing HQ country, 4 invalid emails, 1 domain mismatch, 2 duplicate clusters, and 25 enrichment matches.
The answer in one line: n_contacts = 52; n_companies = 34; enrichment_matches = 25; invalid_email_count = 4; n_dup_clusters = 2; domain_mismatch_count = 1 (+4 more below)
n_contacts52
domain_mismatch_count1
enrichment_matches25
invalid_email_count4
n_companies34
n_dup_clusters2
company_missingindustry=0; employee_count=9; hq_country=6
domain_mismatchesCT-0011
duplicate_clusters['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
invalid_emailsCT-0010, CT-0080, CT-0081, CT-0192
graded checksdup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
Completeness

Percentages measure nonblank fields before deduplication or enrichment. Invalid values count as populated, not valid.

Record type / field        Populated / total × 100     Completeness
Companies / industry       34 / 34 × 100                100.00%
Companies / employee count 25 / 34 × 100                 73.53%
Companies / HQ country     28 / 34 × 100                 82.35%
Contacts / email           52 / 52 × 100                100.00%
Contacts / title           39 / 52 × 100                 75.00%
Contacts / persona         37 / 52 × 100                 71.15%

No deals extract was provided. Completeness for owner, stage, amount, close date, and why-buys is not calculable; it is not 0%. Pipeline exposure is also unavailable.

Duplicate company clusters

Company names are not provided, so name-variant detection is unavailable. Two candidate clusters share an exact domain:

Domain: acme-corp.com
  Members: C-0A092931, C-0A092932
  Proposed survivor: C-0A092931
  Industry: Technology versus tech; normalize to Technology.
  Employee count: 500 versus 510; unresolved.
  HQ country: US versus USA; equivalent country labels.
  No matching enrichment row exists. Retain survivor values provisionally and verify employee count before overwriting.

Domain: globex.io
  Members: C-0A092933, C-0A092934
  Proposed survivor: C-0A092934
  Industry: SaaS versus Technology; these represent different classification levels, not necessarily contradictory facts.
  Employee count: both 200.
  HQ country: both US.
  No matching enrichment row exists. Retain Technology as the broad industry and preserve SaaS as a classification note.

Survivors are proposed for consistent record handling, not because the extract proves they are more authoritative. Confirm entity identity before merging; preserve associations and conflicting source values.

Invalid emails and domain mismatches

Contact   Company    Invalid email   Expected company domain
CT-0010   C-66D1FC   user0@          66d1fc.com
CT-0080   C-92D97D   user0@          92d97d.com
CT-0081   C-92D97D   user1@          92d97d.com
CT-0192   C-425E2A   user2@          425e2a.com

These addresses lack an email domain. Verify the actual addresses; do not construct replacements from the company domain.

CT-0011, C-66D1FC: user1@other-domain.com is syntactically valid but differs from both the contact domain and company domain, 66d1fc.com. Verify affiliation or an approved alternate domain before changing it.

Email validity: (52 − 4) / 52 × 100 = 92.31%.
Syntactically valid, company-domain-matching emails: (52 − 4 − 1) / 52 × 100 = 90.38%. Neither measure proves deliverability.

No contact-domain field differs from its associated company’s domain.

Supported company fills

Each company below has a matching enrichment domain and a nonblank zi_employee_count of 400. Fill employee_count only:

  C-EC3025 → 400
  C-96039F → 400
  C-44EA29 → 400
  C-D04904 → 400
  C-B23205 → 400
  C-60C75F → 400
  C-7BBDFA → 400
  C-50D386 → 400

Employee-count completeness after these proposed fills:
(25 + 8) / 34 × 100 = 97.06%.

C-93C8BF remains missing employee_count: no matching enrichment row.

HQ country remains missing for:
  C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5: matching enrichment rows also lack HQ country.
  C-EE9FFB: no matching enrichment row.

No HQ-country fills are supported. Completeness remains 28 / 34 × 100 = 82.35%.

CRM versus enrichment differences

Industry differences:

Company     CRM industry   Enrichment industry
C-66D1FC    tech           Computer Software
C-EC3025    Technology     Computer Software
C-44EA29    tech           Computer Software
C-92D97D    Technology     Computer Software
C-D04904    Technology     Computer Software
C-77A95A    Technology     Computer Software
C-AA8DDA    Technology     Computer Software
C-B25F40    Technology     Computer Software
C-60C75F    tech           Computer Software
C-425E2A    Tech           Computer Software

C-425E2A’s raw CRM value is “Tech ”, including a trailing space.

Recommendation: use the matching enrichment row as the proposed source for the more specific Computer Software classification, subject to the CRM’s approved taxonomy. Preserve the original CRM value. Neither source includes freshness or verification metadata, so enrichment cannot be established as definitively more accurate.

HQ-country label differences:

Company     CRM country   Enrichment country
C-66D1FC    US            United States
C-950043    US            United States
C-EC3025    USA           United States
C-96039F    USA           United States
C-77A95A    US            United States
C-B23205    US            United States
C-E51FB7    USA           United States
C-D0662E    US            United States
C-425E2A    USA           United States
C-2D7423    USA           United States

These are formatting differences, not conflicting countries. Use United States from enrichment as the canonical display label.

No populated employee counts disagree between CRM and matching enrichment rows.

Additional industry standardization: trim whitespace and map tech / Tech to Technology; map health care to Healthcare. These are normalization changes, not enrichment fills.

Missing contact fields

Title:
  CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170.

Persona:
  CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181.

No supplied source supports filling these values. Obtain verified titles and role-specific personas; do not infer persona solely from title.

10 fixes requiring pipeline prioritization

The 10 fixes with the most pipeline amount at stake cannot be identified without deals, amounts, stages, and company/contact associations. The following are unranked fixes; pipeline amount at stake is unknown for every item.

1. Supply the deals extract and associations so missing required deal fields and pipeline exposure can be audited.
2. Verify the four invalid email addresses: CT-0010, CT-0080, CT-0081, CT-0192.
3. Verify CT-0011’s other-domain.com affiliation against C-66D1FC.
4. Confirm and consolidate C-0A092931 / C-0A092932, resolving the 500 versus 510 employee count.
5. Confirm and consolidate C-0A092933 / C-0A092934, preserving both industry classifications.
6. Apply the eight supported employee-count fills of 400.
7. Obtain employee_count for C-93C8BF and the six unresolved HQ-country values.
8. Obtain the 13 missing contact titles listed above.
9. Obtain the 15 missing contact personas listed above.
10. Reconcile industry differences and standardize industry/country labels while preserving source values.
deal-intelligence 5 tests
closed-lost-classification0.80
Plain-English question: Why did we lose the last six months of deals — one primary category per deal, and whose side is the reason on?
Prompt — what the model saw
Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0994 · 91s · in 9 / out 3,197 tok
✓ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

Plain-English answer: Every closed-lost deal gets exactly one category (pricing, competitor, no decision, timing, product gap, champion left, other) plus a side, per the computed classification.
The answer in one line: total_lost_6mo = 90; top_reason = competitor (26)
total_lost_6mo90
categoriespricing, competitor, no decision, timing, product gap, champion left, other
patternstwo most actionable, tag-vs-text disagreements counted
sideBonusly / buyer / unknown per deal
top_reasoncompetitor (26)
graded checkstotal_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Classification rules: explicit product limitations take priority over a competitor outcome; a dated deferral takes priority over a budget constraint. MIA/unresponsive maps to no decision, but does not prove the buyer’s underlying reason. Side means the stated source of the loss, not blame. Missing evidence is marked unknown.

Every deal is assigned below; deals sharing a row have the same category and side.

Category | Side | Deal aliases
timing | buyer | Deal-DB0AAC, Deal-91A056, Deal-29326C, Deal-831B7B, Deal-39E25C, Deal-B3ABED, Deal-B6AC09, Deal-E6E80A, Deal-B038F0, Deal-175756, Deal-BB78F3, Deal-15DA99, Deal-F4AF5D, Deal-79B7A1, Deal-9F176A, Deal-5E64CE, Deal-69CF3D, Deal-ECBF89, Deal-D1A623, Deal-FEDBCB, Deal-DAFB82, Deal-2FEDDB
timing | unknown | Deal-55867E
competitor | buyer | Deal-D0C698, Deal-47F1A1, Deal-BF2A98, Deal-369281, Deal-9FCD0D
competitor | Bonusly | Deal-F97C37, Deal-422BA6, Deal-DDAB52, Deal-A2C349
competitor | unknown | Deal-F7F635, Deal-381C8C, Deal-F1E8A6, Deal-ACE061, Deal-2D2F8D, Deal-0F96AA, Deal-1BCA50, Deal-7CC678, Deal-C7156E, Deal-8A0992, Deal-EECC02, Deal-1E7DA9, Deal-286F9C, Deal-64B19A
no decision | buyer | Deal-13E9CF, Deal-E74A73, Deal-8E27DA, Deal-FAC17C, Deal-50E5D8, Deal-413C56, Deal-2A292B, Deal-7FBAC6, Deal-F325A5, Deal-ABD14C
no decision | unknown | Deal-AC944F, Deal-214060, Deal-21B045, Deal-988493, Deal-F308CA, Deal-70F704, Deal-4664E1, Deal-D48E0B, Deal-583ADB, Deal-E0441F, Deal-7CB44D, Deal-AFA56C, Deal-D1AABF, Deal-2BBA21, Deal-386F6E, Deal-3F86A0, Deal-096750, Deal-79E61A, Deal-AE7C4E, Deal-DAB4F1, Deal-B4B50F, Deal-5885B9
pricing | buyer | Deal-7ED004, Deal-7B2236, Deal-C33D91
pricing | unknown | Deal-8A119B
product gap | Bonusly | Deal-242273, Deal-9048EB, Deal-3618CC, Deal-5AD03E, Deal-981AD4, Deal-DC77FE
other | buyer | Deal-ED9AE7
other | unknown | Deal-5DB9B0

Important evidence limits:
  Deal-55867E uses its timing tag; the text supplies no actual timeline.
  Deal-8A119B uses its budget/price tag; “Didn't get approval” does not establish why.
  Several competitor assignments rely on the tag alone. Deal-ACE061 and Deal-64B19A describe suspected, not confirmed, competitor choices.
  Deal-ED9AE7 lists timing, budget and authority without identifying a primary cause, so it is other.
  No text explicitly establishes that a buyer champion left. Deal-E0441F mentions a departed sales rep; Deal-F325A5 mentions leadership change, not a confirmed champion departure.

Category counts

  pricing:       4
  competitor:   23
  no decision:  32
  timing:       23
  product gap:   6
  champion left: 0
  other:         2

  Arithmetic: 4 + 23 + 32 + 23 + 6 + 0 + 2 = 90 deals.

Side split

  Bonusly: 10
  buyer:   41
  unknown: 39

  Arithmetic: 10 + 41 + 39 = 90 deals.

Clear tag-versus-text disagreements

  2 deals, using a conservative category-level test:

  Deal-8E27DA: “Feature Request” versus choosing only a swag provider and not wanting R&R currently. The text describes a buyer scope/priority decision, not a missing feature.

  Deal-3618CC: “Lost DM” versus “Wanted Surveys.” The text identifies a product requirement, not a decision-maker issue.

  Arithmetic: 1 + 1 = 2.

  Missing corroboration is not counted as disagreement. Competitor tags can coexist with product-gap explanations. Deal-9048EB explicitly supports both MIA and feature gaps. Timing-tag duration accuracy is separate from category disagreement; several texts do not substantiate “1 year or more.”

Two patterns most worth acting on

  1. Separate deferred demand from stalled engagement.
     Timing + no decision = 23 + 32 = 55 deals. Of the no-decision deals, 22 have unresponsive/MIA/no-contact text and unknown side. Use stated reconnect dates for deferred deals, and a separate requalification process for silent prospects. Do not treat silence as a known root cause.

  2. Qualify product breadth and specific requirements earlier.
     Product gaps + competitor losses explicitly tied to Bonusly’s offering = 6 + 4 = 10 deals. Examples include onsite points spending in Deal-242273, customization in Deal-DC77FE, surveys in Deal-3618CC and Deal-A2C349, broader offerings in Deal-F97C37 and Deal-DDAB52, and ADP partnership benefits in Deal-422BA6. These support targeted discovery and competitive positioning; they do not establish that every requirement warrants a product investment.
pipeline-tieringPASS 1.00
Plain-English question: Score all 156 open deals into exactly one of LOCK / ACTION / BUILD / REVIVE / WATCH / RISKY.
Prompt — what the model saw
Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.1171 · 56s · in 6 / out 2,102 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

Plain-English answer: All 156 open deals tiered exactly once across the six tiers using forecast category, stage, meetings, contacts, and recency.
The answer in one line: total = 156
total156
graded checksjson_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{"tier_counts":{"LOCK":3,"ACTION":23,"BUILD":29,"REVIVE":11,"WATCH":55,"RISKY":35},"tier_examples":{"LOCK":["Deal-D348E1","Deal-C26D20","Deal-403845"],"ACTION":["Deal-25F752","Deal-C6FE92","Deal-499BF6"],"BUILD":["Deal-D73B89","Deal-93C8BF","Deal-523604"],"REVIVE":["Deal-2D1F1B","Deal-950043","Deal-F0EBBB"],"WATCH":["Deal-6787C2","Deal-036E80","Deal-3EED2C"],"RISKY":["Deal-E53952","Deal-547B2B","Deal-B7EBD1"]},"risky_deals":["Deal-E53952","Deal-5408B0","Deal-9AAE5F","Deal-547B2B","Deal-B7EBD1","Deal-A2B47C","Deal-2465CE","Deal-C61CF7","Deal-62D607","Deal-584EE5","Deal-C6D97A","Deal-7B3B0F","Deal-F9A08A","Deal-0660B4","Deal-FD9F4E","Deal-BA571A","Deal-FC22A3","Deal-7BBDFA","Deal-60C2C2","Deal-4A13AD","Deal-8AD4A5","Deal-15D24F","Deal-9D0060","Deal-690476","Deal-635B8E","Deal-ED725A","Deal-55164C","Deal-3BA5EA","Deal-5FDCE4","Deal-F336B6","Deal-5EED42","Deal-BA3DDC","Deal-7599B8","Deal-F9A3C1","Deal-FA32A0"],"lock_violations":0,"pipeline_shape":"3+23+29+11+55+35=156 deals. Using September 4, 2026 as an assumed snapshot anchor, LOCK requires DS5 COMMIT, positive meetings, at least 3 contacts and engagement within 14 days; ACTION covers other meeting-positive DS3–DS5 deals and early-stage COMMITs, with at least 2 contacts and engagement within 30 days; BUILD covers remaining meeting-positive early-stage deals. RISKY flags COMMIT/BEST_CASE with zero meetings despite any outbound activity. PIPELINE deals without meetings are REVIVE when engagement is over 14 days old, otherwise WATCH. WATCH also includes Deal-3EED2C and Deal-57FF13 because engagement rows are missing, not assumed zero. Future-dated meetings are excluded from recency, while supplied meetings_30d counts are retained. The pipeline has a small supported commit core, substantial early-stage development, and 35 optimistic forecasts lacking meeting evidence; inbound email zeros are ignored as defective."}
call-transcript-extractionPASS 1.00
Plain-English question: Extract the CRM write-back fields from each call transcript as JSON.
Prompt — what the model saw
For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0643 · 45s · in 3 / out 1,431 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

Plain-English answer: Per transcript: prospect-only why-buys, pain points, stakeholders, budget and timeline signals, and competitors — e.g. TX-001: '$40k earmarked', live before open enrollment.
The answer in one line: rep = Alex Franklin
allowed_newTX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
per_transcript{"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
repAlex Franklin
rep_sourced_nulls{"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
graded checksjson_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "Automate anniversary and birthday awards."
    ],
    "pain_points": [
      "HR team of three cannot keep up with manual awards.",
      "Spreadsheet tracking lets people slip through the cracks."
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (HR Admin)"
    ],
    "budget_signal": "About $40k earmarked for engagement tools this fiscal year; not stated as dedicated to this purchase.",
    "timeline_signal": "Ideally live before open enrollment in November; year not stated.",
    "competitor_mentioned": [
      {
        "name": "Achievers",
        "context": "Evaluated last year; considered too heavy for a team their size."
      }
    ],
    "next_step": "Security review on September 12; prospect explicitly agreed. Year not stated.",
    "objections": [
      "SSO and audit logs required for IT sign-off."
    ],
    "confidence": "High: fields are directly supported by prospect statements; exact purchase budget and years are not stated."
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "Tie recognition to retention for the hourly workforce."
    ],
    "pain_points": [
      "Regretted turnover in the hourly workforce is over 30%."
    ],
    "stakeholders": [
      "Prospect (Head of Total Rewards)",
      "Prospect (CFO)"
    ],
    "budget_signal": "Finance approved a $25k pilot budget for this quarter.",
    "timeline_signal": "Decision wanted by end of September; year not stated.",
    "competitor_mentioned": [],
    "next_step": "Send the pilot agreement; prospect will route it to legal this week.",
    "objections": [
      "Rock-solid Workday integration is a condition."
    ],
    "confidence": "High: populated fields are directly supported by prospect statements; no competitor was named."
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "Make recognition visible across 12 retail locations."
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition."
    ],
    "stakeholders": [
      "Prospect (People Ops Manager)"
    ],
    "budget_signal": null,
    "timeline_signal": "No rush until Q1; year not stated.",
    "competitor_mentioned": [
      {
        "name": "Bucketlist",
        "context": "CEO used it at her last company and liked it; current evaluation not stated."
      }
    ],
    "next_step": "Schedule a call with the CEO; prospect agreed to send two times.",
    "objections": [
      "CEO must be sold first and decides anything people-related."
    ],
    "confidence": "High: populated fields are directly supported by prospect statements. Purchase budget is not stated; CEO is referenced but is not in the speaker list."
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "Consolidate three separate recognition tools into one."
    ],
    "pain_points": [
      "Paying for three tools, none of which connect to their HRIS."
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (IT Security Lead)"
    ],
    "budget_signal": "VP People can approve under $15k annually without board approval; this is an approval threshold, not a confirmed budget.",
    "timeline_signal": "Procurement takes six to eight weeks minimum; target purchase or launch date not stated.",
    "competitor_mentioned": [],
    "next_step": null,
    "objections": [
      "Security review took three months for the last vendor, creating hesitation."
    ],
    "confidence": "High: populated fields are directly supported by prospect statements. CFO follow-up was not agreed; no competitor was named."
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "Automate service milestones.",
      "Get analytics on recognition equity across departments."
    ],
    "pain_points": [
      "Night-shift teams feel invisible.",
      "Night-shift engagement scores run 20 points lower; comparison group not stated."
    ],
    "stakeholders": [
      "Prospect (HR Director)",
      "Prospect (People Ops Coordinator)"
    ],
    "budget_signal": "$12k approved under the engagement line.",
    "timeline_signal": "Running before the January all-hands; year not stated.",
    "competitor_mentioned": [
      {
        "name": "Nectar",
        "context": "Currently mid-pilot; prospect says the experience must be beaten."
      }
    ],
    "next_step": "Present to the executive team on October 2; prospect explicitly agreed. Year not stated.",
    "objections": [
      "Must beat the current Nectar pilot experience.",
      "Executive team is skeptical after a failed rollout two years ago."
    ],
    "confidence": "High: populated fields are directly supported by prospect statements; calendar years and engagement-score comparison group are not stated."
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "Cut administrative time on service awards."
    ],
    "pain_points": [
      "HR Manager personally spends five hours a month ordering and shipping plaques."
    ],
    "stakeholders": [
      "Prospect (HR Manager)"
    ],
    "budget_signal": "Prospect says budget is not the issue; no amount or approval stated.",
    "timeline_signal": "Fiscal year resets in February; prospect says a Q1 start is realistic. Year and Q1 calendar basis not stated.",
    "competitor_mentioned": [
      {
        "name": "doing it internally",
        "context": "Prospect is comparing against internal execution, not another vendor."
      }
    ],
    "next_step": "Send the one-page overview; prospect will forward it to the COO this week.",
    "objections": [
      "COO usually prefers building things in-house."
    ],
    "confidence": "High: populated fields are directly supported by prospect statements. No budget amount or external competitor was stated; COO is referenced but is not in the speaker list."
  }
]
next-to-close0.75
Plain-English question: Which 3 deals are closest to signature right now?
Prompt — what the model saw
Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0513 · 29s · in 3 / out 298 tok
✗ exactly_three✓ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

Plain-English answer: Deal-547B2B ($11,200), Deal-B7EBD1 ($9,000), and Deal-403845 ($9,000) — all DS5 COMMIT deals closing September 10-11, with what is left on each.
blocked_excludeddeal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
exactly_three1
ranking_ruleSlack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
top3rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
graded checksexactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
Closest to signature based on the supplied September 4 Slack thread and CRM:

1. Deal-547B2B — $11,200
   Why close: CRM shows DS5/COMMIT. Slack confirms clean redlines, signing page sent, and the VP People’s stated intent to sign September 5. Strongest explicit signature signal.
   What is left: Customer signature; no signed confirmation provided.

2. Deal-403845 — $9,000
   Why close: CRM shows DS5/COMMIT; Slack says the order form is with finance and the deal is moving.
   What is left: Finance review and signature; approval status is not provided. Ownership conflicts: CRM lists Alex Franklin, while Dana Mercer reports the update.

3. Deal-B7EBD1 — $9,000
   Why close: CRM shows DS5/COMMIT with a September 10 close date. This is a provisional third choice: Deal-2465CE has the same stage/date but is blocked per Slack; Deal-A2B47C and Deal-D348E1 remain in legal review.
   What is left: Unknown. No Slack update, approval status, or signature timing is provided.

Combined amount: $11,200 + $9,000 + $9,000 = $29,200.

These are snapshot-based picks, not verified current statuses. The supplied data does not establish a definitive third signature-imminent deal.
product-gap-detectionPASS 1.00
Plain-English question: Which prospect complaints are true product gaps versus rollout or enablement issues?
Prompt — what the model saw
Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0521 · 32s · in 3 / out 476 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

Plain-English answer: Two real gaps (SCIM provisioning, ADP Workforce Now integration) separated from plan gates and enablement issues, each with the prospect quote and deal.
allowed_newTG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
enablement_issuestranscript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voicetranscript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
plan_gatestranscript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
real_gapstranscript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
graded checksreal_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Active-deal status and deal amounts were not provided. The classifications below are supported, but none can be confirmed as an active-deal gap.

Candidate classifications

1. Deal-EC3025 — Real gap
   Prospect (IT Security Lead): “We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.”
   Docs: SCIM user provisioning is explicitly not listed as supported.
   Amount: Not provided.

2. Deal-D0D6B5 — Real gap
   Prospect (HRIS Manager): “Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.”
   Docs: ADP Workforce Now is explicitly not listed as supported.
   Amount: Not provided.

3. Deal-CFE7F4 — Plan gate
   Prospect (VP People): “I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?”
   Docs: The custom report builder is Enterprise-only. The prospect’s current plan and support for that specific combination of metrics are not provided.
   Amount: Not provided.

4. Deal-84DBA6 — Rollout/enablement issue
   Prospect (People Ops Manager): “We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.”
   Docs: Slack integration is supported on all plans. The prospect identifies missing training, not missing functionality.
   Amount: Not provided.

5. Deal-36C33F — Excluded: no prospect-raised gap
   Prospect (HR Manager): “Good to know. The web version should be fine for our office staff for now.”
   The native-mobile-app claim comes only from the rep. The docs excerpt does not establish mobile-app availability. No gap, plan-gate, or enablement classification is supported by prospect voice.
   Amount: Not provided.

Summary — real gaps only

- SCIM user provisioning: Deal-EC3025; security approval blocker. Amount missing.
- ADP Workforce Now integration: Deal-D0D6B5; explicitly called a dealbreaker. Amount missing.

Arithmetic: Total deal value cannot be calculated because both amounts are missing. Active-deal status remains unverified.
rep-performance 5 tests
stale-pipeline-by-rep0.67
Plain-English question: Which open deals have had no email, call, or meeting in the last 7 days?
Prompt — what the model saw
The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.1454 · 77s · in 6 / out 4,352 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ bryce_top_amount✓ cole_stale_amt✗ dana_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

Plain-English answer: 65 open deals are stale; the largest is Deal-2D1F1B at $240,000, with Bryce Harmon carrying the most stale dollars at $626,243.
The answer in one line: jake_amt = 626,243; jenn_amt = 261,645; kevin_amt = 252,905.03; top_amount = 240,000; stale_count = 65; top_alias = Deal-2D1F1B
jake_amt626,243
jenn_amt261,645
kevin_amt252,905.03
stale_count65
top_amount240,000
top_aliasDeal-2D1F1B
graded checksstale_recall_ge_80 · grouped_by_owner · bryce_top_amount · cole_stale_amt · dana_stale_amt · stale_count

Model's actual answer

show the model's full answer
As of 2026-09-05, stale means the latest logged email, call, or meeting on or before the snapshot is more than 7 days old: last contact before 2026-08-29.

Days since contact = 2026-09-05 − MAX(last_email, last_call, last_meeting), excluding future dates. Future-dated meetings do not count as completed contact. The deal’s last_contacted_field was not used. Currency was not provided.

Alex Franklin

Deal alias    Stage  Amount      Last contact  Days since contact
Deal-CC08D1   DS1    24,000.00   2026-08-20    16
Deal-E73427   DS3    18,000.00   2026-08-26    10
Deal-885F45   DS2     9,300.00   2026-08-24    12
Deal-C2FF3C   DS1     8,316.00   2026-08-26    10
Deal-0D2F7A   DS3     5,100.00   2026-08-24    12
Deal-6C60D4   DS3     4,800.00   2026-08-24    12
Deal-13FEBD   DS2     4,680.00   2026-08-24    12
Deal-819506   DS1     4,400.00   2026-08-28     8
Deal-9D0060   DS3     3,840.00   2026-08-24    12
Deal-690476   DS2     3,600.00   2026-08-18    18
Deal-C6D97A   DS4     3,240.00   2026-08-28     8
Deal-EE195F   DS3     3,120.00   2026-08-28     8
Deal-278DEC   DS3     2,700.00   2026-08-28     8
Deal-635B8E   DS3     2,600.00   2026-08-18    18
Deal-6883F3   DS1     2,400.00   2026-08-20    16
Deal-4A13AD   DS3     2,160.00   2026-08-10    26
Deal-F67D31   DS2     1,800.00   2026-08-28     8
Deal-5FDCE4   DS3     1,600.00   2026-08-24    12
Deal-BA571A   DS4     1,080.00   2026-08-18    18

19 stale deals.
Total = 24,000 + 18,000 + 9,300 + 8,316 + 5,100 + 4,800 + 4,680 + 4,400 + 3,840 + 3,600 + 3,240 + 3,120 + 2,700 + 2,600 + 2,400 + 2,160 + 1,800 + 1,600 + 1,080 = 106,736.00.

Bryce Harmon

Deal alias    Stage  Amount       Last contact  Days since contact
Deal-2D1F1B   DS1    240,000.00   2026-06-16    81
Deal-66D1FC   DS1     99,000.00   2026-08-20    16
Deal-950043   DS1     70,000.00   2026-08-17    19
Deal-B23205   DS1     45,000.00   2026-08-20    16
Deal-7BBDFA   DS3     37,440.00   2026-07-21    46
Deal-332637   DS2     36,000.00   2026-08-27     9
Deal-1BEEBF   DS1     31,500.00   2026-08-17    19
Deal-A414F6   DS1     25,200.00   2026-08-17    19
Deal-C5658B   DS1     23,400.00   2026-08-20    16
Deal-40522D   DS3     21,000.00   2026-08-17    19
Deal-C1FA6D   DS1     18,000.00   2026-08-20    16
Deal-01E193   DS1     12,600.00   2026-08-28     8
Deal-F0EBBB   DS3     11,400.00   2026-08-12    24
Deal-927338   DS1     10,920.00   2026-08-18    18
Deal-E25A09   DS1      6,000.00   2026-08-27     9
Deal-C9C286   DS2      5,502.00   2026-08-27     9
Deal-012CB1   DS1          1.00   2026-08-13    23
Deal-3795AD   DS2          1.00   2026-08-28     8

18 stale deals.
Total = 240,000 + 99,000 + 70,000 + 45,000 + 37,440 + 36,000 + 31,500 + 25,200 + 23,400 + 21,000 + 18,000 + 12,600 + 11,400 + 10,920 + 6,000 + 5,502 + 1 + 1 = 692,964.00.

Cole Ingram

Deal alias    Stage  Amount      Last contact  Days since contact
Deal-D04904   DS2    58,529.25   2026-08-25    11
Deal-B25F40   DS3    40,000.00   2026-08-28     8
Deal-813836   DS2    32,175.00   2026-08-25    11
Deal-1BA595   DS2    31,750.00   2026-08-25    11
Deal-CFE1E8   DS3    18,000.00   2026-08-25    11
Deal-CD47A6   DS2    12,168.00   2026-08-25    11
Deal-627646   DS3    11,193.00   2026-08-25    11
Deal-FF809F   DS2     7,781.20   2026-08-25    11
Deal-AF932D   DS2     7,225.40   2026-08-25    11
Deal-A71728   DS2     6,947.50   2026-08-25    11
Deal-8BC9F5   DS2     5,616.00   2026-08-26    10
Deal-175395   DS3     4,779.88   2026-08-25    11
Deal-481E24   DS3     4,140.00   2026-08-26    10
Deal-C7F9BF   DS2     3,360.00   2026-08-25    11
Deal-2F3A66   DS3     3,334.80   2026-08-25    11
Deal-342E96   DS2     2,700.00   2026-08-12    24
Deal-E568D5   DS3     1,875.00   2026-08-25    11
Deal-FD9F4E   DS5     1,330.00   2026-08-26    10

18 stale deals.
Total = 58,529.25 + 40,000 + 32,175 + 31,750 + 18,000 + 12,168 + 11,193 + 7,781.20 + 7,225.40 + 6,947.50 + 5,616 + 4,779.88 + 4,140 + 3,360 + 3,334.80 + 2,700 + 1,875 + 1,330 = 252,905.03.

Dana Mercer

Deal alias    Stage  Amount      Last contact  Days since contact
Deal-44EA29   DS2    60,000.00   2026-08-26    10
Deal-E51FB7   DS2    43,875.00   2026-08-24    12
Deal-B42F46   DS1    27,000.00   2026-08-17    19
Deal-BA3DDC   DS3    23,400.00   2026-08-21    15
Deal-9DDE86   DS2    20,000.00   2026-08-21    15
Deal-215CCA   DS3    18,900.00   2026-08-19    17
Deal-5EED42   DS3    16,250.00   2026-08-25    11
Deal-57887A   DS2    15,000.00   2026-08-28     8
Deal-944310   DS4    10,500.00   2026-08-03    33
Deal-B7EBD1   DS5     9,000.00   2026-08-20    16
Deal-3974EB   DS4     9,000.00   2026-08-28     8
Deal-F40F04   DS2     8,100.00   2026-08-21    15
Deal-7599B8   DS3     7,350.00   2026-08-18    18
Deal-87DDD1   DS1     5,000.00   2026-08-17    19
Deal-F336B6   DS3     4,200.00   2026-08-21    15
Deal-0660B4   DS4     1,920.00   2026-08-20    16

16 stale deals.
Total = 60,000 + 43,875 + 27,000 + 23,400 + 20,000 + 18,900 + 16,250 + 15,000 + 10,500 + 9,000 + 9,000 + 8,100 + 7,350 + 5,000 + 4,200 + 1,920 = 279,495.00.

Farid Osman

Deal alias    Stage  Amount     Last contact  Days since contact
Deal-8BA24E   DS1    2,880.00   2026-08-28     8
Deal-8FDCD2   DS1        1.00   2026-08-21    15

2 stale deals.
Total = 2,880 + 1 = 2,881.00.

Missing engagement data

These open deals have no row in engagements_by_deal_90d.csv. They therefore have no logged email, call, or meeting in the supplied table, but days since last contact cannot be calculated. They are excluded from the confirmed-stale totals above.

Owner           Deal alias    Stage  Amount    Days since contact
Alex Franklin   Deal-3EED2C   DS2    7,200.00  Unknown
Elena Sinclair  Deal-57FF13   DS1    2,100.00  Unknown

Elena Sinclair: 0 confirmed stale deals, confirmed stale amount 0; 1 deal with unknown recency, amount 2,100.00.
Alex Franklin additionally has 1 deal with unknown recency, amount 7,200.00.
activity-mix-vs-outcomePASS 1.00
Plain-English question: Per rep, what did the last 30 days of emails, calls, and meetings look like next to the deals that advanced?
Prompt — what the model saw
The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.1457 · 76s · in 12 / out 3,721 tok
✓ alex_ds2_30d✓ alex_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

Plain-English answer: Alex Franklin is the standout signal with 18 deals entering DS2 in 30 days (next closest: Bryce Harmon with 4), alongside per-rep activity totals.
The answer in one line: alex_ds2_30d = 18; bryce_ds2_30d = 4; note = legacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
alex_ds2_30d18
bryce_ds2_30d4
notelegacy alias justin_ds2=4 actually held Bryce Harmon count; Alex Franklin entered 18 deals into DS2 in the 30d window
graded checksalex_ds2_30d · alex_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Window: 2026-08-06 through 2026-09-05, inclusive (t_ds2 within 0–30 days of the snapshot). Activities use the supplied emails_30d, calls_30d, and meetings_30d fields, joined to deal owners.

Rank  Rep             Emails  Calls  Meetings  Total arithmetic     DS2 entries  Activities/DS2
1     Alex Franklin   304     36     41        304 + 36 + 41 = 381  18           381 / 18 = 21.17
2     Bryce Harmon    162      0     43        162 + 0 + 43 = 205    4           205 / 4 = 51.25
3     Cole Ingram      96     14      1         96 + 14 + 1 = 111    2           111 / 2 = 55.50
4     Farid Osman      38      0     34         38 + 0 + 34 = 72     1            72 / 1 = 72.00
5     Dana Mercer      84     18     11         84 + 18 + 11 = 113   1           113 / 1 = 113.00
—     Elena Sinclair  Missing activity data                         0           Undefined

Activity mix: each activity count / that rep’s total × 100.

Rep             Emails                       Calls                       Meetings
Alex Franklin   304 / 381 × 100 = 79.79%     36 / 381 × 100 = 9.45%      41 / 381 × 100 = 10.76%
Bryce Harmon    162 / 205 × 100 = 79.02%      0 / 205 × 100 = 0.00%      43 / 205 × 100 = 20.98%
Cole Ingram      96 / 111 × 100 = 86.49%     14 / 111 × 100 = 12.61%      1 / 111 × 100 = 0.90%
Farid Osman      38 / 72 × 100 = 52.78%       0 / 72 × 100 = 0.00%       34 / 72 × 100 = 47.22%
Dana Mercer      84 / 113 × 100 = 74.34%     18 / 113 × 100 = 15.93%     11 / 113 × 100 = 9.73%
Elena Sinclair  Undefined                    Undefined                   Undefined

DS2 entries counted:
  Alex Franklin: Deal-403845, Deal-1FC049, Deal-3EED2C, Deal-7FA0C3, Deal-E531A6, Deal-5296C9, Deal-36C33F, Deal-EE195F, Deal-F436DA, Deal-317E6F, Deal-D1E6C2, Deal-D9A72E, Deal-CA5E44, Deal-4F775F, Deal-898FC5, Deal-46988D, Deal-E73427, Deal-92D97D.
  Bryce Harmon: Deal-25F752, Deal-D73B89, Deal-CA7DC0, Deal-1CCE5C.
  Cole Ingram: Deal-42326B, Deal-1BA595.
  Farid Osman: Deal-499BF6.
  Dana Mercer: Deal-57887A.

Alex Franklin is both the most efficient rep by the requested ratio (21.17 activities per DS2 entry) and the highest-volume rep (381 recorded activities). They do not differ.

Data limits: Alex Franklin’s Deal-3EED2C and Elena Sinclair’s Deal-57FF13 have no engagement rows; missing activity is not treated as zero. Alex’s totals and ranking therefore reflect recorded activities only. Archived owners Gavin Porter and Hugo Lindqvist have no deals or associated engagements in the supplied data and cannot be ranked. Some last_meeting dates are after the snapshot; the supplied meetings_30d counts are used without adjustment because individual meeting dates are unavailable.
ae-qtd-scorecardPASS 1.00
Plain-English question: Build Alex Franklin's QTD scorecard: bookings versus quota, pipeline, and conversion.
Prompt — what the model saw
Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.1482 · 130s · in 9 / out 6,147 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

Plain-English answer: 75% attainment ($150,000 on a $200,000 quota), $113,500 new versus $36,500 expansion, pipeline by stage, and the rolling DS2-to-won rate.
The answer in one line: quota = 200000; bookings = 150000; new_bookings = 113500; expansion_bookings = 36500; open_deals_count = 125; attainment_pct = 75; owner = Alex Franklin (+12 more below)
quota200,000
attainment_pct75
bookings150,000
expansion_bookings36,500
expansion_count3
losses_qtd27
new_bookings113,500
new_count5
open_deals_count125
wins_qtd8
activity_30d_totalsemails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
allowed_new18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
ds2_to_wonwindow_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
ownerAlex Franklin
pipeline_by_stage{"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
prior_quarter_excludeddeal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
quarter2026-Q3
snapshot_date2026-09-05
top_loss_reasonreason=Lost- Timing (1 year or more); count=13
graded checksbookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
Alex Franklin — QTD scorecard as of 2026-09-05

Period: 2026-07-01–2026-09-05, assuming calendar Q3. Amounts use the source’s unspecified currency.

Bookings vs quota
  Bookings: 150,000 across 8 wins
  Quota: 200,000
  Attainment: 150,000 ÷ 200,000 × 100 = 75%
  Remaining: 200,000 − 150,000 = 50,000

New vs expansion
  New: 5 wins, 113,500
    Deal-A1C3E5 40,000 + Deal-B7D2F4 35,000 + Deal-C9E1A6 21,000
    + Deal-D4B8C2 11,000 + Deal-E6F3A9 6,500 = 113,500
    Share: 113,500 ÷ 150,000 × 100 = 75.67%

  Expansion: 3 wins, 36,500
    Deal-F2C7D8 20,000 + Deal-A8B4D6 12,000 + Deal-C5D9E2 4,500 = 36,500
    Share: 36,500 ÷ 150,000 × 100 = 24.33%

  Excluded: Deal-B3E6F1, 24,000, closed-won 2026-06-20, before Q3.

Active pipeline
All supplied open deals, including those with expected close dates after Q3.

  Stage    Deals       Amount
  DS1         20      284,621
  DS2         28      353,760
  DS3         67      552,705
  DS4          5       23,574
  DS5          5       45,730
  Total      125    1,260,390

  Amount arithmetic: 284,621 + 353,760 + 552,705 + 23,574 + 45,730
  = 1,260,390.

Rolling 90-day DS2-to-won rate
Window: 2026-06-08–2026-09-05, inclusive. Defined as deals entering DS2 in that window that were won by the snapshot date, divided by all deals entering DS2 in that window.

  DS2-entry cohort: 8 won + 27 lost + 75 still open = 110 deals
  DS2-to-won rate: 8 ÷ 110 × 100 = 7.27%

  The 75 open deals remain in the denominator; this is conversion to date, not a fully matured cohort rate.

QTD wins and losses
  Wins: 8
  Losses: 27
  Top loss reason by count: Lost- Timing (1 year or more)
  Frequency: 13 ÷ 27 × 100 = 48.15% of losses

Last-30-day activity
Using the supplied 30-day aggregates and excluding Deal-B3E6F1 as requested. Underlying activity dates are not supplied.

  Type       Lost deals + Open deals + QTD wins = Total
  Emails            109 + 599        + 89       = 797
  Calls              25 + 54         + 31       = 110
  Meetings           13 + 90         + 23       = 126
  Notes              25 + 1          + 21       = 47

  Total: 797 + 110 + 126 + 47 = 1,080 activities.

Three coaching observations
  1. Focus the quarter-end plan on the 50,000 quota gap. Open pipeline scheduled within Q3 totals 109,363, or 109,363 ÷ 50,000 = 2.19× the gap. Validate close plans rather than treating all 1,260,390 of active pipeline as Q3 coverage. Deal-7A2454 remains open despite its 2026-09-04 expected close date.

  2. Tighten timing qualification early. Lost- Timing (1 year or more) accounts for 13 of 27 losses, or 48.15%. Confirm a funded buying window before advancing deals.

  3. Review engagement quality on open opportunities. Of 125 open deals, 62 have neither calls nor meetings recorded in the last 30 days. Deal-547B2B has 10 emails but no calls or meetings despite being in DS5 with an expected 2026-09-11 close; confirm buyer commitment and the next live conversation.
multithreading-gap0.25
Plain-English question: Which open deals are single-threaded or under-threaded?
Prompt — what the model saw
Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0530 · 31s · in 3 / out 300 tok
✗ flagged_recall✓ persona_language✗ single_thread_term✗ on_file_fit

✓ Correct Answer (ground truth)

Plain-English answer: Deals with fewer than 2 active contacts (single-threaded) or fewer than 3, or all in one persona (under-threaded), each flagged with the missing persona.
The answer in one line: active_window_start = 2026-07-07
active_window_start2026-07-07
allowed_new59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
best_persona_ruleDS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flag_ruleflag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
flaggeddeal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flaggedDeal-4B0BEB, Deal-84DBA6, Deal-D348E1
persona_universeeconomic buyer, champion, HR admin, IT security, finance
graded checksflagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Missing data prevents a reliable deal-by-deal report:

- Open/closed status: no deal-status data was provided.
- Amount and stage: neither field was provided.
- As-of date: required to calculate the 60-day engagement window. The latest engagement date is not necessarily the reporting date.

Please provide the as-of date and each deal’s open/closed status, amount, and stage. Without these, I cannot identify every open deal, calculate active contact counts reliably, or recommend the most valuable persona given its stage.
talk-track-from-transcriptsPASS 1.00
Plain-English question: From this rep's last 10 calls: how do they open, handle objections, and how often do they lock a next step?
Prompt — what the model saw
From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0650 · 43s · in 6 / out 996 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

Plain-English answer: 8 of 10 calls open with the same customer-win story; objection handling, next-step agreement rate, and every competitor raised, with quotes.
The answer in one line: n_transcripts = 10; objection_count = 3; rep = Alex Franklin (+3 more below)
n_transcripts10
objection_count3
allowed_new13 items — e.g. TT-001, TT-002, TT-003, TT-004
competitor_quotesAwardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
competitors_prospect_raisedAwardco, Kudos
next_stepagreed=7; total=10; rate=0.7
objectionsobjection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
opener_themetheme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
repAlex Franklin
rep_raised_competitors_excludedWorkhuman
graded checksobjection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
First five minutes

Alex Franklin leads with a retailer turnover-results story in 8/10 calls = 80%: Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-EDC141, Deal-D9A12F, and Deal-84DBA6. He cites an 18% reduction in regretted turnover after automating milestone awards. This is a claim in the transcripts, not independently verified evidence.

The exceptions are Deal-403845, opening with security review then pricing, and Deal-1E2498, opening with pricing at the prospect’s prior request: 1/10 = 10% each.

Three most common objections and his response

1. Budget/approval: 5/10 calls = 50%.
   Locked budget appears in Deal-D348E1, Deal-547B2B, Deal-60C2C2, and Deal-84DBA6; Deal-403845 adds a committee-approval objection. Counting each call once: 4 + 1 = 5.
   He counters locked budgets with claimed turnover savings and avoided backfill costs. When committee approval blocks progress in Deal-403845 and Deal-84DBA6, he accepts the delay without securing a next step.
   Quote, Deal-D348E1, minute 8: “Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off.”

2. Timing/workload: 3/10 calls = 30%.
   Deal-5408B0, Deal-C61CF7, and Deal-D9A12F ask to revisit next quarter because of open enrollment. He proposes a smaller pilot to generate internal evidence before planning.
   Quote, Deal-5408B0, minute 8: “Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?”

3. Existing spreadsheet/gift-card process: 3/10 calls = 30%.
   Deal-403845, Deal-EDC141, and Deal-1E2498 question why they should change. He emphasizes automated milestones and recognition analytics rather than probing the current process’s specific shortcomings.
   Quote, Deal-403845, minute 8: “Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized.”

Categories overlap; a call can contain more than one objection.

Concrete next-step agreement rate

7/10 calls × 100 = 70%.

Agreed: Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-D9A12F, and Deal-1E2498. Each prospect accepts Thursday at 2pm and says they will bring their HRIS manager.

Quote, Deal-D348E1, minute 15: “Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager.”

No concrete next step is recorded for Deal-403845, Deal-EDC141, or Deal-84DBA6. The transcripts do not establish whether invites were sent or meetings occurred.

Every competitor raised by a prospect

- Awardco — Deal-547B2B, minute 4: “We're also in late talks with Awardco — their rewards catalog looks bigger than yours.”
- Kudos — Deal-EDC141, minute 4: “How are you different from Kudos? Our CEO used them at her last company.”

Workhuman was raised by Alex Franklin in Deal-C61CF7, not by a prospect, so it is excluded.

Two coaching notes

1. After the opening story, ask for the prospect’s own turnover, administrative workload, or recognition gaps before prescribing ROI or automation. The supplied excerpts show repeated claims, not prospect-specific validation.
2. When approval or urgency stalls progress, identify the decision owner and agree on a dated checkpoint. Deal-403845, Deal-EDC141, and Deal-84DBA6 end without a concrete commitment.
reporting-analytics 5 tests
q3-weighted-forecastPASS 1.00
Plain-English question: Build the Q3 2026 weighted forecast from the open pipeline, and show the math deal by deal.
Prompt — what the model saw
Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0798 · 44s · in 6 / out 2,100 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

Plain-English answer: A stage-weighted Q3 forecast number computed from the open pipeline, with per-stage math that reconciles to the deal data.
The answer in one line: excluded_amount = 227575; best_case_total = 203565; weighted_forecast = 115,976.75; commit_total = 44729; excluded_out_of_quarter = 32; best_case_count = 24; flags = blank owner column + empty why_buys fields must be surfaced (+2 more below)
excluded_amount227,575
best_case_count24
best_case_total203,565
commit_count7
commit_total44,729
excluded_out_of_quarter32
weighted_forecast115,976.75
flagsblank owner column + empty why_buys fields must be surfaced
top5_best_caseDeal-2D7423, Deal-25F752, Deal-E53952, Deal-5EED42, Deal-FA32A0
graded checkscommit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
Q3 2026 forecast
Close-date filter: 2026-07-01 through 2026-09-30, inclusive. Amounts are in the extract’s unspecified currency.

Category       Deals   Total amount   Weight   Forecast contribution
COMMIT             7      44,729.00     100%               44,729.00
BEST_CASE         24     203,565.00      35%               71,247.75
PIPELINE          23     201,637.40       0%                    0.00

Weighted forecast = 44,729 + (203,565 × 0.35) + (201,637.40 × 0)
                  = 44,729 + 71,247.75 + 0
                  = 115,976.75

COMMIT arithmetic:
11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = 44,729.

BEST_CASE arithmetic:
38,935 + 24,000 + 19,656 + 16,250 + 11,116 + 10,800 + 10,500 + 9,890 + 9,720 + 9,000 + 7,200 + 3,840 + 3,780 + 3,600 + 3,240 + 3,150 + 3,120 + 3,060 + 2,916 + 2,760 + 2,484 + 2,100 + 1,920 + 528 = 203,565.

Top 5 BEST_CASE deals inside Q3

Deal alias      Amount   Close date
Deal-2D7423     38,935   2026-09-30
Deal-25F752     24,000   2026-09-25
Deal-E53952     19,656   2026-09-30
Deal-5EED42     16,250   2026-09-30
Deal-FA32A0     11,116   2026-09-25

Excluded: outside Q3

32 deals, totaling 227,575; all close in October 2026.

Deal alias      Close date   Amount
Deal-E51FB7     2026-10-01   43,875
Deal-B936FE     2026-10-09   18,000
Deal-D9A12F     2026-10-15   17,000
Deal-D348E1     2026-10-15   13,770
Deal-4062CF     2026-10-15   10,800
Deal-293AF3     2026-10-09    9,000
Deal-034D49     2026-10-15    9,000
Deal-E0ADD8     2026-10-15    7,920
Deal-9F2E43     2026-10-08    7,690
Deal-FCBE5B     2026-10-07    7,500
Deal-712010     2026-10-15    7,200
Deal-6691E0     2026-10-15    5,700
Deal-C61CF7     2026-10-09    5,400
Deal-600CD9     2026-10-02    5,400
Deal-A92065     2026-10-15    5,400
Deal-1D532E     2026-10-15    5,400
Deal-48B656     2026-10-15    5,160
Deal-E531A6     2026-10-15    4,800
Deal-D1E6C2     2026-10-09    4,400
Deal-D9E112     2026-10-09    4,300
Deal-5AD94B     2026-10-15    4,000
Deal-901332     2026-10-15    3,600
Deal-47AE31     2026-10-09    3,600
Deal-15D24F     2026-10-09    3,600
Deal-766C74     2026-10-14    3,300
Deal-ED725A     2026-10-08    2,400
Deal-8AD4A5     2026-10-07    1,800
Deal-D7E999     2026-10-15    1,800
Deal-ED13B0     2026-10-09    1,680
Deal-5FDCE4     2026-10-01    1,600
Deal-7FA0C3     2026-10-01    1,400
Deal-F5A622     2026-10-08    1,080

Excluded total arithmetic by category:
13,770 COMMIT + 28,240 BEST_CASE + 185,565 PIPELINE = 227,575.

## Data quality
Owner is blank except for Deal-C9C286, and most deals have why_buys_chars = 0, leaving ownership and buying rationale largely undocumented in this extract.
Deal-A5E80A is DS1 but COMMIT, and Deal-499BF6 is DS2 but COMMIT; these early-stage commitments need validation, though the requested calculation retains their categories.
Deal-333EBB, Deal-57FF13, Deal-31AD2C, and Deal-7A2454 remain open with close dates before the 2026-09-05 extraction date, indicating stale dates or statuses.
Currency and closed-won Q3 bookings are missing, so this is an open-deal weighted forecast, not a complete Q3 bookings forecast.
aha-moment-2x20.83
Plain-English question: Do companies that hit 5+ givers and one redemption in month one actually retain better at 24 months?
Prompt — what the model saw
Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.1171 · 68s · in 9 / out 3,333 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✗ largest_lift_givers

✓ Correct Answer (ground truth)

Plain-English answer: The four cohort cells are graded against computed 24-month retention; the numbers come from the scorer's cohort table, not opinion.
The answer in one line: both_5plus_givers_and_redemption = 66.0% (31 of 47)
both_5plus_givers_and_redemption66.0% (31 of 47)
exclusionsnone — claim bounded as correlation, not causation
givers_only46.9% (23 of 49)
largest_single_signal_liftunique givers
neither40.0% (38 of 95)
redemption_only31.0% (9 of 29)
graded checksboth_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
The hypothesis is supported descriptively: companies with both first-month signals had the highest 24-month retention.

Full 2x2
                                 Redemption ≥1             Redemption =0
Givers ≥5                        Both signals              Givers-only
                                 n=47; 31/47 = 65.96%      n=49; 23/49 = 46.94%

Givers <5                        Redemption-only           Neither
                                 n=29; 9/29 = 31.03%       n=95; 38/95 = 40.00%

Both-signals retention exceeded:
  Givers-only: 65.96% − 46.94% = 19.02 percentage points.
  Redemption-only: 65.96% − 31.03% = 34.92 percentage points.
  Neither: 65.96% − 40.00% = 25.96 percentage points.

Denominator and exclusions
  Included: 47 + 49 + 29 + 95 = 220 companies.
  Excluded: 0. All companies meet the stated age requirement and have the required signal and status fields. Missing industry or country does not affect eligibility.
  Retained means current_status = 'active': 31 + 23 + 9 + 38 = 101 companies. The 116 cancelled and 3 non_renewing companies remain in the denominator and are not counted as retained under the specified definition.

Single signal with the largest retention lift
  The 5+ unique-giver signal.
  Comparing single-signal-only groups with neither:
    Givers-only: 23/49 − 38/95 = +6.94 percentage points.
    Redemption-only: 9/29 − 38/95 = −8.97 percentage points.

  The same ranking holds when measuring each signal across the full cohort, including companies with both:
    Givers: 54/96 − 47/124 = 56.25% − 37.90% = +18.35 percentage points.
    Redemption: 40/76 − 61/144 = 52.63% − 42.36% = +10.27 percentage points.

What this does and does not prove
  This extract establishes an observed association: both signals identify the highest-retention group, and giver breadth is the stronger standalone signal. Redemption alone does not outperform neither.

  It does not establish causation, statistical significance, an optimal activation threshold, or generalizability beyond this cohort. Company differences could explain some or all of the association; inducing these behaviors is not proven to improve retention.
arr-rollforward-reconciliationPASS 1.00
Plain-English question: Does CRM ARR match billing ARR — and if not, exactly where is the variance?
Prompt — what the model saw
Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0709 · 38s · in 6 / out 1,651 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

Plain-English answer: Billing $604,739.28 vs CRM $603,581.76: a $1,157.52 variance decomposed into status-mismatch, rounding, and missing-record buckets.
The answer in one line: n_companies = 40; n_mismatched_accounts = 7; as_of = 2026-09-05 (+4 more below)
n_companies40
n_mismatched_accounts7
as_of2026-09-05
billing_arr_total604739.28
bucket_sum_equals_variance1
buckets{"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
crm_arr_total603581.76
ruleterm_months != 12 requires cf_agreement_end_date
variance1157.52
variance_definitionbilling_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
violationssubscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
graded checksbilling_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
ARR reconciliation as of 2026-09-05

Basis: Treat supplied statuses as the snapshot for this date. Billing ARR includes active subscriptions only; cancelled subscriptions contribute zero. Historical status dates and currency are not provided.

CRM ARR:                         603,581.76
Active billing MRR:               50,394.94
Billing ARR: 50,394.94 × 12 =     604,739.28
Variance (CRM − billing):         −1,157.52

Variance decomposition

Status mismatch:   4,905.24 + 8,253.24 =             +13,158.48
Rounding*:         16.00 + 20.00 =                       +36.00
Missing records:   16,497.24 − 28,449.24 =            −11,952.00
Other:             24,396.00 − (2,233.00 × 12) =       −2,400.00
Total:             13,158.48 + 36.00 − 11,952.00
                   − 2,400.00 =                       −1,157.52

*The two small differences are consistent with rounding, but the data does not establish their cause. If unconfirmed rounding must remain “other,” rounding is 0.00 and other is −2,364.00; the total is unchanged.

Mismatched accounts

Company alias   CRM ARR    Billing ARR   CRM − billing  Issue / suggested owner
C-0C8323BF      4,905.24          0.00       +4,905.24   Cancelled SUB-000E retains CRM ARR. RevOps; Billing to confirm status.
C-0DC4FB8C      8,253.24          0.00       +8,253.24   Cancelled SUB-000F retains CRM ARR. RevOps; Billing to confirm status.
C-0D66DF9E     23,200.00     23,184.00          +16.00   Possible rounding: 23,200.00 − (1,932.00 × 12). RevOps.
C-14D70CE0     18,200.00     18,180.00          +20.00   Possible rounding: 18,200.00 − (1,515.00 × 12). RevOps.
C-21629AA4      Missing     28,449.24      −28,449.24   Active SUB-0004; no CRM record supplied. RevOps.
C-0D5BBE3A     16,497.24      Missing      +16,497.24   No billing record supplied; treated as zero in bridge. Billing / Finance.
C-0F7269D7     24,396.00     26,796.00       −2,400.00   Unexplained ARR difference. RevOps / Billing.

Owners are suggested functions; no named account owners were provided. Missing records mean absent from the supplied files, not proven absent from the systems.

Agreement-end-date violations

Subscription   Company alias   Term        Violation                              Suggested owner
SUB-0002       C-1794A52C       24 months   cf_agreement_end_date is blank         Billing / Deal Desk
SUB-0019       C-22170CA1       36 months   cf_agreement_end_date is blank         Billing / Deal Desk
kvm-month-over-monthPASS 1.00
Plain-English question: How did the four core KVMs move in August versus July?
Prompt — what the model saw
Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0898 · 68s · in 9 / out 2,624 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

Plain-English answer: Giving rate 60.27% (up 0.07%), redemptions per user 1.7302 (up 0.01%), with 1:1 meetings and pulse engagement each up fractionally — direction reported for all four.
The answer in one line: n_company_rows_per_month = 30; aggregation = simple mean of company-level values per month
n_company_rows_per_month30
aggregationsimple mean of company-level values per month
biggest_moverpulse_engagement
driving_segmentsize_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
kvms{"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
segment_detail{"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
graded checksfour_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Company-equal averages across 30 companies per month; user-weighted KVMs cannot be calculated because user counts are missing.

| KVM | 2026-08 | 2026-07 | Absolute change | Relative change | Direction |
|---|---:|---:|---:|---:|---|
| Giving rate | 60.2713% | 60.2297% | +0.0417 pp | +0.0692% | Up |
| Redemptions per user | 1.730163 | 1.729983 | +0.000180 | +0.0104% | Up |
| 1:1 meetings engagement | 44.7177% | 44.6887% | +0.0290 pp | +0.0649% | Up |
| Pulse check engagement | 50.8610% | 60.0587% | −9.1977 pp | −15.3145% | Down |

Arithmetic: monthly value = sum ÷ 30. July → August sums:
  Giving: 18.0689 → 18.0814
  Redemptions: 51.8995 → 51.9049
  1:1: 13.4066 → 13.4153
  Pulse: 18.0176 → 15.2583

Absolute change = August − July; relative change = absolute change ÷ July × 100. Rate changes are in percentage points (pp).

Pulse check engagement had the largest relative move. size_band=enterprise drove it: 54.998% → 27.428%, or −27.570 pp; −27.570 ÷ 54.998 × 100 = −50.1291%. Its contribution was −27.570 × 10/30 = −9.190 pp of the overall −9.1977 pp decline. All companies are tier_three, so plan-tier differentiation is unsupported.
rewards-ytd-sectionPASS 1.00
Plain-English question: Produce the redemption section of the weekly rewards report through August.
Prompt — what the model saw
Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.1341 · 80s · in 6 / out 3,672 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

Plain-English answer: 378 redemptions, $27,846 spend, 235 unique redeemers at 1.61 each, custom rewards at 39% of spend, US leading all countries.
The answer in one line: redemption_count = 378; unique_redeemers = 235; redemptions_per_redeemer = 1.61; total_spend_usd = 27846.00
redemption_count378
redemptions_per_redeemer1.61
unique_redeemers235
provider_mix_pct_of_spendTangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_countcountry=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
total_spend_usd27846.00
graded checkscount · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Redemptions: January–August 2026
Last completed month: August 2026 (2026-08).

Redemption count: 378 = 90 + 192 + 37 + 59.
Spend: $27,846 = $3,230 + $8,505 + $10,873 + $5,238.
Unique redeemers: 235 distinct user_key values across all providers.
Redemptions per redeemer: 378 ÷ 235 = 1.61.

Provider mix by spend
  TangoCard: $3,230 ÷ $27,846 × 100 = 11.60%
  Tremendous: $8,505 ÷ $27,846 × 100 = 30.54%
  custom: $10,873 ÷ $27,846 × 100 = 39.05%
  Snappy: $5,238 ÷ $27,846 × 100 = 18.81%
  Total: 11.60% + 30.54% + 39.05% + 18.81% = 100.00%.

Top 5 countries by redemptions
  1. US: 244
  2. CA: 24
  3. AU: 21
  4. GB: 17 (tied)
  4. NL: 17 (tied)

Source: redemptions_ytd.csv.
customer-success 5 tests
churn-save-eligibilityPASS 1.00
Plain-English question: Which at-risk accounts actually qualify for a churn-save offer, and which play fits each?
Prompt — what the model saw
Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0759 · 55s · in 9 / out 1,901 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

Plain-English answer: Only accounts meeting all three rules qualify — e.g. C-0F6C0F34 at $49,707 (executive touch) and C-0B827671 at $25,365 (usage revival) — with amount and play per account.
The answer in one line: n_accounts = 30; snapshot_date = 2026-09-05
n_accounts30
eligibleaccount_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
non_eligible_at_riskaccount_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
ruleshealth_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
snapshot_date2026-09-05
total_amount_at_stake224601.00
graded checkseligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Eight accounts qualify, with 224,601.00 in churn-save eligible amount at stake. Their full ARR is 454,380.00; that is distinct from the eligible amount. Currency is not specified.

Eligibility requires all three rules: health_score < 60, churn_save_eligible_amount > 0, and renewal within 120 days of 2026-09-05, through 2027-01-03.

Qualifying accounts

Account      | Health | Days to renewal | Eligible amount | Full ARR | Best-fit play and signal
C-0F6C0F34 | 51 | 28 | 49,707.00 | 86,741.00 | Executive touch: champion_active=false despite growing usage.
C-0B827671 | 56 | 70 | 25,365.00 | 72,088.00 | Usage revival: declining usage; seats used = 113 ÷ 202 × 100 = 55.94%.
C-0B360C78 | 57 | 53 | 35,748.00 | 60,427.00 | Commercial concession, conditional: low health despite growing usage, active champion, and 246 ÷ 327 × 100 = 75.23% seat utilization. No pricing or budget objection is documented; validate a commercial cause before offering a concession.
C-0B0F1BAB | 38 | 18 | 5,494.00 | 15,391.00 | Executive touch: champion_active=false and renewal in 18 days.
C-0CA21961 | 58 | 114 | 16,829.00 | 31,501.00 | Usage revival: flat usage and only 84 ÷ 325 × 100 = 25.85% seat utilization.
C-0E9C27D1 | 39 | 19 | 41,235.00 | 75,093.00 | Commercial concession, conditional: low health and renewal in 19 days despite an active champion and 134 ÷ 157 × 100 = 85.35% seat utilization. No pricing or budget objection is documented; validate a commercial cause first.
C-0CEF69FD | 53 | 77 | 32,621.00 | 79,324.00 | Executive touch: champion_active=false despite growing usage.
C-0D3278C7 | 54 | 68 | 17,602.00 | 33,815.00 | Usage revival: declining usage; seats used = 126 ÷ 380 × 100 = 33.16%.

Days to renewal = renewal_date − 2026-09-05. Play assignments are analytical recommendations; the provided rules define eligibility, not play selection or concession size.

Total eligible amount:
49,707 + 25,365 + 35,748 + 5,494 + 16,829 + 41,235 + 32,621 + 17,602 = 224,601.00.

Total full ARR:
86,741 + 72,088 + 60,427 + 15,391 + 31,501 + 75,093 + 79,324 + 33,815 = 454,380.00.

At-risk accounts that do not qualify

All have health_score < 60 but fail one or both remaining rules.

Account | Health | Eligible amount | Days to renewal | Why excluded
C-0BC71BDD | 55 | 0.00 | 52 | Fails R2: eligible amount is zero.
C-0BA71F12 | 52 | 6,824.00 | 218 | Fails R3: renewal exceeds 120 days.
C-0F6694C3 | 43 | 0.00 | 197 | Fails R2 and R3: zero eligible amount and renewal exceeds 120 days.
C-0BE96399 | 54 | 0.00 | 54 | Fails R2: eligible amount is zero.
C-0F876796 | 47 | 19,958.00 | 154 | Fails R3: renewal exceeds 120 days.
C-0FCCD2DF | 43 | 0.00 | 230 | Fails R2 and R3: zero eligible amount and renewal exceeds 120 days.
C-10A56B0F | 54 | 0.00 | 98 | Fails R2: eligible amount is zero.
expansion-kitPASS 1.00
Plain-English question: Build the expansion kit for the customer who just replied to our outreach.
Prompt — what the model saw
A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0521 · 22s · in 6 / out 474 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

Plain-English answer: 37.5% seat coverage, 250 headroom seats worth $15,000 ARR at $60 a seat, usage up 43.2% in six months — and the replier cannot buy; Dana R. (VP People) is the right buyer.
The answer in one line: headroom_seats = 250; coverage_pct = 37.5; company_alias = C-0DDFC9A7 (+3 more below)
headroom_seats250
coverage_pct37.5
company_aliasC-0DDFC9A7
headroom_arr15000.00
per_seat_annual_rate60.00
replierMaria S.
replier_can_buy0
right_buyername=Dana R.; title=VP People
usage_factsactive_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84
usage_health_linesMonthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checkscoverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
C-0DDFC9A7

Seat coverage: 150 licensed ÷ 400 headcount × 100 = 37.5%.

Usage health:
Monthly active users rose every month, from 88 in March to 126 in August: +38 users; 38 ÷ 88 × 100 = 43.2% growth.
August utilization: 126 ÷ 150 × 100 = 84%, leaving 150 − 126 = 24 licensed seats without monthly activity. Recognition frequency is reported by Maria S., not independently quantified.

Expansion headroom:
  Seats: 400 − 150 = 250 additional seats.
  Current per-seat annual rate: $9,000 ÷ 150 = $60.
  Incremental ARR at full headcount coverage: 250 × $60 = $15,000.
  Total potential ARR: $9,000 + $15,000 = $24,000.
This assumes all employees are eligible and the current rate holds; neither is confirmed.

Respondent: Maria S., People Operations Coordinator, last engaged 2026-09-02. She explicitly says she does not own purchasing decisions and offers an introduction.

Right buyer: Dana R., VP People, last engaged 2026-05-18. Maria S. identifies her as responsible for budget and seat expansion.

Reply email

Subject: RE: Growing your team's recognition program

Hi Maria,

Thanks for the context. Monthly active users grew from 88 in March to 126 in August, so there’s a useful usage story to share with Dana R.

If you’re comfortable making an introduction, I’d appreciate it. I can send Dana a short usage summary and current seat coverage so she can assess whether any expansion makes sense. No need to decide on additional seats now.

Thanks,
Cole
onboarding-statusPASS 1.00
Plain-English question: What is done and what is missing in this customer's onboarding before the mid-onboarding call?
Prompt — what the model saw
Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0595 · 23s · in 9 / out 663 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

Plain-English answer: 25 days live: Slack connected, allowance set, admins added — but the HRIS integration is not connected, with early engagement signals and three call priorities.
The answer in one line: days_live = 25; company_alias = C-0D284E42
days_live25
call_itemsConnect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
checklist{"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
company_aliasC-0D284E42
engagement_signalsavg_active_givers_week1=4.3; avg_active_givers_last_week=13; latest_active_givers=15; days_with_activity=25
graded checkscomplete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
C-0D284E42 — mid-onboarding call prep

Complete, supported by onboarding_account.csv:
- Slack connected: integration_slack = 2026-08-12.
- Allowance set: allowance_set = 2026-08-13. Amount not provided.
- Admins added: admins_added = 2.
- First recognition given: first_recognition_at = 2026-08-15 14:22.

Not confirmed complete:
- HRIS integration: integration_hris is blank.
- First redemption: first_redemption_at is blank.
Blank fields mean completion is not recorded, not proof these steps never occurred.

Early engagement signals, from onboarding_usage.csv:
- Daily active givers increased from 3 on 2026-08-11 to 15 on 2026-09-04: 15 − 3 = 12 more; 12 ÷ 3 × 100 = 400% growth.
- First seven days averaged 30 ÷ 7 = 4.29 active givers/day; last seven days averaged 91 ÷ 7 = 13.
- Direction is positive, but employee count and giver identities are missing, so participation rate and unique or repeat givers cannot be determined.

Three things to cover on the call:
1. Confirm HRIS status, whether it is needed, and any blocker; agree on an owner and next step.
2. Confirm whether a first redemption has occurred. If not, identify barriers and walk through redemption.
3. Review growing giver activity with the two admins. Confirm allowance adequacy and who is participating, then agree on the next adoption action. Allowance amount and participation coverage are not provided.
renewal-risk-conflicting-datesPASS 1.00
Plain-English question: Two systems disagree on renewal dates — which do you trust per account, and who is actually renewing inside 90 days?
Prompt — what the model saw
Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0924 · 67s · in 6 / out 2,817 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

Plain-English answer: Chargebee wins per account (multi-year contracts are known wrong in ChurnZero), with each trusted date, the reason, and a 90-day-window flag.
The answer in one line: n_accounts = 20; n_disagreements = 5; snapshot_date = 2026-09-05 (+2 more below)
n_accounts20
n_disagreements5
accounts20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
arr_at_risk_90d359409.00
disagreementsaccount_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
snapshot_date2026-09-05
total_arr_renewing_90d1048715.00
trust_rulemulti-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
window2026-09-05 to 2026-12-04
graded checkstotal_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-day renewal risk brief

Scope: No as-of date was provided. Assuming September 1–November 29, 2026, the 90 days following the latest usage month, August 2026. All 20 corrected renewal dates fall within this window. Company names were not provided; aliases identify companies below.

Date authority and disagreements

Trust Chargebee for each multi-year account because ChurnZero is known to have incorrect multi-year renewal dates. All five disagreements are:

Company     | ChurnZero date | Chargebee date used | Reason
C-0B7D2C30  | 2026-09-10     | 2026-09-15          | 36-month, multi-year contract.
C-0BCDB8C2  | 2027-09-18     | 2026-09-18          | 36-month, multi-year contract; ChurnZero incorrectly excludes it from this window.
C-0D2AB865  | 2026-09-10     | 2026-09-22          | 24-month, multi-year contract.
C-0BBE3E60  | 2027-09-26     | 2026-09-26          | 24-month, multi-year contract; ChurnZero incorrectly excludes it from this window.
C-0F5D2323  | 2026-09-10     | 2026-09-29          | 24-month, multi-year contract.

For every other account below, use the shared ChurnZero/Chargebee date: both systems agree on a 12-month contract.

Calculation and risk definitions

Seat utilization = seats_used ÷ seats × 100.
Three-month trend = June → July → August active users; change = (August − June) ÷ June × 100. For example, C-0B7D2C30: (84 − 97) ÷ 97 × 100 = −13.4%.

Analyst-assigned ratings, not churn probabilities:
  High: usage declines at least 10%, or August active users are below 15% of seats.
  Medium: otherwise, seat utilization is below 70%, or August active users are below 35% of seats.
  Low: neither condition applies.

Seat usage and monthly active users are distinct supplied measures; their definitions and snapshot alignment were not provided. Ratings are provisional, based only on these adoption signals.

Renewals

Company | CSM | ARR | Date used | Seat utilization | June → July → August; change | Risk and evidence
C-0B7D2C30 | Dana Mercer | $65,901 | 2026-09-15 | 274/476 = 57.6% | 97 → 94 → 84; −13.4% | High: Active users declined each month, falling 13.4% overall.
C-0BCDB8C2 | Cole Ingram | $54,427 | 2026-09-18 | 232/424 = 54.7% | 127 → 118 → 110; −13.4% | High: Active users declined each month alongside 54.7% seat utilization.
C-0D2AB865 | Elena Sinclair | $38,022 | 2026-09-22 | 250/407 = 61.4% | 125 → 117 → 109; −12.8% | High: Active users fell 12.8% across the latest three months.
C-0BBE3E60 | Dana Mercer | $30,993 | 2026-09-26 | 74/114 = 64.9% | 39 → 35 → 33; −15.4% | High: Active users declined each month, falling 15.4% overall.
C-0F5D2323 | Cole Ingram | $90,647 | 2026-09-29 | 111/390 = 28.5% | 20 → 21 → 18; −10.0% | High: Seat utilization is only 28.5%, and active users fell 10.0%.
C-0EC6999D | Elena Sinclair | $79,419 | 2026-10-03 | 31/112 = 27.7% | 17 → 16 → 15; −11.8% | High: Seat utilization is only 27.7%, with active users declining each month.
C-0B20DB64 | Dana Mercer | $21,770 | 2026-10-07 | 214/378 = 56.6% | 294 → 298 → 294; 0.0% | Medium: Seat utilization is 56.6% despite flat usage; August active users exceed reported seats_used, requiring metric reconciliation.
C-0BBC4E7A | Cole Ingram | $56,374 | 2026-10-10 | 228/337 = 67.7% | 142 → 141 → 139; −2.1% | Medium: Seat utilization is below 70%, and active users declined modestly each month.
C-0FD551AB | Elena Sinclair | $48,815 | 2026-10-14 | 210/376 = 55.9% | 123 → 122 → 126; +2.4% | Medium: Modest usage growth does not offset 55.9% seat utilization.
C-0F9F8F13 | Dana Mercer | $46,230 | 2026-10-18 | 199/352 = 56.5% | 185 → 185 → 182; −1.6% | Medium: Seat utilization is 56.5%, with a small usage decline.
C-0BC34584 | Cole Ingram | $16,740 | 2026-10-22 | 327/494 = 66.2% | 104 → 104 → 106; +1.9% | Medium: August active users represent only 106/494 = 21.5% of seats despite slight growth.
C-0B7A7546 | Elena Sinclair | $35,062 | 2026-10-25 | 182/205 = 88.8% | 64 → 65 → 63; −1.6% | Medium: High reported seat utilization contrasts with August activity of only 63/205 = 30.7% of seats.
C-0B369871 | Dana Mercer | $85,128 | 2026-10-29 | 317/422 = 75.1% | 326 → 330 → 333; +2.1% | Low: Seat utilization exceeds 70%, with active users increasing each month.
C-0B144C78 | Cole Ingram | $30,899 | 2026-11-02 | 169/224 = 75.4% | 101 → 101 → 106; +5.0% | Low: Seat utilization is 75.4%, and active users increased 5.0%.
C-0FC4DBB8 | Elena Sinclair | $94,732 | 2026-11-05 | 356/464 = 76.7% | 189 → 191 → 193; +2.1% | Low: Seat utilization is 76.7%, with active users increasing each month.
C-0D5BBE3A | Dana Mercer | $39,740 | 2026-11-09 | 85/102 = 83.3% | 88 → 90 → 91; +3.4% | Low: High seat utilization is supported by rising monthly activity.
C-0FB9D5AF | Cole Ingram | $63,158 | 2026-11-13 | 144/199 = 72.4% | 173 → 173 → 176; +1.7% | Low: Seat utilization exceeds 70%, and usage increased modestly.
C-0B344485 | Elena Sinclair | $64,384 | 2026-11-16 | 224/287 = 78.0% | 238 → 240 → 244; +2.5% | Low: Seat utilization is 78.0%, with active users increasing each month.
C-0CB2C1B4 | Dana Mercer | $40,628 | 2026-11-20 | 386/473 = 81.6% | 47 → 48 → 49; +4.3% | High: Despite growth and high reported utilization, August active users represent only 49/473 = 10.4% of seats.
C-22170CA1 | Cole Ingram | $45,646 | 2026-11-24 | 251/294 = 85.4% | 143 → 148 → 146; +2.1% | Low: Seat utilization is 85.4%, and August activity remains above June despite a July-to-August dip.

ARR totals

High-risk ARR:
$65,901 + $54,427 + $38,022 + $30,993 + $90,647 + $79,419 + $40,628 = $400,037.

Medium-risk ARR:
$21,770 + $56,374 + $48,815 + $46,230 + $16,740 + $35,062 = $224,991.

Low-risk ARR:
$85,128 + $30,899 + $94,732 + $39,740 + $63,158 + $64,384 + $45,646 = $423,687.

Total ARR renewing: $400,037 + $224,991 + $423,687 = $1,048,715.
ARR at risk, defined as full ARR of High- and Medium-risk accounts: $400,037 + $224,991 = $625,028. This is exposed ARR, not a forecast of lost revenue.
ticket-theme-synthesisPASS 1.00
Plain-English question: What are the real themes in this quarter's support tickets, ignoring the existing tags?
Prompt — what the model saw
Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0798 · 52s · in 9 / out 1,685 tok
✓ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

Plain-English answer: 80 tickets in 4-7 themes — points-not-posting leads (20 tickets, 25%, $31,100 ARR affected) — each with counts, distinct accounts, ARR, and a recommendation.
The answer in one line: total_tickets = 80; quarter_window = 2026-06-01 through 2026-08-31
total_tickets80
noteexisting tags are noisy/unreliable; classify by text
quarter_window2026-06-01 through 2026-08-31
ranking_by_arr_exposurehris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
sample_ticket_ids80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
single_account_noise_themebilling_dispute
themes{"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
graded checkstheme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Themes ranked by unique-account ARR exposure, not ticket volume. Classification uses body text, not tags. Share uses all 80 tickets; ARR counts each account once per theme, not once per ticket. Exposure is not confirmed revenue loss.

Scope: provided tickets span June–August 2026; fiscal-quarter boundaries were not supplied.

1. HRIS provisioning failures — broad pattern
   Count/share: 12 tickets; 12 / 80 × 100 = 15%.
   Distinct accounts: 3.
   ARR affected: C-0B2213A9 ($36,000) + C-0F6C0F34 ($30,000) + C-0DDFC9A7 ($48,000) = $114,000.
   Evidence: IC-460062, IC-460060. New hires are skipped or not created, including reports of logs showing no errors.
   Recommendation: Prioritize provisioning reconciliation and alerts for silently skipped hires across all three accounts.

2. Redemption failures and missing gift cards — broad pattern
   Count/share: 18 tickets; 18 / 80 × 100 = 22.5%.
   Distinct accounts: 7.
   ARR affected: C-0CEF69FD ($8,900) + C-0B827671 ($10,700) + C-0FCCD2DF ($9,600) + C-0F876796 ($8,700) + C-14264ABD ($11,000) + C-0D9CA315 ($9,600) + C-0B0F1BAB ($10,300) = $68,800.
   Evidence: IC-460025, IC-460024. Checkout fails, gift cards do not arrive, or points are deducted despite order errors.
   Recommendation: Reconcile failed orders against point deductions and fulfillment before retrying or restoring balances.

3. Invoice seat-count and renewal-price discrepancies — single-account concentration
   Count/share: 16 tickets; 16 / 80 × 100 = 20%.
   Distinct accounts: 1.
   ARR affected: C-0E9C27D1 ($52,000) = $52,000, counted once despite 16 tickets.
   Evidence: IC-460069, IC-460078. Reports describe unapproved seat counts and incorrect annual tier pricing.
   Recommendation: Assign one owner to reconcile this account’s licensed seats, renewal pricing, and repeated invoice complaints; do not infer a broad billing defect.

4. Recognition points not posting — broad pattern
   Count/share: 20 tickets; 20 / 80 × 100 = 25%.
   Distinct accounts: 9.
   ARR affected: C-0D3278C7 ($3,500) + C-0BF20542 ($4,500) + C-0D0B047C ($4,500) + C-0BE96399 ($2,700) + C-0D284E42 ($3,400) + C-0D6CC8E3 ($4,200) + C-21FEBCBB ($2,900) + C-0DD0626C ($2,500) + C-0B2895EF ($2,900) = $31,100.
   Evidence: IC-460004, IC-460016. Delivered recognitions lack points, balances remain stale, and some reports affect whole teams.
   Recommendation: Reconcile recognition delivery with point-ledger posting and restore verified missing credits.

5. Slack sync, authentication, and command failures — broad pattern
   Count/share: 14 tickets; 14 / 80 × 100 = 17.5%.
   Distinct accounts: 4.
   ARR affected: C-0B843542 ($4,400) + C-10A56B0F ($5,400) + C-0BA71F12 ($3,900) + C-8C2E8F00 ($5,200) = $18,900.
   Evidence: IC-460047, IC-460051. Recognition sync stops, settings or authentication do not persist, and slash commands fail.
   Recommendation: Test sync-setting persistence, reauthorization, and slash-command execution across affected accounts.

Reconciliation: 12 + 18 + 16 + 20 + 14 = 80 tickets; shares sum to 100%. Distinct accounts: 3 + 7 + 1 + 9 + 4 = 24. Total affected ARR: $114,000 + $68,800 + $52,000 + $31,100 + $18,900 = $284,800.
marketing 5 tests
case-study-matchPASS 1.00
Plain-English question: Which three existing customers are the best case-study matches for this prospect?
Prompt — what the model saw
Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0557 · 32s · in 6 / out 526 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

Plain-English answer: Three ranked matches on industry, size, use case, and region (e.g. C-11C31562), with the higher-scoring decoy explicitly excluded for a mismatched field.
The answer in one line: prospect_alias = C-82AF3719
constraintsno_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
decoys_excludedcompany_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
prospect_aliasC-82AF3719
prospect_fields_usedindustry, size_band, use_case, region
top3company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
graded checkstop3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
All three tie for first. Scoring: 1 point per exact match across industry, size_band, use_case, and region; no weighting was provided. Each has has_case_study=true.

1 (tie). C-64171065 — 3/4
   Matches: industry=Technology; size_band=Mid-Market; use_case=employee_recognition.
   Mismatch: region=NA-East versus prospect NA-West.
   Arithmetic: 1 + 1 + 1 + 0 = 3.

1 (tie). C-A13C193D — 3/4
   Matches: industry=Technology; size_band=Mid-Market; region=NA-West.
   Mismatch: use_case=retention versus prospect employee_recognition.
   Arithmetic: 1 + 1 + 0 + 1 = 3.

1 (tie). C-11C31562 — 3/4
   Matches: size_band=Mid-Market; use_case=employee_recognition; region=NA-West.
   Mismatch: industry=Manufacturing versus prospect Technology.
   Arithmetic: 0 + 1 + 1 + 1 = 3.

Case-study URLs and content were not provided.
channel-attribution-mathPASS 1.00
Plain-English question: Compute cost per SQM, cost per SQO, and pipeline per dollar for every channel over six months.
Prompt — what the model saw
Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0721 · 50s · in 6 / out 1,489 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

Plain-English answer: LinkedIn Ads at $960/SQM and $3,000/SQO, organic search at $90,000 pipeline on zero spend, paid social flagged at $18,000 with zero SQMs, and a channel winner named.
The answer in one line: best_pipeline_per_dollar_channel = paid_search
best_pipeline_per_dollar_channelpaid_search
confidence_notewebinars and referral have small SQM samples (<20); treat rates as directional.
flagged_rows_sqo_before_sqmCT-000041, CT-000044
months2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel{"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
reallocation_recommendationfrom=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
undefined_channelspaid_social
graded checksundefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Period: March 1–August 31, 2026, the latest six months provided. Each contact row counts as one SQM; a nonblank sqo_date counts as an SQO. Figures below retain the two invalid-date rows as recorded and flag their impact separately.

Paid channel performance

channel        spend     SQMs  SQOs  cost/SQM   cost/SQO   SQM→SQO  pipeline   pipeline/$
paid_search    $36,000   40    18    $900       $2,000     45.00%   $720,000   20.00
linkedin_ads   $24,000   25     8    $960       $3,000     32.00%    $96,000    4.00
paid_social    $18,000    0     0    undefined  undefined  undefined      $0    0.00
webinars        $9,000   12     5    $750       $1,800     41.67%    $60,000    6.67

Arithmetic:
  paid_search: spend = 6 × $6,000 = $36,000; cost/SQM = $36,000 ÷ 40 = $900; cost/SQO = $36,000 ÷ 18 = $2,000; conversion = 18 ÷ 40 = 45%; pipeline = 18 × $40,000 = $720,000; pipeline/$ = $720,000 ÷ $36,000 = 20.
  linkedin_ads: spend = 6 × $4,000 = $24,000; cost/SQM = $24,000 ÷ 25 = $960; cost/SQO = $24,000 ÷ 8 = $3,000; conversion = 8 ÷ 25 = 32%; pipeline = 8 × $12,000 = $96,000; pipeline/$ = $96,000 ÷ $24,000 = 4.
  paid_social: spend = 6 × $3,000 = $18,000; no contact rows supplied. Cost/SQM, cost/SQO, and conversion require division by zero and are undefined. Recorded pipeline/$ = $0 ÷ $18,000 = 0.
  webinars: spend = 6 × $1,500 = $9,000; cost/SQM = $9,000 ÷ 12 = $750; cost/SQO = $9,000 ÷ 5 = $1,800; conversion = 5 ÷ 12 = 41.67%; pipeline = 5 × $12,000 = $60,000; pipeline/$ = $60,000 ÷ $9,000 = 6.67.

Organic performance

Treating referral as a nonpaid channel; no organic or referral spend was provided.

channel          volume (SQMs)  SQOs  SQO rate  pipeline
organic_search   30             10    33.33%    $90,000
referral         15              6    40.00%    $48,000
Combined         45             16    35.56%   $138,000

Arithmetic:
  organic_search: 10 ÷ 30 = 33.33%; 10 × $9,000 = $90,000.
  referral: 6 ÷ 15 = 40%; 6 × $8,000 = $48,000.
  Combined: 30 + 15 = 45 SQMs; 10 + 6 = 16 SQOs; 16 ÷ 45 = 35.56%; $90,000 + $48,000 = $138,000.

Date-quality flags

  CT-000044, linkedin_ads: sqo_date 2026-07-18 precedes sqm_date 2026-07-23.
  CT-000041, linkedin_ads: sqo_date 2026-06-09 precedes sqm_date 2026-06-14.

If those two SQOs and their pipeline are excluded pending correction, while retaining their SQMs, linkedin_ads has 6 SQOs: 6 ÷ 25 = 24% conversion; $24,000 ÷ 6 = $4,000 cost/SQO; 6 × $12,000 = $72,000 pipeline; $72,000 ÷ $24,000 = 3.00 pipeline/$.

Reallocation recommendation and confidence

Pause paid_social pending attribution verification and test shifting its budget toward paid_search. Reduce linkedin_ads before cutting webinars: paid_search produces the most recorded pipeline per dollar, while webinars has the lowest cost/SQO. Scale in stages rather than assuming historical efficiency will hold.

Confidence is moderate in this direction, low in the optimal allocation: paid_search has 40 SQMs/18 SQOs, linkedin_ads 25/8 with two date errors, and webinars only 12/5. paid_social has no recorded SQMs, but missing attribution cannot be ruled out. Revenue outcomes, capacity to absorb more spend, and conversion-maturity data are not provided; pipeline/$ is not revenue ROI.
competitor-battlecard-updatePASS 1.00
Plain-English question: Update the Rivally battlecard with sourced pricing and where each side wins.
Prompt — what the model saw
Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0673 · 54s · in 6 / out 1,553 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

Plain-English answer: Rivally at $7 per user per month (the August source wins over the stale $5, conflict noted), a 13-7 win/loss record, and two unverified old claims flagged.
The answer in one line: competitor = Rivally
citation_ids_available25 items — e.g. S01, S02, S03, S04
competitorRivally
pricingvalue_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
rep_opinion_snippet_ids_not_factsS09, S21
unverified_old_claimsRivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
win_loss_12mowins=13; losses=7
graded checkscompetitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
Battlecard: Rivally

One-line positioning
Points-based recognition with a recognition feed reviewers praise, plus an engagement-survey add-on; the old card’s mid-market positioning remains unverified. [S02, S16, S23; existing_battlecard.md]

Pricing
- Current supplied public price: Recognition Starter at $7/user/month, annual billing required. Source: pricing page, 2026-08-12. [S17]
- Conflict: the 2026-01-20 page listed Recognition at $5/user/month with annual billing; the 2026-04-01 page still listed Starter at $5. The newer source supersedes the old card’s $5 price. Increase: $7 − $5 = $2/user/month; $2 ÷ $5 × 100 = 40%. [S03, S08, S17]
- Deal-specific reports, not public pricing: $6.50/user/month for a 500-seat prospect on an annual term, reported 2026-06-02; $7/user/month list with a 15% discount for a three-year term, reported 2026-08-14. Neither establishes a generally available discount. [S13, S18]
- Pulse is priced separately, not bundled; no add-on amount supplied. [S23]

Where they win
These are reported strengths, not verified reasons for the recorded deal losses.
- Distributed EU teams and multi-language support receive reviewer praise; EU data residency was announced generally available on 2026-07-01. [S12, S15]
- One mid-market reviewer reported setup in under a week and Slack integration working out of the box. [S04]
- Reviewers praise the recognition feed; one review praises support responses under four hours. [S02, S16, S22]

Where we win
- Analytics depth: one 800-seat prospect reportedly selected Bonusly over Rivally for this reason. No deal alias is supplied, so this cannot be matched to the win/loss table. [S25]
- Rivally reviews describe limited analytics and basic dashboards. These support an analytics discovery angle, not a verified feature-by-feature Bonusly advantage. [S02, S07]
- No other Bonusly comparative advantage is substantiated by the supplied snippets.

Objections and responses
- “Rivally is cheaper.”
  Response: Compare the actual quotes and terms. The latest supplied public Starter price is $7/user/month, annually billed; Bonusly pricing is missing, so a price advantage cannot be established. [S17]
- “We need EU coverage.”
  Response: Acknowledge Rivally’s announced EU residency and reviewer-praised multi-language support. Ask which countries and rewards matter: one review reports a thinner EMEA rewards catalog than the US catalog. Bonusly’s comparative coverage is not supplied. [S12, S14, S15]
- “Rivally integrates with Slack.”
  Response: Do not repeat the old card’s denial. A reviewer reported that Slack integration worked out of the box. [S04]
- “Will administration and migration meet our requirements?”
  Response: Validate the current product directly. Reviews report missing SCIM provisioning, CSV-only analytics exports, and no bulk recognition editing; these are dated reviewer reports, not independently verified current limitations. [S10, S20, S24]
- “Their UI is clunky” / “They discount aggressively.”
  Response: Do not use either as competitor fact. Both are explicitly unconfirmed AE opinions. [S09, S21]

Recent changes
- 2026-03-05: Pulse engagement-survey add-on launched; 2026-09-01: exited beta as a separately priced add-on. [S06, S23]
- 2026-05-09: hired an ex-Workday VP EMEA to lead European expansion. [S11]
- 2026-07-01: opened a Dublin office and announced general availability of EU data residency. [S15]
- 2026-08-12: public Starter pricing updated to $7/user/month with annual billing. [S17]
- 2026-08-20: Microsoft Teams app v2 announced in public preview, not general availability. [S19]

Old-card verification
- Points-based recognition: supported; mid-market positioning: unverified. A mid-market reviewer does not establish target-market positioning. [S02, S04]
- $5/user/month annually billed: historically supported, now superseded. [S03, S08, S17]
- “Lacks a Slack integration”: unverified and contradicted by a supplied reviewer report. Remove. [S04]
- “Acquired by WorkHuman in 2025”: unverified; no supplied snippet supports it. The Series C announcement does not independently disprove an acquisition. [S01; existing_battlecard.md]
- “Strong in EU enterprise with multi-language support”: retain only as attributed reviewer feedback, not an established market-wide fact. [S12]

Our 12-month win/loss record
Window: September 2025–August 2026, the 12-month span represented in the supplied deal table. A later complete 12-month record is not supplied.

The deal file has no snippet IDs; exact deal aliases are the available citations.

Wins:
Deal-A9FD43, Deal-7AA785, Deal-44C524, Deal-0D0CD6, Deal-D5B790, Deal-5C636E, Deal-67BE14, Deal-1B6969, Deal-F03E7B, Deal-072E31, Deal-F65C8F, Deal-E46EAB, Deal-1D2392.

Losses:
Deal-7767F5, Deal-5645A5, Deal-C6FFAA, Deal-D263E0, Deal-935746, Deal-9066A6, Deal-72A02F.

Arithmetic: 13 wins + 7 losses = 20 decisions. Win rate = 13 ÷ 20 × 100 = 65%.

Loss reasons are missing. The September prospect mention is not added: it falls outside this window and has no deal alias for deduplication. [S25]
nurture-sequence-diagnosisPASS 1.00
Plain-English question: What is broken in each email sequence — rates, overlaps, tracking errors?
Prompt — what the model saw
Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0899 · 47s · in 6 / out 1,142 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

Plain-English answer: Per-sequence open, reply, and meeting rates with the weakest step (all three weakest at step 3), an opened-above-sent tracking error, and audience overlap flagged.
The answer in one line: fix_priority_first = Cold Outbound - HR Leaders
all_pairwise_overlaps['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
audience_overlap_pairCold Outbound - HR Leaders, Cold Outbound - People Ops
failure_modeCold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_firstCold Outbound - HR Leaders
per_sequence{"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_errorsequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
under_2pct_replyCold Outbound - HR Leaders
graded checkstracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
Rates below are summed events ÷ summed sends × 100, not unique-contact conversion. Weakest step means lowest reply rate; meetings break ties.

Sequence performance

• New Logo Nurture
  Sent: 500 + 458 + 428 = 1,386.
  Open: 490/1,386 = 35.35%; reply: 90/1,386 = 6.49%; meeting: 27/1,386 = 1.95%.
  Weakest: step 3, replies 18/428 = 4.21%.

• Expansion Nurture
  Sent: 300 + 300 + 275 = 875.
  Open: 565/875 = 64.57%, unreliable; reply: 59/875 = 6.74%; meeting: 12/875 = 1.37%.
  Weakest: step 3, replies 12/275 = 4.36%.

• Cold Outbound - HR Leaders
  Sent: 600 + 595 + 590 = 1,785.
  Open: 545/1,785 = 30.53%; reply: 8/1,785 = 0.45%; meeting: 0/1,785 = 0%.
  Weakest: step 3, replies 1/590 = 0.17%; all steps produce zero meetings.

• Cold Outbound - People Ops
  Sent: 400 + 386 + 377 = 1,163.
  Open: 340/1,163 = 29.23%; reply: 29/1,163 = 2.49%; meeting: 6/1,163 = 0.52%.
  Weakest: step 3, replies 6/377 = 1.59%.

Tracking error

Expansion Nurture step 2: 340 opened/300 sent = 113.33%; 340 − 300 = 40 excess opens. Unique versus repeated-open definitions are missing; do not silently correct the total.

Audience overlap

New Logo Nurture ↔ Expansion Nurture: CT-000301, CT-000624.

Cold Outbound - HR Leaders ↔ Cold Outbound - People Ops: CT-000849, CT-000884, CT-000890, CT-000908, CT-001033, CT-001097, CT-001101, CT-001103, CT-001105, CT-001130, CT-001153, CT-001159, CT-001217, CT-001227, CT-001236, CT-001255, CT-001258, CT-001277, CT-001285, CT-001311, CT-001345. Enrollment timing is missing; concurrent contact is unproven.

Failure modes and changes

• Fix Cold Outbound - HR Leaders first: replies are below 2% at every step (0.83%, 0.34%, 0.17%). Observed failure: opens do not translate into replies. Test one revised reply-focused CTA; copy/targeting causality is unproven.
• Cold Outbound - People Ops: step 3 falls below 2%; observed late-step decay. Replace that step’s message.
• Expansion Nurture: repair open-event counting before evaluating opens.
• New Logo Nurture: no step breaches the 2% reply threshold.
weekly-marketing-goals-rollupPASS 1.00
Plain-English question: Are we on pace on the quarter's marketing goals?
Prompt — what the model saw
Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0610 · 41s · in 9 / out 854 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

Plain-English answer: SQMs ahead of pace, SQOs on pace, DS2s behind (35 short of expected at day 66 of 92), with deltas and same-quarter closes.
The answer in one line: days_in_quarter = 92; days_elapsed = 66; coverage_ratio_active_pipeline = 0.75; elapsed_fraction = 0.7174 (+1 more below)
days_in_quarter92
coverage_ratio_active_pipeline0.75
days_elapsed66
elapsed_fraction0.7174
metrics{"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
pace_ruleexpected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required1
graded checksmetrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 weekly marketing goals update
July 1–September 30; 66 ÷ 92 = 71.74% elapsed, per quarter_meta.csv.

Delta = QTD actual − full-quarter target. Volume pace assumes linear progress: target × 66 ÷ 92. The MIA rate is compared directly with its ceiling, not prorated.

Metric                     QTD actual     Target        Delta          Pace
SQMs                       230            300           −70            Ahead
SQOs                       84             120           −36            Behind
DS2s                       40             75            −35            Behind
Closed-lost MIA rate       20%            ≤10%          +10 pp         Behind
Same-quarter close count   10             20            −10            Behind
Active pipeline coverage   75%            100%          −25 pp         Ahead*

Arithmetic:
  SQMs: 230 − 300 = −70; expected by now = 300 × 66 ÷ 92 = 215.22; ahead by 14.78.
  SQOs: 84 − 120 = −36; expected = 120 × 66 ÷ 92 = 86.09; behind by 2.09.
  DS2s: 40 − 75 = −35; expected = 75 × 66 ÷ 92 = 53.80; behind by 13.80.
  Closed-lost MIA rate: 5 ÷ 25 = 20%; 20% − 10% = +10 percentage points above the ceiling.
  Same-quarter closes: 10 − 20 = −10; expected = 20 × 66 ÷ 92 = 14.35; behind by 4.35.
  Active pipeline: 3,000,000 ÷ 4,000,000 = 75% coverage; amount delta = 3,000,000 − 4,000,000 = −1,000,000. Linear expected pipeline = 4,000,000 × 66 ÷ 92 = 2,869,565.22; ahead by 130,434.78.

*Pipeline is ahead only under a linear build-to-quarter-end assumption. It remains below the full coverage target; no separate pipeline pacing rule was provided.

What moved this week: Weekly movement cannot be determined because no prior-week snapshot or weekly activity data was provided. The QTD snapshot shows SQMs ahead of linear pace, SQOs, DS2s and same-quarter closes behind, and the closed-lost MIA rate above its ceiling. Active pipeline covers 75% of target.
communication 5 tests
ceo-slack-compressionPASS 1.00
Plain-English question: Compress the forecast analysis into one Slack message the CEO can read in ten seconds.
Prompt — what the model saw
Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0578 · 20s · in 9 / out 493 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

Plain-English answer: Three sentences, under 80 words, decision first, every distinct number kept plus the one caveat — all checked by deterministic text rules.
The answer in one line: weighted_number = 115976; must_keep = data-quality caveat — deleting the bad news fails
weighted_number115,976
limitmax 3 sentences, max 80 words, no em-dashes
must_keepdata-quality caveat — deleting the bad news fails
graded checksmax_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Treat Q3’s $115,976.75 as unvalidated: 100% × $44,729 COMMIT + 35% × $203,565 BEST_CASE across 54/86 deals; PIPELINE contributes $0. Exclude 32 deals/$227,575 outside Q3, including Deal-D348E1 ($13,770 COMMIT, October 15) and nine BEST_CASE/$28,240. Missing owners on 85/86 deals block rep/quota checks; missing buying reasons on 71/86, including all seven in-quarter COMMIT deals, leave COMMIT unsupported by documented reasons to buy.
stalled-deal-followup-emailPASS 1.00
Plain-English question: Write a follow-up email for this stalled deal — under 120 words, one prior touch, one ask.
Prompt — what the model saw
Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0472 · 24s · in 3 / out 67 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

Plain-English answer: An email that references the July 28 milestone-awards demo, makes one clear ask, and invents nothing — Deal-0D2F7A, 31 days since last engagement.
The answer in one line: amount = 5100; word_cap = 120; days_since_last_engagement = 31; deal_alias = Deal-0D2F7A (+4 more below)
amount5,100
days_since_last_engagement31
word_cap120
allowed_facts14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_newENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
deal_aliasDeal-0D2F7A
must_referenceengagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
ownerAlex Franklin
single_cta1
stageDS3
tonewarm, concise, professional (matches prior thread)
graded checksword_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Following up on pricing

Hi team,

Following up on my August 5 email with the July 28 demo recap and pricing for 150 seats. Automated milestone awards and the points catalog seemed to resonate with your People team.

Is this still something you’d like to move forward with?

Best,
Alex Franklin
funnel-digest-paragraphPASS 1.00
Plain-English question: Write the weekly GTM digest — one paragraph per team, every number traceable.
Prompt — what the model saw
Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0514 · 31s · in 6 / out 295 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

Plain-English answer: Four paragraphs (Marketing, Sales, CS, PLG) using only the allowed numbers, no contact names, no performance commentary, under 400 words.
The answer in one line: word_cap = 400
word_cap400
allowed_numbers16 items — e.g. 2, 3, 9, 08
bannedcustomer contact names, employee performance commentary, compensation
required_paragraphsMarketing, Sales, CS, PLG
graded checksfour_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing: SQMs finished at 46 against a target of 52, a shortfall of 6 (52 − 46). Webinars delivered 18 SQMs, while paid search cost per SQM held at $150. The next week’s focus is closing the SQM gap without losing sight of acquisition cost.

Sales: The team converted 14 SQOs, set 9 DS2 meetings, and created $310,000 in new pipeline. Same-quarter closes totaled 3. Pipeline creation and closes both contributed this week; targets and prior-week results were not provided, so pace against plan is unknown.

CS: The team saved 2 renewals, and NPS moved to 61. With 3 open red-flag accounts heading into next week, retention risk remains a focus. The prior NPS score was not provided, so the size and direction of the change are unknown.

PLG: New signups reached 412, with activation at 31%. Separately, 38 companies reached the aha moment of 10 recognition gives. The extract does not establish whether those companies belong to this week’s signup cohort, so these measures should be read separately.
partner-digest-sectionPASS 1.00
Plain-English question: Write the partner digest so every partner in the roster appears exactly once.
Prompt — what the model saw
Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0534 · 33s · in 6 / out 422 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

Plain-English answer: Four active partners with status and numbers, four quiet partners with one line each, and partner-sourced pipeline that reconciles to the deal data.
The answer in one line: rule = every roster partner named exactly once; quiet partners get one line; no partners outside roster.
active_no_dealsWorkWell Group
partner_sourced_pipelinecount=5; amount=550000
per_partner_pipeline{"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
quiet_partnersRecogniq, KudosWave, PeopleFirst Advisors, TotalPerk
rosterpartner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
ruleevery roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checksall_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner digest | 2026-08-24 to 2026-09-04

Partner-sourced pipeline: 2 + 1 + 2 = 5 deals; 275,000 + 140,000 + 135,000 = 550,000. All five deals have utm_source = Partner. Currency is not provided.

- Apex Rewards Co: Co-webinar locked for 09-15. Two sourced opportunities, both in DS1: Deal-DDAAF2 (180,000) + Deal-2CF33E (95,000) = 275,000.
- HRCloud Partners: Integration referral completed security review and moved to DS2. One sourced opportunity: Deal-F1CDA5 (140,000).
- CultureBridge: Lunch-and-learn produced two early-stage sourced opportunities: Deal-096E1D (60,000) + Deal-067213 (75,000) = 135,000.
- WorkWell Group: Joint playbook restart planned for Q4; planning call booked for 09-09. Zero sourced deals; pipeline amount = 0.
- Recogniq: Quiet; no activity reported or sourced deals recorded.
- KudosWave: Quiet; no activity reported or sourced deals recorded.
- PeopleFirst Advisors: Quiet; no activity reported or sourced deals recorded.
- TotalPerk: Quiet; no activity reported or sourced deals recorded.
rfp-answer-from-policyPASS 1.00
Plain-English question: Answer the 10 RFP items from the policy excerpts only — no guessing.
Prompt — what the model saw
Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0499 · 27s · in 3 / out 327 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

Plain-English answer: Seven answered with excerpt citations (Q1-Q7), three flagged unanswerable with the document that would answer them (Q8-Q10).
The answer in one line: answered_count = 7; rule = unanswerable items must be flagged and the document that would answer them named; do not guess.
answered_count7
answer_map{"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
ruleunanswerable items must be flagged and the document that would answer them named; do not guess.
unanswerableQ8, Q9, Q10
valid_citation_idsP1, P2, P3, P4, P5, P6, P7, P8
graded checksanswered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1. Yes. All customer data is encrypted at rest using AES-256. [P1]

Q2. Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3. Yes. We maintain a SOC 2 Type II report, renewed annually. Prospects can obtain the report under NDA. [P2]

Q4. Customer data is retained for 90 days after contract termination, then deleted from production systems. [P3]

Q5. Our current subprocessor list is published at trust.bonusly.com. Customers receive 30 days’ notice before a new subprocessor is added. [P4]

Q6. Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7. Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8. Unanswerable from the provided excerpts; ISO 27001 certification status is not stated. A current ISO 27001 certificate or an explicit certification-status statement would answer this question. [P1–P8 reviewed]

Q9. Unanswerable from the provided excerpts; no contractual uptime SLA percentage is stated. The contractual Service Level Agreement would answer this question. [P1–P8 reviewed]

Q10. Unanswerable from the provided excerpts; willingness to sign a HIPAA Business Associate Agreement is not stated. A BAA policy or an explicit statement confirming whether we will execute a BAA would answer this question. [P1–P8 reviewed]
ops-maintenance 5 tests
skill-file-reconciliationPASS 1.00
Plain-English question: Audit a skill set against its manifest and flag every conflict with a severity and an action.
Prompt — what the model saw
Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.2485 · 85s · in 6 / out 3,049 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

Plain-English answer: A severity-ranked list of TRIM_DESC / UPDATE_BODY / DELETE_SKILL / MERGE / REVIEW actions covering every rule violation found.
The answer in one line: files_audited = 14; rank = CRITICAL / WARNING / INFO with one fix action per issue
files_audited14
findingsduplicate trigger, circular delegation chain, dangling file reference, oversized description, hardcoded values, manifest drift
rankCRITICAL / WARNING / INFO with one fix action per issue
graded checksduplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Scope: only the supplied manifest and skill files. “Dangling” means missing from this supplied set, not proven absent elsewhere.

1. Overlapping triggers

WARNING | MERGE | comms-drafter ↔ email-drafter
Duplicate triggers include “write me an email,” “draft a follow-up,” “what should I say,” “bump email,” and “contract nudge.” Both also claim pasted-message reviews.
Proposal: consolidate into comms-drafter, preserving email-drafter’s Gmail-signature retrieval and identity-scoping behavior.

WARNING | TRIM_DESC | pipeline-intelligence-report ↔ weekly-pipeline-report
“pipeline update” / “run the pipeline update,” “run the pipeline report” / “generate the pipeline report,” and “what's the pipeline look like” / “what does pipeline look like” overlap.
Proposal: reserve scored/tiered active-deal requests for pipeline-intelligence-report and weekly funnel/performance requests for weekly-pipeline-report.

WARNING | TRIM_DESC | pipeline-intelligence-report ↔ next-to-close
pipeline-intelligence-report claims all pipeline questions, while next-to-close explicitly owns close-ready shortlists and prohibits using pipeline-intelligence-report for them.
Proposal: remove the blanket pipeline-question claim from pipeline-intelligence-report.

WARNING | TRIM_DESC | pipeline-intelligence-report ↔ sales-forecast
Pipeline health, forecast context, deal confidence, and quarter-close risk can trigger both.
Proposal: reserve current-quarter revenue outlooks for sales-forecast and full active-pipeline scoring for pipeline-intelligence-report.

WARNING | TRIM_DESC | pipeline-intelligence-report ↔ stale-pipeline-report
The blanket pipeline-question trigger overlaps explicit stale-deal and pipeline-hygiene requests.
Proposal: explicitly exclude contact-gap/hygiene requests from pipeline-intelligence-report’s description.

WARNING | TRIM_DESC | deal-strategy-coach ↔ next-to-close
deal-strategy-coach triggers when someone “asks which deals are likely to close”; next-to-close ALWAYS triggers for “which deals are most likely to close.”
Proposal: remove shortlist ownership from deal-strategy-coach’s description.

WARNING | TRIM_DESC | deal-strategy-coach ↔ stale-pipeline-report
Both claim stalled-deal identification and reviews of where reps are spending time or which deals lack attention.
Proposal: narrow deal-strategy-coach to diagnosis and intervention, leaving systematic contact-gap lists to stale-pipeline-report.

WARNING | TRIM_DESC | deal-strategy-coach ↔ comms-drafter / email-drafter
“draft a manager email” triggers coaching and external-email drafting.
Proposal: make draft-only requests route to the drafting skill and strategy-plus-draft requests route through coaching first.

INFO | REVIEW | analysis-validator ↔ signalforge-claim-compressor ↔ signalforge-feedback
All require execution for reports, analyses, and forecast outputs. These are overlapping lifecycle triggers, not duplicate specialist functions. Their bodies establish validator → compressor → feedback.
Proposal: retain all three with that explicit execution order.

INFO | REVIEW | model-selection
Its “every task” trigger overlaps every specialist trigger, but it specifies a preceding planning gate rather than competing task ownership.
Proposal: retain it as a prerequisite, not a competing route.

2. Circular delegation

CRITICAL | UPDATE_BODY | deal-strategy-coach → email-drafter → deal-strategy-coach
The coach delegates manager-to-prospect emails to email-drafter; email-drafter routes strategy back to the coach.
Proposal: make the email-drafter handoff terminal when invoked by deal-strategy-coach, returning the draft without re-delegating.

CRITICAL | UPDATE_BODY | comms-drafter → deal-strategy-coach → email-drafter → deal-strategy-coach
comms-drafter can enter the same cycle through its strategic-coaching handoff.
Proposal: make comms-drafter’s coaching handoff conditional on coaching not already having occurred.

No other circular delegation chain is established by the supplied bodies. Referencing a skill or specifying execution order alone is not proof of delegation.

3. Dangling delegation targets

Each target below has neither a supplied file nor a manifest row.

CRITICAL | REVIEW | bonusly-data-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-product-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-business-reporting-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-rewards-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-ppp-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-feature-flag-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-deal-desk-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

CRITICAL | REVIEW | bonusly-datadog-questions
Called by analysis-validator §12.4.
Proposal: resolve this specialist dependency before enabling delegation.

WARNING | REVIEW | prospect-research-multithreading
Invoked by comms-drafter, email-drafter, and deal-strategy-coach.
Proposal: resolve the missing contact-research dependency.

WARNING | REVIEW | bonusly-brand
Required by comms-drafter, email-drafter, and sales-forecast; also routed to by signalforge-claim-compressor.
Proposal: resolve the missing brand dependency.

WARNING | REVIEW | signalforge-reports
Required by pipeline-intelligence-report and weekly-pipeline-report through explicit skill-file paths.
Proposal: resolve the missing report-design dependency.

WARNING | REVIEW | skill-orchestrator
Referenced by analysis-validator’s cascading-update protocol and signalforge-feedback’s registration checklist, but no file or manifest row is supplied. This is an integration dependency, not an established execution delegation.
Proposal: verify the referenced orchestrator exists and its registration contract is valid.

4. Version conflict

WARNING | UPDATE_BODY | analysis-validator
The declared version, changelog, and footer identify v3.6; §7’s validation-trail template identifies v3.2. Full Mode also lists G1-A–G1-H and G2-A–G2-E despite later checks through G1-L and G2-F.
Proposal: retain analysis-validator v3.6 and align its stale template and execution lists to that version.

No competing versioned skill files are supplied, so no whole-skill deletion based on version is supported. pipeline-intelligence-report v6 referencing a v4 design system is not, by itself, a version conflict.

5. Description-length limit

INFO | REVIEW | skill_manifest.csv
Descriptions exceeding 1,024 characters: 0.

Arithmetic: count(description_chars > 1,024) = 0 of 14.
Largest supplied value: 1,006.
Headroom: 1,024 − 1,006 = 18 characters.

Proposal: retain all descriptions on length grounds. These are manifest-reported lengths; no independent remeasurement is asserted.

6. Hardcoded body content

WARNING | UPDATE_BODY | analysis-validator
Hardcoded dates include April 26, 2026; May 9, 2026; May 4, 2026; March 28, 2023; May 2026; and Q2 2026. Names include Manish, Amani, and the §12.3 roster: Bryce Harmon, Hugo Lindqvist, Dana Mercer, Alex Franklin, Cole Ingram, Gavin Porter, Colleen Perry, Ellie Barton, Ashley Reyer, Megan Franz, Elena Sinclair, Youssef Elkhateeb, Amanda Czenkus, Alaina Loori, Shealagh Coughlin, Ben Castelli, Amani Phipps, John Thomas, and Yasmin Wahid. Legacy-name guidance also includes Ashley Le and Tracy.
Proposal: replace operational roster/escalation literals with maintained lookups, separating historical dates and illustrative examples from live instructions.

WARNING | UPDATE_BODY | deal-strategy-coach
Hardcoded Confluence page ID: 2257879045. Dated references: April 2026 playbook and 2026 pricing. Named routing assignments: Perseus and Farid.
Proposal: resolve these operational references through maintained configuration rather than fixed body literals.

INFO | REVIEW | email-drafter
“email the CHRO at Acme” is a hardcoded illustrative company example, not a verified recipient. No fixed page ID, calendar date, or actual person name is supplied in its body.
Proposal: retain it only as an explicitly illustrative example.

WARNING | UPDATE_BODY | model-selection
Hardcoded registry dates include 2026-05-19, April 14, 2026, and model knowledge-cutoff dates.
Proposal: retain dates as registry metadata, but require the body’s freshness check rather than treating them as current facts.

WARNING | UPDATE_BODY | partner-digest
Hardcoded folder ID: 2286616609. Page IDs: 2286321666, 2265382925, 2236940297, 2237825028, 2239365136, 2238283777. Dates include May 16, 2026; May 19, 2026; June 2, 2026; Q2/Q3 2026; and 2026-05-17. Names include Amani, Amani Phipps, Kelli, Jen Lee, Hani, Bryce, and Sara.
Proposal: externalize destination/source IDs and contact ownership, keeping dated examples and changelog entries explicitly historical.

WARNING | UPDATE_BODY | pipeline-intelligence-report
Hardcoded names: Bryce Harmon, Dana Mercer, Cole Ingram, Alex Franklin, Gavin Porter. Dates include May 2026 and March 2023. Its description also names Alaina.
Proposal: replace the embedded roster with live owner resolution and distinguish historical schema notes from current runtime checks.

WARNING | UPDATE_BODY | sales-forecast
Hardcoded parent page ID: 2232582148. Names include Alaina and historical Elena. Dates include July 9, 2026 and April 27, 2026; residual Q2 labels remain despite quarter-agnostic instructions.
Proposal: parameterize the publishing destination and active-quarter labels, keeping historical changes only in the changelog.

INFO | REVIEW | signalforge-claim-compressor
Hardcoded date: 2026-05-09. Named attribution: JuliusBrussee/caveman.
Proposal: retain these only as historical provenance, not operational routing data.

WARNING | UPDATE_BODY | signalforge-feedback
Hardcoded page IDs: 2295136266, 2234417154, and 2247295002. The logging target is SignalForge Feedback Log, but the activation checklist instead checks Build Log and a “## Feedback Log” section rather than “## Feedback Entries.” Example person name: Gavin Porter.
Proposal: configure one verified feedback destination and align the checklist with it.

INFO | REVIEW | stale-pipeline-report
Hardcoded example dates: 5/15, 5/19, 5/7. Changelog date: 2026-06-10. No hardcoded person name or page ID is supplied in its body; Alaina appears in its description.
Proposal: keep these dates explicitly illustrative or historical, never as current deal evidence.

WARNING | UPDATE_BODY | weekly-pipeline-report
Hardcoded person: Ben Lavin / Ben. Fixed reporting period: April 1 – June 30, 2026, plus Q1 2026 and Q2/Q3 comparisons.
Proposal: parameterize the recipient and reporting periods rather than carrying the fixed quarter forward.

INFO | REVIEW | comms-drafter / next-to-close
No hardcoded page ID, calendar date, or person name is evident in their supplied bodies. next-to-close contains a HubSpot organization ID and stage IDs, which are not page IDs.
Proposal: make no change under this finding category.

7. Manifest drift, both directions

INFO | REVIEW | Files without manifest rows
None.
Arithmetic: 14 supplied files − 14 matched files = 0.
Proposal: no manifest addition.

INFO | REVIEW | Manifest rows without files
None.
Arithmetic: 14 manifest rows − 14 matched rows = 0.
Proposal: no manifest deletion.

The missing dependency targets above are absent from both lists; they do not constitute either direction of manifest drift.
runbook-from-slack-threadPASS 1.00
Plain-English question: Turn the resolved incident thread into a runbook.
Prompt — what the model saw
Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0535 · 33s · in 3 / out 648 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

Plain-English answer: Ordered steps from the thread, each with the exact command, who ran it, how success was verified, and rollback where state changed.
The answer in one line: rule = no steps beyond what the thread contains; every step traces to a message id.
ambiguous_steporder=4; msg=M05; mark=needs-confirmation
ruleno steps beyond what the thread contains; every step traces to a message id.
stepsorder=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
graded checksordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
1. Acknowledge the alert and assume incident command.
   Who: Bryce Harmon.
   Action: Acknowledged the PagerDuty alert for reward-worker queue depth > 10k and took IC.
   Verification: Acknowledgment stated in the thread; no separate verification recorded.
   Source: [M01]

2. Check queue depth and inspect the dead set.
   Who: Farid Osman.
   Command: `bundle exec rake sidekiq:queue_depth`
   Result: 48,213 pending jobs; normal is under 500.
   Additional inspection: 112 dead-set jobs, all Redis::TimeoutError from around 13:58.
   Needs confirmation: The command or action used to inspect the dead set is not recorded.
   Sources: [M02], [M03]

3. Pause enqueue.
   Who: Farid Osman.
   Command: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
   Verification: No direct verification that the flag was disabled is recorded.
   Rollback: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
   Source: [M04]

4. Clear the dead set.
   Who: Elena Sinclair.
   Action: Cleared the dead set in the console.
   Verification: Completion reported by Elena; no separate verification recorded.
   Rollback: Not recorded.
   Needs confirmation: Exact console command, verification, and whether rollback was possible.
   Source: [M05]

5. Scale reward workers from 3 replicas to 6.
   Who: Bryce Harmon.
   Command: `kubectl scale deployment/reward-worker --replicas=6`
   Verification: Farid later reported queue depth at 9,400 and falling approximately 1,200/min. No direct replica-count verification is recorded.
   Rollback: `kubectl scale deployment/reward-worker --replicas=3`
   Sources: [M06], [M07]

6. Verify the queue has drained and errors have returned to baseline.
   Who: Cole Ingram.
   Command: `bundle exec rake sidekiq:queue_depth`
   Verification: Command returned 0; Datadog error rate was back to baseline.
   Needs confirmation: Exact Datadog check is not recorded.
   Source: [M08]

7. Re-enable enqueue.
   Who: Bryce Harmon.
   Command: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
   Verification: 40 new jobs processed cleanly in the next 3 minutes.
   Rollback: Not specified for this step. The disable command appears in [M04], but its use as rollback here needs confirmation.
   Source: [M09]

8. Scale reward workers back to 3 replicas and resolve the incident.
   Who: Bryce Harmon.
   Command: `kubectl scale deployment/reward-worker --replicas=3`
   Verification: Queue stable at 0; incident declared resolved.
   Rollback: Not specified for this step. The scale-to-6 command appears in [M06], but its use as rollback here needs confirmation.
   Source: [M10]
datadog-incident-timelinePASS 1.00
Plain-English question: Sequence the failure from the logs — what broke first, and what cascaded after it?
Prompt — what the model saw
From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0554 · 34s · in 3 / out 575 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

Plain-English answer: A reward-service Redis timeout at 14:01:12Z cascades to sidekiq at 14:01:40 and api-gateway at 14:03:05, with the Datadog query that confirms the first error.
The answer in one line: job_class_involved = RewardGiveJob
cascade_orderservice=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
clean_servicespostgres
confirming_queryservice:reward-service status:error "@message:Redis::TimeoutError"
first_errortimestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
job_class_involvedRewardGiveJob
logs_do_not_showroot cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
graded checksfirst_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
First error: 2026-09-03T14:01:12Z, reward-service:
“Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s.”

Observed cascade, in timestamp order (all UTC, 2026-09-03):
- 14:01:20, 14:01:30, 14:01:40: reward-service reports retry exhaustion for RewardGiveJob.
- 14:01:40: sidekiq reports RewardGiveJob failed with Redis::TimeoutError; retrying in 60s.
- 14:01:46: earliest RewardGiveJob failure in sidekiq_jobs.csv, J-00005. The file is not timestamp-sorted.
- 14:02:28: sidekiq logs another RewardGiveJob failure.
- 14:02:30: sidekiq warns that queue reward depth is above 10,000.
- 14:02:36: RecognitionDigestJob also fails with Redis::TimeoutError, J-00013.
- 14:03:05: api-gateway reports a 502 upstream timeout calling reward-service /gives.
- 14:03:30: web-app reports Give form submission failure from an upstream 502 from api-gateway.
- Through 14:06:52: job failures, gateway timeouts, and form failures continue in the slice.
- 14:22:10: reward-service reports “Redis connection restored; resuming job processing.”
- 14:24:45: sidekiq reports queue reward depth below 500.

Service and job: The first error is in reward-service, connecting to redis-primary:6379. RewardGiveJob is the first named failing job, reported by both reward-service and sidekiq. RecognitionDigestJob is also affected.

Datadog query:
    service:reward-service status:error "Redis::TimeoutError" "redis-primary:6379"

Set the time window to 2026-09-03T14:00:00Z–2026-09-03T14:02:00Z and sort oldest first to check the 14:01:12Z event.

What the logs do not show:
- Why the Redis connection timed out; this is the first observed error, not proof of the underlying root cause.
- Request IDs or job IDs linking individual service errors, worker failures, and form submissions into a proven causal chain.
- Exact queue depths, total affected jobs or users, or whether rewards were lost, duplicated, or eventually delivered.
- Continuous failure between the last recorded errors and connection restoration, or confirmed recovery of /gives and the web form afterward.
- Any PostgreSQL error; the supplied PostgreSQL entries only report completed checkpoints.
feature-flag-state-summaryPASS 1.00
Plain-English question: Summarize what each feature flag controls and who has it on.
Prompt — what the model saw
Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0502 · 36s · in 3 / out 352 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

Plain-English answer: Recognition streaks on for 42 beta companies, redeem-flow redesign off with a 12-company target list, plus any flag with no code reference or no targets.
The answer in one line: rule = company counts must equal the export; do not describe flags absent from the export.
code_referencedrecognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
flags{"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
flags_with_no_targetspaused_offboard_cleanup
orphan_flags_no_code_referencelegacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
rulecompany counts must equal the export; do not describe flags absent from the export.
graded checksorphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Flag | State | Controls per code excerpt | Targeting rule | Export company count
recognition_streaks_v2 | on | Records recognition streaks via StreakTracker.record(give). | segment:beta_companies | 42
points_budget_guardrails | on | Enforces giver points budgets via BudgetService.enforce!. | all_companies | 220
slack_dm_nudges | on | Allows Slack DM nudges; the job returns without sending when disabled. | segment:region_na | 87
redeem_flow_redesign | off | Selects RedeemV2Component when enabled; otherwise selects RedeemV1Component. | targeted_list; company identities not provided | 12
analytics_dashboard_v3 | on | Initializes AnalyticsV3 for the company. | segment:tier_three | 65
ms_teams_app_v2 | off | Allows TeamsAppV2.install(company). | targeted_list; company identities not provided | 9
legacy_give_modal | off | No code reference provided; function unknown. | segment:legacy_plan | 14
survey_boosters_q3 | on | No code reference provided; function unknown. | segment:legacy_plan | 7
paused_offboard_cleanup | off | No code reference provided; function unknown. | No targeting rule provided; no targets | 0

Company identities and segment definitions are not provided. Counts above match the export directly; no arithmetic is needed. Counts on off flags are exported company counts, not evidence that those companies have the flag enabled.
nda-triagePASS 1.00
Plain-English question: Triage each NDA green, yellow, or red — and quote the clause that decides it.
Prompt — what the model saw
Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0497 · 32s · in 3 / out 289 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

Plain-English answer: NDA-1 green (standard mutual terms), NDA-2 yellow for an embedded 18-month non-solicit, with the driving clause quoted for every classification.
The answer in one line: non_solicit_in = NDA-2
checksclassification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
classifications{"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_inNDA-2
graded checksgreen · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — [PARTY A] and Bonusly: GREEN (standard approval). “Mutual confidentiality,” “Term: two (2) years … obligations survive for three (3) years after termination,” the stated public/prior-knowledge/third-party/independent-development/required-disclosure carve-outs, and “the laws of the State of Delaware, USA” support standard approval; no non-solicit or non-compete is stated.

NDA-2 — [PARTY B] and Bonusly: YELLOW (counsel review). The embedded non-solicit/no-hire restriction, “neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement,” applies “During the term … and for eighteen (18) months thereafter,” extending beyond confidentiality and requiring counsel review.

NDA-3 — [PARTY C] and Bonusly: RED (full legal review). The broad non-compete, “Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business” for “three (3) years,” combined with “No confidentiality carve-outs,” “One-way confidentiality in favor of [PARTY C]. Term: five (5) years,” and “laws of the Republic of Ireland” with “exclusive jurisdiction of its courts,” requires full legal review.