What a year of testing open models against real work actually taught me

Open models did this work as well as the expensive ones. That held up across 117 models, 40 real tasks, and 4,582 graded answers. The most useful findings go past the models themselves.

Amani Phipps, Head of Revenue Architecture at Bonusly
RevenueBench census, October 2026. 117 models. 40 tasks. 4,582 graded answers.

1. The problem

My old process was a models page and a benchmarking page. See what is popular and new, then go try it. That is how a Tencent model named HY3 ended up doing real work at our company. It showed up near the top of everything, I used it, and it was genuinely good.

What the leaderboard never gave me was a so what. I kept using it, watched the actual cost, noticed it ran more verbose than what I already had, and started running small side by side comparisons against the four or five models actually in rotation: GLM, Qwen, DeepSeek, and the Anthropic and OpenAI models I was splitting between.

I could see something real in the open field. I could also see several hundred more models I had not touched, and testing them one at a time was not a plan. That gap is RevenueBench.

2. Method: real work, not synthetic prompts

The first design question was how to make the results trustworthy. The answer was to test against our own skills rather than benchmark style tasks. By that point we had built well over a hundred skills for our AI instance, with a deliberate rollout behind them: foundation first, alignment on governance, a real jobs to be done check before anything shipped.

It was not tidy from day one. We had a token heavy month early on where we just explored, because people needed to be using this stuff before anyone could see where value was hiding. Once that phase did its job, the work slowed down and got specific: skills built for one team, solving a problem that team had already said was worth solving.

The bench itself is 40 revenue operations tasks drawn from real GTM extracts: pipeline hygiene, forecasting, renewal risk, attribution, customer health. Five tests in each of eight categories. Every task ships with its source data and a plain English question, plus ground truth computed by code from that same data. Answers are graded by fixed code, never by a model. A name whitelist catches invented accounts, owners, and numbers. That is what makes it possible to ask an honest question: is this model cheaper than what I already pay for the same job?

117 models · 40 tasks · 4,582 graded answers · 103 full coverage · 14 partial

3. Results: price did not buy accuracy

The headline result: open models did this work as well as the expensive ones. The top score belongs to deepseek/deepseek-pro-latest at 0.9846, with 37 of 40 tasks perfect, for $0.99 across the full suite. The priciest suite in the run, openai/gpt-5.4-pro at $131.87, scored 0.9421 with 32 of 40 perfect. That is roughly 133 times the spend for a lower score. Thirty nine full coverage models outscore it.

Six models share the next best mark of 36 perfect tasks. None cleared all 40. The spread opens on tasks that mix arithmetic with judgment: which deals to exclude, what counts as stale, when to say the data cannot answer the question.

#ModelMeanPerfect
1deepseek/deepseek-pro-latest0.984637/40$0.99
2fireworks/ember-10.979636/40$7.31
3z-ai/glm-5.3-flashx0.978136/40$1.34
4meta/muse-spark-1.10.977335/40$2.17
5moonshotai/kimi-k30.976236/40$7.67
6xiaomi/mimo-v2.5-pro0.973836/40$0.69
7qwen/qwen3.8-flash0.969235/40$0.34
8z-ai/glm-5.3-flash0.967935/40$0.28
Table 1: Top 8 full coverage models by mean score. Suite cost is the full 40 task spend. Five of the eight cost under a dollar. The cheapest row, glm-5.3-flash, costs $0.28 at a 0.9679 mean. Fifty four full coverage models score at least 0.93. Twenty one of them cost $2 or less.

Speed turned out to be its own axis, separate from accuracy and price. tencent/hy3-preview averaged 82 seconds a task at 0.9677. tencent/hy3 took about 27 minutes a task at essentially the same score (0.9671). If someone is waiting on an answer in Slack, that difference decides the purchase.

Failure modes were specific. 17 of 117 models invented something: a name, an account, a renewal owner that was not in the data. The whitelist caught each case and the answer scored zero. That is the failure to guardrail first. Thirteen models ran short of the full 40, mostly slow responders timing out. Those tasks sit ungraded and marked partial. We never score work a model did not do.

4. The model depends on the job

The top model on my list could not read a screenshot. DeepSeek is a text only model. It leads the board on text work and it cannot process files, images, or video. I learned that one live: I sent it a screenshot of a bug the way I would send one to any coworker, and it just could not take it. Best model on the list, wrong model for that ten seconds of work.

This matters for anything that lives in Slack. People talk to an agent there the same way they talk to the person next to them, and the natural response to "here is what I am seeing" is a screenshot. A model that cannot see it is out of the running for that job, no matter where it ranks.

So the model you use depends on your use case. And the line moves. Harnesses and IDEs can encode formats the model itself cannot take. A screenshot becomes a text extract, a PDF becomes pasted text, and a text only model is back in the game. Hermes does this kind of preprocessing before the model ever sees the input, which softens the boundary the bench scores draw.

Keep one thing in mind. This bench is a benchmark on my work. My tasks, my data, my jobs to be done. The scores tell you where I would start looking. For your own work, you have to try and see for yourself. No benchmark, mine included, picks the model for you.

5. The data in real life meets Vera

We knew we needed the benchmark. We also knew a score alone would never tell us if the models were good in real work. So we built Vera, a RevOps employee who lives in Slack. We gave Vera a personality and the full setup: core connectors, skills, memory, doc backed responses.

Vera is where the open models met the exact same work I had been doing in Claude. Real requests, real data, inside our own infrastructure. We ran the work through Vera on open models and compared it against what Claude gave us for the same job. Vera worked great. In many cases Vera worked better, which was the fun surprise.

Vera also reads the benchmark. Every week Vera reads the RevenueBench report, looks back at the work from the past week, and flags what looks interesting: hey, this model might be worth a try. That loop is how dozens of models nobody would have picked on their own made it into rotation. This run's top scorer came out of that loop, not from a leaderboard page.

Vera in Slack stating it runs GLM 5.3 FlashX via OpenRouter with a Muse Spark 1.3 fallback on the Railway vera gateway
Exhibit 1: Vera lives in Slack on open models. Asked what it runs, Vera answers: GLM 5.3 FlashX via OpenRouter, fallback Muse Spark 1.3, on the Railway vera gateway. That pick came straight from RevenueBench results on our real work. We started on a few core models and have since tested dozens more.

Three models at the top tell three different stories, so here is the full breakdown. The scores look close on paper. How each one responds and interacts in real work is where they separate. Muse, Kimi, and Ember all earn their place for different reasons, and picking between them comes down to the job in front of you.

ModelFull bench20 test subsetAvg latency
fireworks/ember-10.9796, 36/400.946$7.31124.8s
meta/muse-spark-1.10.9773, 35/400.949$2.1740.2s
moonshotai/kimi-k30.9762, 36/400.980$7.67102.1s
Table 2: The three way comparison. Full bench mean and perfect count, an isolated 20 test comparison, suite cost, and average latency. All three went 14 for 14 on a separate analytics mockup with zero fabrication.

Kimi is my daily driver. It costs more than Muse ($7.67 against $2.17 for the full suite) and I pay it gladly. It topped the 20 test subset at 0.980, went clean on the mockup, and holds 36 perfect tasks on the full bench. Against the frontier models it replaces, the gap is not close. GPT-5.5-pro costs $123.62 for a lower score, more than sixteen times what Kimi costs. That is the whole trade in one line: slightly more than the cheapest good answer, a fraction of the famous ones.

Muse Spark 1.1 is the speed pick. Forty seconds a task, fastest in the top tier, at $2.17. When someone is waiting in a channel, that is the call.

Ember 1 is the accuracy runner up at 0.9796 with 36 perfect. It trailed the subset at 0.946, which is exactly why the subset exists. One number never tells you enough.

Neither number alone picks the model. The bench is the filter, the product is the verdict, and you need both or you will promote a model that is right on paper and wrong in the room.

Vera in Slack handling six tasks at once across spreadsheets, HTML summaries, bookings report, deals doc, page mockup, and weekly brief
Vera dispatching all six jobs in three parallel subagent batches, then delivering batch 2 of 6 with a verified next to close table
Exhibit 2a: The bulk ask, dispatched. Six jobs go in. Vera dispatches three subagent batches in parallel, verifies every file, and posts each deliverable back in the thread. Batch 2 of 6 lands first: next to close top 5, validator verified against 259 open deals.
Exhibit 2b: The stress test. Six jobs at once across data pulls, HTML docs, and a UI mockup. Vera responded by deploying batches of agents to answer, then left the responses in the thread. Fourteen replies, one thread. This is the half of testing a score cannot do.

Exhibits 1, 2, and 3 are all real work. We gave Vera real data and let Vera answer across a bunch of different jobs to show what these models actually produce. A lot of it is really cool. Some of it is imperfect. Vera misses things sometimes, and building on open source means you own those misses. That is the honest tradeoff in this section.

Vera Slack report showing adoption curves with a chart of chat turns per month against seat penetration
Exhibit 3: Vera live fire, adoption curves. Vera in Slack delivering a research report as an HTML file: volume against penetration, era by era. The pro is the work itself, genuinely incredible output from open models. The con is the tradeoff: bugs ship sometimes, a bad placeholder broke attachments in this very thread, and someone has to go fix them.

6. Run it yourself

The short version: if you want to run this kind of test on your own work, the repo is ready. Plug in your own tasks, structure a couple of tests, and try it out. That is what revenuebench-starter is for.

People send me ideas for what to add. I take some of them. The bench grows when a suggestion earns its place, and stays small otherwise. Take it, run it, and see what your own data says.

Vera itself runs Railway only: the teammate, the weekly bench read loop, and the model routing all live there. Local runs are cron only. That split is deliberate. The product and the measurement stay on the same rails.

7. What it actually saved, and the compounding

Vera is effectively an open source build on the same ideas as Claude Tags: skills, memory, and doc backed responses, running on open models. Over 350 real uses, all on open models, cost less than $250 total. The stack runs primarily on Muse and GLM. It is fast, it holds up in threads, and it showed us this path can work in production.

I am a heavier AI spender than almost anyone else at the company, which is not an accident: I build the tools most people here use. Building on open models has brought that cost down by a lot, not a little.

Vera is the clearest proof. More than 350 real uses, all on open models, cost less than $250 in total. The stack runs primarily on Muse and GLM. It answers fast, it reads well in Slack, and it keeps up with daily work from me and from other people using it every day.

Put that next to frontier pricing and the gap is stark. Running Vera on open models has run about 20 to 25 percent of what the same work costs on frontier models. I ran Opus 5 for 2 days and those 2 days cost $88. That is roughly what all the open model use cost over several weeks of daily use, by me and by others. Same jobs. A fraction of the bill.

Diagram showing every turn going to Muse before, and triage routing to Muse, GLM, or HY3 by confidence with fail open fallback after
Exhibit 4: The routing. Every turn used to go to Muse. Now triage routes each turn by confidence and fails open to Muse on doubt. Measured on the same 50 asks, routed spend came down about 30 percent and 42 percent of turns moved off Muse. The lesson goes past the router: several models can do each job well, so you can get specific about which model serves which turn.

I started with Jev, a simple triage pass. Later I rebuilt it as a more complex router on the Decisions API. Either way the idea is the same: tune each step for a different lever. Fast and cheap where speed matters, stronger models where quality matters, cheap models where volume piles up. Model selection stops being one big choice and becomes a set of small ones.

SignalForge was the second thing we built end to end. It is a collection of skills, processes, business intelligence, new queries, infrastructure, the works. It does a tremendous amount of work, and we built the whole thing on frontier models. Vera was similarly complex work. The difference is Vera was built on open source models, which cut the cost in half. Vera took about two weeks against SignalForge's month and a half, mostly because we already had so much infrastructure built. Even so, plenty needed redesigning: skills and logic tuned for Claude did not drop straight into a Slack app, and we reworked them until they fit how Vera actually gets used.

That is the compounding benefit of doing things well at each stage. Good skills transfer. Good governance transfers. Good documentation transfers. Each build pays for part of the next one, and the team gets faster without cutting corners to get there.

8. There is no right way, only the contextually correct way

It struck home for me when we released this report at Pavilion GTM 2026. The room was full of operators with top down mandates, bottom up demand, 5 people or 5,000. Everyone was asking some version of the same question: how do we use AI to change how the company operates in a way that lasts?

My take after dozens of conversations this year: there is no universally right way to do AI today. There are contextually correct ways. Open source will make a lot of sense in some setups and very little in others. It comes down to your constraints and what you are optimizing for.

Are you in a stage where working the margins and being efficient is the right call, or are you in growth mode where speed matters more than unit cost? What is your appetite for risk across governance, controls, security, and data access? When do you build and when do you buy? What outcome are you actually trying to move, stated as a metric and not a slogan? How do you make people better while giving them more AI, so judgment and skill keep growing? How do you measure whether any of this works, beyond adoption and usage? Open or closed models, one provider or several, and where do you even start? My suggestion is data. And who gets let into the kitchen: who builds, who learns these skills, who shapes the infra you may one day depend on?

A year ago most teams had not built enough to answer any of that. Now many have scars. They know where AI works well and where it does not. That means we can finally learn from each other.

We are under 100 people. We are not public. We do not carry some categories of sensitive data. We can move in ways a 500 person company cannot. Our risk profile, resources, speed, and constraints point to open models for a large share of the work. Your answer will look different, and it should.

9. Bad data in means bad data out, no matter what model you use

Chasing the perfect open model and building agents around it before you understand your use case is how teams burn months and end up worse off than if they had stayed on Claude. We got conviction on this early: good data, good systems, documentation that means something, consistency across everything we built. That took months, and it is not something you build once and walk away from. Workflows get re checked on a schedule, because the foundation slips if nobody keeps an eye on it.

If this research proved one thing more than any other, it is that the model cannot save bad data. Garbage in, garbage out, at any price.

10. Scale changes the answer

For us, scale meant thinking through two futures: what it looks like to scale open source against what it looks like to scale frontier models. We are a growth stage company. That means we optimize for revenue and growth first, not for the most efficient bill. Even loving open models as much as we do, the answer keeps going back and forth. Plenty of weeks the open models carry the load. Plenty of times a frontier model does a good enough job that reaching for anything else would be precious.

So this is less a story about Vera as the future and more a running question: what makes sense for now? Every day something new shows up on the horizon. The honest version is that we re-answer it all the time.

One caveat on cost structure. Open model spend does not run flat month to month. Usage shifts as you move between models, and the bill moves with it. But do not pretend a Claude or OpenAI bill runs flat either. Ours never did. Usage based pricing varies everywhere. The variance is similar. The difference is who you call when it spikes.

The deeper cost is operational. On frontier you buy predictability and governance: one vendor, one bill, controls someone else maintains. On open source you own everything. SOC reviews, compliance, the code itself, baking models into your infrastructure, deploying where you already deploy, domains and gateways and all the plumbing. That is real work. Over time it can absolutely be worth it. Go in knowing the price is paid in engineering, not tokens.

My one strong conviction: the company agent is worth owning. Its context, its knowledge, how it answers for you. If there is one thing in the stack I want to hold, it is that. Everything else can stay in motion.

11. Who should do what

One note first: if you can build open source into your existing monolith, awesome. It takes more work and more effort than anyone quotes you, so treat these recommendations as directional and size the team before you start.

PathWhenPayoff
Run open models direct
you work the margins
Your infra is easy to change, your data is in good shape, and someone owns the stack. You can swap models, watch the numbers, and re check on a schedule.This is where the full win lives. Best scores at the lowest cost, and honestly it is fun. Picking the model per job never gets old.
Ride a managed platform
you want the savings, not the job
You want open model pricing without running the bench yourself. A platform like Hoonify gives you a small, steady set of great models. They are not doing the routing for you. You pick a model and you get the benefit there. Someone else handles the hosting and the updates.Most of the benefit, none of the tinkering. Fewer models to choose from, but that is the point.
Stay on a single vendor
predictability beats price
Your org needs a flat bill and a single vendor to hold accountable. Several layers of approvals, hundreds or thousands of employees, no appetite for costs that move month to month. Stay on an enterprise plan where you keep artifacts and the controls you need, and let frontier infra carry the jobs where that matters.Claude is a perfectly fine answer. Open model costs shift as you move between models in ways a single vendor bill does not. If that variance worries you more than the price tag, stay put.
Table 3: Which path fits. Directional, not prescriptive. Plenty of things can move you between rows: risk appetite, data shape, team size, compliance load, who owns the stack.

Whichever path you take, three rules travel with you. Match the modality to the job before you compare scores. Keep a live fire loop, because the bench narrows the field and the product makes the call. And guardrail first: 17 of 117 models invented something, and the whitelist is the only reason those answers scored zero instead of shipping.

12. My favorite models

Scores pick the shortlist. Living with the models picks the winners. These are mine, and I want to be clear up front: the frontier models are good. The GPT models are good. Opus models are good. Sonnet 5.5 is pretty good. I am not knocking any of them. The open models below earned their place next to those, not instead of them.

Muse Spark. Awesome model, full stop. I did not expect it to score near the top, and then it kept doing incredible work at a great price. $2.17 against $7.67 for Kimi across the full suite, fastest in the top tier at 40 seconds a task. When someone is waiting in a channel, this is the call.

GLM. Awesome suite of models. GLM 5.3 FlashX is in Vera every day at 0.9781 with 36 perfect tasks for $1.34, and the smaller GLM Flash is the cheapest strong answer on the board at $0.28. Steady, fast, easy to route around.

Kimi K3. Awesome suite, and my daily driver. 0.9762 on the full bench with 36 perfect tasks, top of the 20 test subset at 0.980, clean on the mockup. It costs more than Muse and I pay it gladly. Against the frontier models it replaces at $123.62 for a lower score, the trade speaks for itself.

DeepSeek. Awesome suite. Top of the board at 0.9846 for $0.99. Yes, I was bummed when I learned it could not see the files I shared in real work (section 4). Even so: fast, affordable, high quality. It earns the top spot.

HY3 and HY4. Fantastic models. They hold a special place because they are night and day cheaper and still do really good work. HY3 is the one I keep coming back to: 36 perfect tasks at $0.33 for the suite, which outscores Haiku 4.5 (29 perfect at $2.34). Where shavings make a pile across CRM and system jobs, HY3 gets the call.

Haiku. Bubbling up and good. It keeps showing up in rotation conversations for the same reason: reliable output at a price that lets you run it everywhere without thinking.

Pareto and Ember. Both on my list. Ember sits second overall at 0.9796, Pareto matches Kimi's 36 perfect at 0.9733. I picture ending up with a suite of five or so core open models per department, with the same three or four underneath everything I do.

Hoonify hosted GLM and Qwen. We ran their hosted GLM and Qwen models against the bench and they were solid. This is the managed path from section 11 working in practice: a few good models, none of the tinkering.

13. The part that is not about cost

The savings matter. The pulse check matters more. Without this I would be paying whatever Claude charges because Claude is what we have always used, and never asking the question. Getting equal or better results out of open models keeps reminding me how much is actually moving right now, and that is worth watching whether or not it saves a dollar this month.

I used this report to find models that work well, and they performed without a shadow of a doubt. I am glad we did it. Open models keep getting better, fast. As Anthropic and OpenAI move toward IPO scale, that progress matters: it gives operators room to work the margins with open models and spend on frontier infra only where it earns its keep.

We will keep testing as long as there is something worth testing. This was the first full pass.


Where this breaks