The Shortest Romance

This dispatch covers two engines at once, because their stories mirror each other. One promised more than anyone and lasted the shortest: we passed the verdict on Grok in under two days. The other promised nothing special, posted the best score in the history of our tests — and still didn't become the main engine. Both stories are about what actually decides.

With Grok, everything went fast. We plugged it in and gave it the same scenarios as everyone: recognize the caller, check the balance, transfer to a human. Out of six transfer attempts it managed two. Out of ten scenarios overall — three. Thirty percent. For scale: the previous candidates scored seventy-five and up on the same tasks.

The core trouble was the very one we asked you to remember two dispatches ago: it refused to call its tools. The context is clear, the tool is right there, the instruction says in plain words "in this situation, call this" — it doesn't. It answers on its own, making it up as it goes. This, incidentally, is honestly acknowledged in the developers' own documentation — we merely confirmed in practice that the admission is not exaggerated.

We didn't give up at once — we tried four approaches. Tightened the rules: zero improvement in ten tries. Cut the instruction to a third, so the important parts wouldn't drown in the middle of the text — no help. Worked around a plumbing bug that was delivering the instruction to the engine late in the first place — slightly better, same ceiling. Rewrote everything in the engine's native format — thirty percent. A wall.

The language stubbornness finished it off. To a caller's polite "could we switch to Russian?" the engine would answer in Russian — and slide back into Ukrainian within two exchanges. Five times out of five. For a contact center in a city where half the people speak one language and half the other, that is not a quirk — it is disqualifying.

We never even had to do the economics: its price was fine, the same five cents, but thirty percent quality doesn't pay off at any price. The pilot was closed, the resources reassigned — and here the quiet hero of this series steps onto the stage.

In parallel with all this circus we were running the second version of OpenAI Realtime. And it posted the best score in the history of our tests — ninety-six point six percent on a slimmed-down instruction. With quirks of its own: a hard cap on instruction size, beyond which the engine silently ran without our rules as if they had never existed; the same familiar slide into the wrong language in a share of the failures; and noticeably longer, more drawn-out conversations — where the future winner wrapped up in a minute or a minute and a half, this one talked for three or four, with extra loops and repeated questions.

But it wasn't the quirks that kept it from the top job. Arithmetic again: eighteen to twenty cents per minute at steady state, against five. A fourfold difference for quality that, on live calls, felt no better — and in places worse: longer, noisier, more tiring for the person on the other end. We left it on the bench, where it sits to this day: if anything happens to the main engine, there is someone to step in.

So by mid-May the score stood as follows: four big names tried, none of them fit. The beautiful voice lost to development speed, the flawless one to price, the biggest one to its own caprices, the most hyped one to itself. The main job went to the fifth — an engine you have most likely never heard of, and that is exactly what we liked about it. The finale of the series — in the next dispatch.