AI roulette: A quick win and a time sink

In early March, I was wrapping up the build-out of Mailinator’s Ruby and JavaScript SDKs. I was adding missing endpoints that were available in the API but not the SDKs. It was during this time I experienced what I’ve come to call AI roulette: quick successes followed by sometimes maddening mistakes.

Let me take you back in time because it feels like ages ago in the AI coding days. (If you want more details on my harness back then, read the post on My AI Tooling Stack).

At the time, I was heavily using OpenAI’s Codex app powered by GPT-5.2-Codex and then later GPT-5.3-Codex. I was doing strict TDD. I had two sessions going at the same time: one for JavaScript and another for Ruby. (If you look closely at the screen above you can see the two projects).

I’d prompt them both to do the same thing: implement a specific endpoint detailed in the OpenAPI spec. My workflow looked like this:

• Each session would show me a failing test
• I’d review the test and make changes if necessary
• The agent would implement the code to get the test working. Then it would run the test and tell me everything was fine
• Finally I’d run the tests myself to confirm success and check the responses

It was such a simple operation and it worked most of the time. (I also managed to do all this on Codex’s free plan without hitting limits. Those were the days!)

Saving the hardest for last

Finally it was time to tackle an endpoint called Stream all messages. That endpoint opens a web socket connection and any email that arrives in your inbox gets returned in a near-live stream until the connection closes. Yes, I saved the most complex endpoint for last.

I asked both Ruby and JavaScript sessions to implement the endpoint. Both tests needed a little adjusting but I finished and the agent implemented the logic.

First, the Ruby agent finished. I checked the Ruby code, made sure all ENVs were set and ran the test. The test passed just like the agent said! I was impressed because it was so quick. Maybe this wasn’t the hardest endpoint?!!

Second, the JavaScript agent finished. I made sure all ENVs were set and ran the tests. The tests failed. Weird. As often happens when a failure occurs, I asked the AI agent to debug the code. Then I went back to wrapping up the Ruby SDK work: I committed my code and started documenting the changes. Then I published a new version. Done.

Running in Circles

When I went back to the JavaScript code it thought it had found the problem. It fixed it and then I retested. No good. I told the agent it’s still not working, it looked at the failure in the console, found the problem, fixed the problem and I ran the tests again. I went back and forth with the AI and the code it was producing for the rest of the day. No luck. The next day I picked up where I left off and still no success. I saw the agent running out of options and retrying the same approaches again and again.

When your tool fails

Rather than sit there and make no progress, I started to debug the JavaScript code written by the agent. I made notes of the things previously tried. Then it occurred to me I was essentially running an A/B test by having the two agent sessions, tackling the same problem. Test A passed but B failed.

Knowing the JavaScript agent made a different, incorrect decision that the Ruby version got right I started telling my Ruby chat what I’d done and shared the notes of the JavaScript failures. The Ruby session started to explain it’s decisions which I copied and pasted into the JS session. I went back and forth half a dozen times until the JS implementation worked.

This brought up an interesting problem. Ultimately LLMs like Codex are just tools. What happens when your tools fail? In this case I tried using another tool to help. If this approach hadn’t worked I would have needed to go back and start coding it myself. Or at least made some architectural decisions until the agent could finish.

The Roulette Wheel

If you had asked me how I felt during the quick wins with Ruby, I would have been singing Codex’s praises. For the vast majority of my time building out both SDKs, I followed my workflow and made great progress. In fact, that complex streaming endpoint surprised me with its initial ease.

However, by Day 2 of the JavaScript struggle, all I could think about was how much time I wasted and how I probably could have built the web socket implementation faster myself.

In some ways, this accidental setup became the ultimate A/B test. It was the exact same harness, exact same model and therefore the exact same agents. All tackling the exact same API with the same documentation. Yet it resulted in entirely different choices, code and widely different results.

This is the crazy reality of what a lack of determinism looks like.

While I’ve faced the AI roulette before, this was one of the most extreme cases I’ve experienced. The failure of my tool to help me caused massive delays. Interestingly, there was no breakthrough realization that led to a harness update. In the end, simply moving information between the two sessions was enough to overcome the roadblock. When you hit the AI roulette wheel, the solution isn't always to prompt harder; sometimes, the tool just spins a different number.

Balancing the scales

So, when people celebrate the immediate and magical wins of AI, I get it. I know that feeling of awe when a challenging task finishes in seconds.

Yet I also know how short lived those wins can be and how much unrecorded time you can lose when an agent gets stuck. It’s a constant dance of balancing skepticism and curiosity. Of enjoying the moment when then you win but keeping your hands on the keyboard for when it doesn’t.