Ephemeral Testing

Richard Gabriel wrote something that has really stuck with me:

> Abstractions must be carefully and expertly designed, especially when reuse or compression is intended. However, because abstractions are designed in a particular context and for a particular purpose, it is hard to design them while anticipating all purposes and forgetting all purposes, which is the hallmark of the well-designed abstractions.

This is one of my favourite quotes on abstraction, because “anticipating all purposes and forgetting all purposes” is such a good summary of what goes into abstraction design.

A language model in an agentic harness cannot (yet) do this at the same level as a good software engineer, but the advantage they have is speed of token generation, so they can actually build the things the engineer tries to imagine, and verifying those is easier. Very cool!

The big challenge with designing an API is that the only way to be confident in the design is to build a bunch of different things on top of it. But why invest all that effort in an API that you don't think is ready yet?

With coding agents the cost of building those prototypes drops to almost nothing. I can exercise a proposed API design five different ways before I commit to the shape.

- We haven't really solved software engineering yet, it's hard to tell upfront whether a design adapts well to future needs. The community has a collection of heuristics which are sometimes at odds with each other.

- Therefore: write all designs, each simulated against a random tree of future needs.

- Pick whichever minimises the average future diff.

In other words, monte carlo simulation for software design. This scheme obviously assumes the price of code production goes even closer to zero.

I was designing a DSL to help make a colleague’s work easier, and I tested it by having a coding agent re-implement some of their notebooks using a draft of the DSL. The LLM output was a decent enough approximation of “typical” use, and it uncovered some warts I didn’t catch by testing it myself. As its designer, I simply wouldn’t have thought to try using it in some of the ways the LLM-generated code did.

Gets annoying pretty quickly

that gut check of 'does this feel good to use' is funny to hear as QA because most of the feedback I give that boils down to 'this sucks to use' nearly universally gets met with 'but the AC! SLO! SLA! it's designed this way on purpose! it's a feature not a bug' etc.

I mean, I'm glad SWEs are finally discovering that using the product of your own work is actually a net good but it's a little like seeing a bullet coffee evangelist author a blog post about how he's found the real secret to preventing congestive heart failure and it's to not drink butter regularly

[0] https://en.wikipedia.org/wiki/Usability_testing

Eh. I think you're just going to end up with slop, or sloppy recommendations?

My experience is that you can make different trade-offs for different reasons. I think even asking for the best answer to "improve the code, make better trade-offs".. even if you got a perfect response, there's no reason to think that it's the same set of trade-offs your actual use cases would benefit from.

Sigh.

Write a C or C++ API that works with pointers. Make it handle null pointers and errors elegantly so that the API user can safely chain calls and only check the final result. Claude decides it's better to be safe than sorry and peppers its code with intermediate nullptr and return value checks anyway.