ME:: It's hilarious that asking LLMs to do TDD resulted in worse tests than just letting it write tests; Dan Luu:: Test and verification technique effectiveness with Astra
Discovered: Nov 12, 2026 15:31 (UTC) ME:: It’s hilarious that asking LLMs to do TDD resulted in worse tests than just letting it write tests; Dan Luu:: Test and verification technique effectiveness with Astra
QUOTE
- Read the whole thing (Dan needs a tl;dr or something :-) ): Dan Luu:: Test and verification technique effectiveness with Astra
When we looked at how well agents used test/verification techniques in August1, we saw that GPT-5.6 Sol generally did what you might call a caricature of various techniques when asked to use a technique or library such as TDD, QuickCheck, Lean 4, etc.; not specifying any particular test method (which we’ll call Default) resulted in many worthless unit tests, which was one of the best performing conditions because asking agents to test in specific ways mostly made them worse. In that eval, we gave agents the Zstd RFC and asked them to implement Zstd (with an addendum for how they should test for all conditions other than Default).