90 points by luu 4 days ago | 17 comments | View on ycombinator
reedf1 3 days ago |
kooi 3 days ago |
Makes me think that LLM assisted formal requirements generation for a roll your own "Aristotle" is possible.
Just thinking about our own projects in the signal processing domain. A nice path could be: human language -> temporal logic specs for black and white box run-time testing.
There are code generators which will turn formal logic into a run time monitor. In the past, I've never used them past toy examples because its so verbose.
> The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.
That's the difference between an engineer and a script kiddie.
Domain expertise will always be valuable.
vessenes 3 days ago |
His general test methods this year have been to provide a specification document of some sort and then prompt a clean room implementation of the spec, which can be tested against holdout tests.
Things move pretty quickly - this summer, his test was Zstd and results were at like 25-30% of tests passing at best. He now considers Zstd saturated, as is IMAP from RFC; pandoc is one that he's used throughout, and is not yet perfect.
With all that in mind, read away.
__mharrison__ 3 days ago |
Otherwise, I love this. I was pretty involved in Python testing circles (there was an old Python in tears mailing lists,) and would love to find similar for agentic testing.
ArtRichards 3 days ago |
jiehong 3 days ago |
Danluu often says in his posts that LLM are not very good at good testing strategies, and also that most people are bad at it too.
But, if true, what are the know better ways?
scotty79 3 days ago |
polskibus 3 days ago |
scotty79 3 days ago |
I'd love to see more of that. I'm currently defaulting to Rust in my software building but F# looks super strong there. On par with Python for correctness with Ultra effort.
I wonder how Rust and other languages are doing in Astra 6/Sol 6.1 and on more projects.
patryk_builds 3 days ago |
haukebri 3 days ago |
makercfd 3 days ago |
lubosPetorvic 3 days ago |