point it at a slow python function and it comes back with a rust version that does the
same thing, faster, with a fuzz test as the receipt. honorable mention at the openai and
modal autoresearch hackathon.
both bars use the same scale. the rust bar is two pixels wide because that is what a
548x speedup looks like when you do not cheat the axis.
the problem
every python codebase has a few functions that dominate its runtime, and rewriting them in
a compiled language is well understood but tedious. the tedious part is not the translation,
it is proving the translation still does the same thing on inputs nobody thought to test.
candidates that fail to compile or disagree with the original are dropped, not repaired.
at 50 per round it is cheaper to discard than to debug.
how it works
fifty langchain agents run per round, each proposing an independent rust rewrite of
the same function. they do not see each other's work, so the round samples the space
instead of converging early on one idea.
each candidate is compiled into a real python extension through pyo3 and imported
back into the original program, so the benchmark measures the thing you would actually ship.
correctness is checked by property fuzzing. hypothesis generates inputs and asserts
the rust version returns what python returns. a candidate that disagrees on any input is
discarded no matter how fast it is.
everything runs on modal serverless containers, so the fifty compile-and-benchmark
jobs happen at once rather than in sequence.
what it means
the speedup is the headline, but the fuzz gate is the part that makes it usable. an llm
rewriting hot code is only interesting if something independent checks the rewrite, and a
property test is cheap enough to run on all fifty candidates every round.