Kelly Buchanan

@ekellbuch

@Stanford with @HazyResearch and @Scott_linderman. Working on data and evals for 🤖🧠 PhD @cu_neurotheory @GoogleDeepmind

San Francisco, CA

ekbuchanan.com

Joined July 2011

2,569 Following

1,859 Followers

Replies Reposts Media

Pinned
Kelly Buchanan
@ekellbuch
May 6
Very excited to release Terminal-Bench 2.1! Coding agents are among the most economically consequential deployments of LLMs to date. As agents improve, benchmark reliability matters more. We audited TB2.0 and found and corrected issues in 28/89 tasks. 30% of the benchmark!
29
Kelly Buchanan
@ekellbuch
Sep 25
Super excited to see this come out. Over the next few months we are going to see a lot more on virtual companies and AI-native enterprises. Congratulations, and please keep sending work in this space my way!
Tim Dettmers
@Tim_Dettmers
Sep 25
We built an all-synthetic simulation of a hospital that can be shared publicly without any issues. Our data quality is so high, physicians cannot tell the difference between real/synthetic patients. The data is fully verified -- perfect for RL.
Kelly Buchanan
@ekellbuch
Sep 22
Super excited and unsurprised! @SnorkelAI has done a fantastic job bringing together the community building better data and benchmarks, and supporting that work through grants and open collaborations. All this while being incredibly candid about what works, what doesn’t, and what
Alex Ratner
@ajratner
Sep 22
I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC. We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week. As AI advances to superhuman capabilities, AI data &
2
Kelly Buchanan
@ekellbuch
Sep 18
Finding evaluation issues is non-trivial. Fixing them is even harder. As models improve, they will continue to discover valid solutions that existing verifiers were never designed to recognize, creating new false negatives. At the same time, the evaluation stack itself will
Yafah Edelman
@YafahEdelman
Sep 17
We audited 15 benchmarks and labeled 9 flawed: - In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues. - In HLE, 46% of the 48 questions we randomly sampled were broken. - In DeepSWE 1.1 we found a bug that can break grading for every task.
Quote
Epoch AI
@EpochAIResearch
Sep 17
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
2
Kelly Buchanan
@ekellbuch
Aug 27
Super excited for this release. Check out these scientific agentic tasks!
Steven Dillmann
@StevenDillmann
Aug 27
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions