Ofir Press

@OfirPress

I push the AI frontier by building realistic yet challenging benchmarks. SWE-bench, ProgramBench. Postdoc @Princeton. PhD @nlpnoah @UW.

NYC

ofir.io/about

Replies Reposts Media

Pinned
Ofir Press
@OfirPress
May 5
1) Our team at Meta has a tough new coding benchmark challenging models to code entire programs including ffmpeg and the PHP compiler from scratch. 2) Top accuracy is 0% 3) We will be making the benchmark harder.
John Yang
@jyangballin
May 5
How much of SQLite, FFmpeg, PHP compiler can LMs code from scratch? Given just an executable and no starter code or internet access. Introducing ProgramBench: 200 rigorous, whole-repo generation tasks where models design, build, and ship a working program end to end. 🧵
38
Ofir Press
@OfirPress
Sep 29
so cool to see all this new research on multi-agent systems, there's so much to explore here- should the agents be organized in a hierarchy or not, should we tell them how to organize or should we let them figure it out, should we assign them roles from the get go or no? cool!
Zhihao Zhan
@zhan8855
Sep 23
Replying to @zhan8855
6/9 We evaluate Agensh on the 5 hardest ProgramBench tasks, using GPT-5.6-sol model with high reasoning effort, and Copilot as the underlying single-agent harness. Scaling 1 → 128 agents raises mean 6h final pass rate from 19.31% → 28.78%, about a 49% relative improvement.
2
Ofir Press
@OfirPress
Sep 28
The scammers are at it again with the fake calendly links, this time pretending to be Reuters journalist Tifafny Wu / Keyaki Yuu. Watch out.
David Galbraith
@daveg
Sep 27
Sophisticated AI phishing scam I nearly fell for. A journalist supposedly from Reuters reaches out with a request that sounds plausible because it is an AI that has read what I've been posting and I quite often get DMs like this. They want to schedule a call chat a bit and then
2
Ofir Press
@OfirPress
Sep 28
claude code doesn't use RAG either. i wouldn't have expected grep to be this good, but seems like simplicity won, again
Tommy Zinnatullin
@its_tommy_zinn
Sep 27
Wild, even Cursor moved away from server-side RAG index and into Grep-like agent tool calls for code search "we’re no longer computing embeddings of your code or storing them on our servers for search." — Jul 20 huh
Quote
Jo Kristian Bergum
@jobergum
Sep 26
it looks like cursor has stopped using vector-based code retrieval and moved away from turbopuffer
8
Ofir Press
@OfirPress
Sep 27
cute mini-bench from rpn on Instagram:
00:00
2