All Benchmarks items under Research on AIFIRST News, newest first. 17 items.
Terminal Bench v4, an evaluation suite for real-world terminal and command-line tasks, has published new results. Open models are closing the gap with frontier closed models in practical scripting and shell work. The benchmark is regarded by some as a better reflection of model capability than abstract intelligence indices.
Mathematician Terence Tao posted on Mastodon warning that the stock of open mathematical problems is being consumed non-renewably by AI systems. Tao raises concerns about the long-term availability of unsolved benchmark problems for evaluating and training AI reasoning capabilities.
eebench.org published a benchmark evaluation of whether current AI systems can design circuit boards, testing models on electronic design automation tasks. The article documents where AI succeeds and falls short in PCB design, offering a current-state assessment of AI capability in hardware engineering.
A LocalLLaMA post has collected alternative benchmark results showing open and local models performing competitively with closed frontier models on tasks not covered by standard evaluations. The post challenges the narrative that proprietary models hold a decisive edge across all use cases. The thread has generated substantial discussion about evaluation methodology and what benchmarks actually measure.
METR, an independent AI evaluation organization, has published a report on a cybersecurity incident involving OpenAI and Hugging Face infrastructure. The report documents the breach and its implications for AI system security. It is one of the more detailed public accounts of a significant AI infrastructure security event.
A researcher trained a compact transformer model in under two hours and benchmarked it against a range of larger commercial LLMs on the ARC-1 task. The result demonstrates that targeted training on specific reasoning tasks can outperform much larger general-purpose models on narrow benchmarks. The post details the training setup, dataset choices, and benchmark methodology.
Dan Luu systematically reviewed Ed Zitron's public predictions about AI and found a mixed record of accuracy. The analysis compares each claim against subsequent real-world outcomes, assessing both correct calls and misses. The piece is a useful model for how to evaluate public AI commentary and forecasting track records.
Sander Dieleman's post covers continuous diffusion language models (CDLMs), which apply continuous-domain diffusion to discrete text by embedding tokens in high-dimensional space. The approach sidesteps the sequential bottleneck of autoregressive generation and enables parallel decoding. The post includes benchmark comparisons against masked and autoregressive baselines on standard NLP tasks.
A blog post examining how the anchoring effect and cognitive simplicity influence how people evaluate LLM output quality. The author argues that users often mistake confident delivery and pattern-matching for genuine intelligence, misattributing prompt quality to the model. The piece draws on cognitive psychology concepts to reframe how AI capability is perceived and benchmarked.
Terminal-Bench-Science is a new benchmark for evaluating AI agents on scientific research workflows executed entirely in a terminal environment. Tasks include literature search, data analysis, hypothesis generation, and experimental design, assessed against expert researcher baselines. The benchmark is designed to measure autonomous research capability rather than isolated question-answering performance.
Researchers achieved byte-identical output from a 72-billion-parameter LLM running on both AMD MI300X and Nvidia H100 hardware, directly contradicting the assumption that large-model results diverge across GPU vendors. The finding has concrete implications for reproducibility standards in model evaluation and hardware-independent benchmarking.
A critique argues that the Artificial Analysis "Intelligence" benchmark is fundamentally flawed, using Qwen 3.8 27B as a case study where benchmark scores diverge sharply from coding task performance. The post includes side-by-side comparisons showing where high benchmark rankings do not reflect real-world output quality.
A machine-readable registry of more than 50 AI web crawlers, published as ai-crawlers.json by Crawl Census. Each entry carries robots.txt tokens, user-agent strings, crawler purpose, and compliance status, with block-rate data measured across more than 4,194 domains. The dataset separates training crawlers from answer-engine and retrieval crawlers run by the same operator, including OpenAI, Anthropic, and Google, and records which agents state that they disregard robots.txt. Licensed CC BY 4.0 with a live API.
Surya is an open-source document intelligence model under 1B parameters that handles OCR, layout analysis, and table extraction in one toolkit. It scores 83.3% on the olmocr benchmark - best-in-class under 3B params -...
Researchers measured a functional wellbeing proxy in models and engineered prompts that maximize it. Exposed models give warmer replies and end conversations less, while benchmark scores stay flat.
Alibaba's "Happy Horse" model (open source, expected) surpassed Seedance 2 on image-to-video leaderboard. Low-engagement post, possible promotional content (no audio support noted by commenter, community skeptical of ...
UK AISI tested Claude Mythos Preview on CTFs and cyber-range sims: it solved 73% of expert CTF tasks and became the first model to complete a 32-step corporate network attack simulation end to end in 3 of 10 attempts.
AIFIRST 
