Automated AI Research Shows Early Gains in Models, Training, and Kernels

Richard Socher proposes a “Eureka machine”: a system designed to automate scientific discovery and eventually generate a large share of humanity’s future inventions. Its operating premise is that scientific work can be expressed as a repeatable loop: generate hypotheses, test them, identify shortcomings, preserve useful results, and choose the next experiment.

The immediate constraint, in Socher’s account, is specialization. Drawing on Stanislaw Lem, he argues that the widening scope of science leaves fewer people able to concentrate on any one niche. There are too many possible research problems for humanity to staff every field at the scale required. Automating discovery is his proposed response to that human-resource constraint.

Socher places that project within a longer account of technological evolution. Biological evolution took billions of years; humanity’s technological progress accelerated through farming, science, and industrialization, creating successive growth regimes even as individual exponentials eventually flatten into S-curves. He agrees with the core proposition in Marc Andreessen’s techno-optimist manifesto that technology is the only perpetual source of economic growth, while saying he does not agree with everything Andreessen writes. He distinguishes material from psychological problems, but argues that humanity has repeatedly addressed material constraints through invention.

His historical reference point is the movement from sustained powered flight to the moon landing within one lifetime. His wager is that AI could move from being worse than people at most work to being better at particular tasks over roughly 30 to 60 years. The operational consequence is not simply more capable software: if agents can run research loops across many narrow problems at once, they could extend scientific work beyond the number of people available to staff each subfield.

Socher’s proposed method is blunt: “The best way to build it is to have it build itself.”

Socher grounds automated research in Karl Popper’s model of variation and selection. A researcher proposes a theory, hypothesis, explanation, or description; it is then subjected to rigorous empirical testing and criticism. A theory that survives is provisionally accepted, rather than permanently settled.

That makes science structurally similar, in Socher’s telling, to evolution. Competing theories must hold up against alternative theories and against evidence. They are selected because they survive demanding tests, not because they are designated true in advance. An automated research system therefore does not need final answers at the outset. It needs to produce candidate ideas, run tests, interpret results, retain useful context, and direct its next experiment.

Socher reduces the loop to ideation, implementation, and validation. The fuller process shown in his presentation begins with an agent proposing an idea, implementing it, running an experiment, validating the outcome, and using what it learned to select another experiment. The system is meant to run many research threads over long horizons, retain context from earlier experiments, and combine promising branches. Validation has a specific role: it should detect reward hacks and variance before an apparent improvement is treated as a real signal.

The intended system is not a general-purpose chatbot that happens to help a scientist. Socher describes an agent swarm coordinating knowledge, data, experiments, and rewards. If a system can first become effective at AI research itself, he argues, the same research process could later extend into other scientific fields.

Socher describes the Eureka machine as a full-stack system rather than a foundation model operating alone. Its architecture has four pillars: a model of existing human knowledge; grounding in scientific measurements and data; high-fidelity simulations for phenomena not yet directly measurable or cheaply testable; and autonomous physical labs for real-world experiments when simulation is insufficient.

An overarching swarm of recursively improving agents is meant to work across those layers: finding and interpreting knowledge, choosing experiments, operating simulations, and handling data and rewards. The order is important to Socher. “You cannot simulate what you do not know,” he argues, making the knowledge layer a prerequisite for simulation and physical experimentation rather than an ancillary retrieval feature.

That knowledge layer cannot be a static model alone. Even powerful models do not contain all current, accurate, task-relevant information. They need search in a form designed for machines rather than people. At You.com, Socher says, the work is on web search for LLMs and agents. He contrasts the familiar human interface of ten blue links and short snippets with agents that can read thousands of much longer passages.

Search, browsers, the internet, and GPUs therefore become infrastructure for scientific superintelligence in his account. Each was largely built around human users and workflows. The opportunity, he says, is to rebuild these layers as tools for AI systems that need to access knowledge, use compute, operate software, test hypotheses, and eventually act on the physical world.

Socher distinguishes automation of a bounded research task from what he calls true recursive self-improvement, or RSI. A system that improves another model, searches for an architecture, or tunes a training setup may be an important demonstration. But it is not necessarily improving itself recursively.

For Socher, RSI requires a system to recognize shortcomings in its own operation and have access to the relevant parts of its own arsenal: pretraining, reinforcement-learning training, harnesses, and the wider system around them. It must then update that entire system into a next version of itself. The recursive element is not merely repeated AI research. It is improving the machinery doing the research, then using that improved machinery again over longer horizons.

As he puts it, true recursive self-improvement means an AI has “a sense of self awareness of its own shortcomings,” full access to its pretraining, RL training, harnesses, and other components, and can update that entire system into a next version of itself.

He sees the current moment as newly plausible because AI is code and can increasingly write code. Models’ ability to complete software tasks over longer durations, he says, has emerged only within the previous six to eight months. That creates the possibility, in his view, for a system to work on its own code and research setup rather than execute only a fixed task.

Socher is explicit that the examples he presents are neither a completed Eureka machine nor full RSI. They are “simple proof points” for automated research on bounded software workloads: improving training quality, training speed, and GPU kernels.

The three demonstrations are benchmarked AI-engineering tasks, not demonstrations of autonomous discovery in biology, medicine, or physical science. Socher presents them as bounded evidence that an automated research loop can generate useful technical changes rather than merely search a narrow hyperparameter space. Together, they address training quality, training speed, and kernel efficiency.

The NanoChat experiment uses a small chat model that can be trained in less than five minutes and targets lower validation bits per byte, or BPB. Socher says the community had reached roughly 0.93, while the automated system reached 0.9109 after a little more than one or two days. The presentation reports a 0.0283-BPB improvement evaluated over 10 random seeds.

What matters to Socher is the kind of change the system produced. Rather than only carefully tuning hyperparameters, he says, it found hashed bigram and trigram embedding tables and mixed local n-gram information into the attention value path through learned gates. The presentation characterizes this as a cheaper way to capture local sequence information than slower convolutional or attention-heavy alternatives.

NanoGPT Speedrun targets training time to a 3.28 validation loss under the benchmark’s specified workload, using full GPT-2 context and a standard validation test loss. Socher says the system reached that target in 77.5 seconds, beating a displayed prior record of 79.7 seconds. Its modifications included FP8 attention projections in the forward pass for greater tensor-core throughput, BF16 backward passes for stability, annealed exploration noise in the optimizer, cautious sign-agreement on embedding updates, and a leaner fused MLP kernel.

The third example is CUDA kernel optimization on SOL-ExecBench, a set of 235 kernel-writing tasks derived from real workloads. In the presentation, higher SOL scores are better and 1.0 represents an analytical estimate of optimal performance. The automated system reached a mean SOL score of 0.754, compared with 0.699 for the displayed leaderboard best, which the slide describes as an 18% reduction in the remaining gap to 1.0.

Socher situates the kernel task in a broader concern about GPU utilization. He says large mixture-of-experts clusters can still run at around 30% utilization, leaving substantial room for efficiency gains. His team did not include people who had spent their careers specializing in CUDA kernels, he says, but worked with NVIDIA to check for reward hacks and other issues. The kernels ultimately checked out and, according to Socher, found the best solutions across the reported categories.

The demonstrations are deliberately narrow: systems propose modifications, implement them, measure results, and retain the changes that outperform a benchmark. For Socher, that is the relevant first step—not evidence that the machine has already become a general scientific researcher, but evidence that parts of the research loop can produce measured technical gains.

Socher does not argue that AI is close to a final ceiling. He argues almost the reverse: talk of an intelligence exponential can obscure how many capabilities remain underdeveloped. His list of “spaces of intelligence” includes visual intelligence, natural-language intelligence, knowledge, reasoning, computational speed, physical intelligence, social intelligence and morality, creativity, metacognition, and survival and replication.

Intelligence, in this view, is not a single line on which a system approaches a known endpoint. It is a multidimensional space with many possible upper bounds. Once those dimensions are specified, Socher says, present systems remain “astronomically far away” from the limits across nearly all of them.

Every exponential can flatten, and Socher says it is genuinely difficult to know how much longer AI progress can continue at its current rate. His claim is narrower than a declaration of arrival: recursively improving research systems may create a new capability regime, beginning with AI engineering and potentially extending the amount of scientific work humanity can undertake.