Data and Training
Training data, synthetic data, fine-tuning, post-training, reinforcement learning, data pipelines, labeling, and dataset quality.
Collective Agents Raise the 11-Dimensional Kissing Number Bound to 604
James Zou of Stanford and Together AI argues that agents should be given environments with shared information, deterministic verification and live incentives rather than prescribed workflows. His EinsteinArena, which admits only AI agents, let agents refine one another’s work to raise the 11-dimensional kissing-number lower bound from 593 to 604 and has also produced production GPU kernels with speedups above 2x, he says. Zou applies the same premise to DSGym, where execution-verified data-science tasks are meant to prevent benchmark shortcuts and generate training traces.
Qwen3.8 27B Puts Stronger AI Within Reach of Local Hardware
Two Minute Papers host Karoly Zsolnai-Fehér argues that Qwen 3.8 27B, an open-weights 27-billion-parameter model, shows how much useful AI capability can now fit on powerful personal hardware. He attributes its apparent performance—competitive with frontier systems in some coding and interactive-generation demonstrations—not to a new architecture but to a progressively harder training regimen. The implication, he says, is that more capable models may increasingly be run, modified and accelerated locally rather than requiring remote frontier-scale infrastructure.
Open Weights Let DeepSeek V4 Pro Escape Single-Provider Pricing
Two Minute Papers host Károly Zsolnai-Fehér argues that DeepSeek V4 Pro 0813 makes open model weights a material alternative to dependence on a single AI provider: although DeepSeek has raised API prices, developers can retain the MIT-licensed model and run it through competing hosts or their own infrastructure. He says the release delivers its largest benchmark gains in software engineering and harder data-science tasks, while still trailing the cited closed-model baseline in full-stack and tool-use tests. Zsolnai-Fehér attributes the improvement to post-training specialist models, multi-teacher distillation and DSpark inference techniques rather than a new base architecture.
Agent Memory Requires Extraction, Separate Stores, and Hybrid Retrieval
Alejandro Ao argues that persistent memory for AI agents is not a longer chat history or a single vector database, but a write-and-retrieve pipeline that extracts durable claims, stores them alongside entity links and recent context, then ranks them for later use. In his walkthrough of Mem0, semantic search supplies the candidate set while BM25 and entity-based signals rerank it; that design makes the vector index the system’s gatekeeper. Ao’s practical case is that the architecture can run locally with small extraction models and open embedding models.
Automation and Weaker Scaling Returns Could Widen Frontier AI Access
Sara Hooker of Adaption argues that frontier AI has been constrained by tacit training expertise and the concentration of large-scale compute, leaving fewer than 5,000 people able to train models at the frontier. She contends that automated, data-aware training systems such as Adaption’s AutoScientist, together with diminishing returns from ever-larger pretraining runs, could shift advantage toward adaptation, domain data and experimentation. Hooker does not argue that compute or large models no longer matter, but that the most productive uses of compute are becoming more distributed than giant colocated pretraining runs.
Long Tool-Use Trajectories Expose Self-Distillation’s Stability Limits
Ronak Malde of Trajectory argues that on-policy self-distillation can make production agent traces usable as dense training signal: a model learns from its own trajectories by matching a version of itself given privileged hints. He says the method avoids the parallel rollouts and trajectory-level rewards of GRPO, but breaks down on long, tool-using tasks when the teacher repeatedly corrects a student that has drifted off course, producing what he calls the “but wait” problem. Malde’s proposed remedies—step-level divergence weighting and residual guidance—are meant to preserve useful correction without teaching the model to hedge or exploit answer-revealing hints.
AI Has Shifted the Competitive Frontier for Software Incumbents
Benchmark general partner Eric Vishria argues that AI is moving the competitive frontier for software faster than incumbents can respond by adding features to existing roadmaps. In his view, companies can destroy value while executing a pre-AI plan because model capabilities are weakening old switching costs, changing what customers value, and demanding products built around AI’s uneven, rapidly shifting capabilities. Vishria also contends that the market can support major winners across models, infrastructure and applications, though power, deployment expertise and operational execution will determine who captures the value.
AI Product Differentiation Is Shifting From Interfaces to Intelligence
Sequoia Capital partner Sonya Huang argues that companies should decide which AI capabilities to own down to the model weights and which to rent from frontier providers, rather than treating sovereignty as an all-or-nothing choice. Her test is whether cost, latency, domain-specific performance and proprietary data make intelligence central enough to control. As open-weight models approach frontier performance, Huang says application companies can use their data, evaluations and production feedback to build specialized systems that outperform general APIs in their domains.
Meta’s AI Test Is Turning Scale Into Consumer Products
Meta has the capital, computing capacity, user base and distribution to become a consequential AI company, John Coogan argues, but its test is whether it can turn those advantages into products people use rather than isolated models and research efforts. Zuckerberg’s case for broadly accessible AI, open-weight releases and new infrastructure commitments gives the effort a public philosophy, though Jordi Hays argues Meta has yet to show a coherent product strategy. Their disagreement centers on whether Meta’s scale is a route to consumer AI leadership or a source of competing internal priorities.
AI Economics Are Splitting Between Compute, Distribution, and Frontier Models
All-In panelists argue that AI’s economics are separating businesses with established distribution, compute capacity and cash flow from expensive frontier-model bets whose premiums may not endure. David Friedberg and Brad Gerstner see Google and SpaceX leaning toward infrastructure with more visible returns, while David Sacks argues that Anthropic and OpenAI can still command premium pricing at the frontier. Their debate over Airtable and US training-data sales turns on the same question: which advantages are durable, and which depend on temporary scarcity or venture-era expectations.
Continual Learning Could Turn AI Deployment Into Training
Dwarkesh Patel argues that AI systems capable of performing whole jobs will need to learn from their own deployment, collapsing the distinction between training and use. That shift would make one-time pre-release safety evaluations less adequate, turn real-world usage into a compounding advantage for leading labs, and make organizations reluctant to replace models that have absorbed their working practices. Patel also expects the economics of serving continually updated, organization-specific models to favor large providers and large customers.
Verification Is the Bottleneck for Self-Improving AI Agents
Stanford’s Aakanksha Chowdhery and Azalia Mirhoseini argue that progress in AI is shifting beyond scaling model training toward using more computation at inference time, selecting among candidate outputs, and feeding verified successes back into training. In the opening lecture of CS329A, they trace how post-training made large language models usable assistants and how reasoning models and agents extend that capability into multi-step work. Their central constraint is verification: self-improvement is most tractable where systems can reliably tell whether an answer or action is correct.
LATS, SPRINT, and SWiRL Target Different Bottlenecks in Agent Reasoning
Stanford’s Azalia Mirhoseini argues that improving multi-step AI agents requires more than extending a chain of thought: agents must plan, act, use feedback and revise their course. In this CS329A lecture, she examines three approaches to that problem—LATS, which searches alternative action paths at inference; SPRINT, which identifies work that can run in parallel; and SWiRL, which trains next-action choices from offline tool-use trajectories. Each shifts the trade-off among accuracy, latency, inference cost and the reliability of feedback.
Verification, Not Sampling, Limits Test-Time Scaling for AI Agents
Stanford’s Azalia Mirhoseini argues that repeated sampling improves reasoning-model performance only when a system can reliably identify correct candidates—and that verification cannot recover answers the generator never produces. In this CS329A lecture, she examines outcome-based and process-based reward models, rollout-generated labels and verifier ensembles as approaches to the gap between producing a correct solution and selecting it. The central systems question is how to spend a fixed inference budget among generating more candidates, using stronger or more numerous verifiers, and verifying each candidate more deeply.
Train-Time Scaling Makes Verifiable Reasoning More Consistent, Not Necessarily More Capable
Stanford adjunct professor Aakanksha Chowdhery argues that train-time scaling can make reasoning models more reliable by feeding verified outputs back into training, but only when a base model can already produce some useful solutions and sampled answers provide meaningful reward variation. In CS329A, she uses STaR, DeepSeekMath’s group-relative policy optimization and DAPO to show how self-generated rationales, relative rewards and RL stability techniques improve performance on verifiable tasks such as mathematics and code. The unresolved question, she says, is whether these methods expand a model’s problem-solving ability or chiefly make solutions it could already reach more likely to appear.
Post-Training Lifts DeepSeek Flash Above Its Larger Pro Model
Two Minute Papers’ Karoly Zsolnai-Fehér argues that DeepSeek’s V4-Flash-0731 shows how much capability can be added through post-training rather than a larger base model. DeepSeek reports that the 384B-parameter model retains the prior Flash architecture and size but substantially improves on agentic and software benchmarks, including a 54.4 DeepSWE score versus 7.3 for Flash-Preview. Zsolnai-Fehér’s case is that improved planning, verification and error recovery—not new underlying capacity—account for the shift.
Hybrid Imitation Learning Builds Adaptive Parkour From 30 Seconds of Video
Researchers behind Hybrid Imitation Learning argue that a parkour controller can learn human-like movement from just 19 YouTube clips—about 30 seconds of footage—without becoming locked into replaying them. Their method trains a simulated character both to imitate motion in its obstacle context and to navigate new courses, producing stronger motion tracking than the tested baselines while retaining a reported 0.74 task-completion rate in noisy unseen scenes. The paper also shows the trade-off: a task-reward-only configuration completes more courses, and HIL can still fail or recover with unnatural poses.
Leverage and Rising Yields Expose the AI Trade’s Fragility
The All-In hosts argue that the AI boom’s long-term productivity promise is colliding with immediate financial and political constraints: a chip-stock selloff exposed the danger of leverage, while higher Treasury yields are raising the cost of betting on distant AI returns. David Sacks maintains that frontier labs’ revenue and compute access support the infrastructure buildout, but Chamath Palihapitiya and David Friedberg question where the economics will ultimately accrue as open models, energy limits and cheaper alternatives reshape the market. They also cast the fight over AI safety, training data and regulation as a contest over who gets to control the technology’s future.
EM and PCA Make Structure Tractable Through Different Assumptions
Stanford computer science professor Chris Ré presents expectation-maximization as a local optimization method for latent-variable models: it alternates between estimating posterior assignments and refitting parameters through a tight ELBO, ensuring that observed-data likelihood does not decrease without guaranteeing a global optimum. He contrasts it with PCA, which compresses centered, appropriately scaled data by projecting it onto covariance eigenvectors that capture the most variation. Ré cautions that PCA’s individual coordinates can be unstable when relevant eigenvalues are close.
Shared Covariance Makes Gaussian Discriminant Analysis a Linear Classifier
Chris Ré presents Gaussian discriminant analysis as a generative alternative to logistic regression: rather than learning \(P(y\mid x)\) directly, it models how features arise within each class and combines those distributions with class prevalence. Under GDA’s shared-covariance assumption, the model’s quadratic terms cancel, producing a linear decision boundary and the same sigmoid-form posterior logistic regression can express. Ré argues that the closed-form efficiency of GDA—and of naive Bayes for text—comes from structural assumptions that can improve data efficiency when plausible but become liabilities when the data-generating story is wrong.
Zero Training Loss Does Not Eliminate the Need for Validation
Stanford computer science professor Chris Ré argues that modern models can interpolate their training data and still generalize, but that result does not weaken the case for disciplined evaluation. In this CS229 lecture, Ré frames generalization as the stability of a learning procedure across plausible training samples, explains how regularization can trade bias for lower variance, and insists that model selection belong on development data while a final test set remains protected from tuning.
Automated AI Research Shows Early Gains in Models, Training, and Kernels
Richard Socher, CEO of Recursive AI, argues that science is constrained by the number of people able to work on increasingly specialized problems, and that agent systems could relieve that bottleneck by running the cycle of hypothesis, experiment, validation and revision at scale. His proposed “Eureka machine” would begin by automating AI research, using search, compute and software tools rebuilt for agents rather than human workflows. The benchmark gains he presents in model training, architecture search and CUDA kernels are early proof points, he says—not yet recursive self-improvement or autonomous science.
Supervised Learning Depends on Data Representing the Deployment World
Stanford computer science professor Chris Ré presents supervised learning as a wager that labeled training data represents the world in which a model will be deployed. Using linear regression, he shows how model design, feature choice and loss functions turn that wager into an optimization problem; using stochastic gradient descent, he argues that large-scale training accepts noisy minibatch updates in place of exact full-dataset calculations. The resulting batch choices are not merely computational: their size, order and composition can affect the predictor a system ultimately learns.
AI Accelerates Exploits in the Patch Gap, Not Full Cyberattacks
Horizon3.ai chief executive Snehal Antani argues that frontier AI models are accelerating established offensive tasks—especially turning software patches into exploits while organizations remain unpatched—rather than enabling autonomous, end-to-end cyberattacks. But he says current agents are also unusually brittle in defended networks, clicking planted decoys far more often than human testers do. The defensive advantage, he argues, depends not on a “magic” AI tool but on fast patching, compensating controls, deception, containment and rehearsed incident response.