News and Tools

“[With AI] we’re not building animals. We’re building ghosts or spirits.” – Andrej Karpathy – Ex-OpenAI, Ex-Tesla AI

Andrej Karpathy, renowned for his leadership roles at OpenAI and Tesla’s Autopilot programme, has been at the centre of advances in deep learning, neural networks, and applied artificial intelligence. His work traverses both academic research and industrial deployment, granting him a panoramic perspective on the state and direction of AI.

When Karpathy refers to building “ghosts or spirits,” he is drawing a conceptual line between biological intelligence—the product of millions of years of evolution—and artificial intelligence as developed through data-driven, digital systems. In his view, animals are “baked in” with instincts, embodiment, and innate learning capacities shaped by evolution, a process unfolding over geological timeframes. By contrast, today’s AI models are “ghosts” in the sense that they are ethereal, fully digital artefacts, trained to imitate human-generated data rather than to evolve or learn through direct interaction with the physical world. They lack bodily instincts and the evolutionary substrate that endows animals with survival strategies and adaptation mechanisms.

Karpathy describes the pre-training process that underpins large language models as a form of “crappy evolution”—a shortcut that builds digital entities by absorbing the statistical patterns of internet-scale data without the iterative adaptation of embodied beings. Consequently, these models are not “born” into the world like animals with built-in survival machinery; instead, they are bootstrapped as “ghosts,” imitating but not experiencing life.

The Cognitive Core—Karpathy’s Vision for AI Intelligence

Karpathy’s thinking has advanced towards the critical notion of the “cognitive core”: the kernel of intelligence responsible for reasoning, abstraction, and problem-solving, abstracted away from encyclopaedic factual knowledge. He argues that the true magic of intelligence is not in the passive recall of data, but in the flexible, generalisable ability to manipulate ideas, solve problems, and intuit patterns—capabilities that a system exhibits even when deprived of pre-programmed facts or exhaustive memory.

He warns against confusing memorisation (the stockpiling of internet facts within a model) with general intelligence, which arises from this cognitive core. The most promising path, in his view, is to isolate and refine this core, stripping away the accretions of memorised data, thereby developing something akin to a “ghost” of reasoning and abstraction rather than an “animal” shaped by instinct and inheritance.

This approach entails significant trade-offs: a cognitive core lacks the encyclopaedic reach of today’s massive models, but gains in adaptability, transparency, and the capacity for compositional, creative thought. By foregrounding reasoning machinery, Karpathy posits that AI can begin to mirror not the inflexibility of animals, but the open-ended, reflective qualities that characterise high-level problem-solving.

Karpathy’s Journey and Influence

Karpathy’s influence is rooted in a career spent on the frontier of AI research and deployment. His early proximity to Geoffrey Hinton at the University of Toronto placed him at the launch-point of the convolutional neural networks revolution, which fundamentally reshaped computer vision and pattern recognition.

At OpenAI, Karpathy contributed to an early focus on training agents to master digital environments (such as Atari games), a direction in retrospect he now considers premature. He found greater promise in systems that could interact with the digital world through knowledge work—precursors to today’s agentic models—a vision he is now helping to realise through ongoing work in educational technology and AI deployment.

Later, at Tesla, he directed the transformation of autonomous vehicles from demonstration to product, gaining hard-won appreciation for the “march of nines”—the reality that progressing from system prototypes that work 90% of the time to those that work 99.999% of the time requires exponentially more effort. This experience informs his scepticism towards aggressive timelines for “AGI” and his insistence on the qualitative differences between robust system deployment and controlled demonstrations.

The Leading Theorists Shaping the Debate

Karpathy’s conceptual framework emerges amid vibrant discourse within the AI community, shaped by several seminal thinkers:

Sutton’s “bitter lesson” posits that scale and generic algorithms, rather than domain-specific tricks, ultimately win—suggesting a focus on evolving animal-like intelligence. Karpathy, however, notes that current development practices, with their reliance on dataset imitation, sidestep the deep embodiment and evolutionary learning that define animal cognition. Instead, AI today creates digital ghosts—entities whose minds are not grounded in physical reality, but in the manifold of internet text and data.

Hinton and LeCun supply the neural and architectural foundations—the “cortex” and reasoning traces—while both Karpathy and their critics note the absence of rich, consolidated memory (the hippocampus analogue), instincts (amygdala), and the capacity for continual, self-motivated world interaction.

Why “Ghosts,” Not “Animals”?

The distinction is not simply philosophical. It carries direct consequences for:

• Capabilities: AI “ghosts” excel at pattern reproduction, simulation, and surface reasoning but lack the embodied, instinctual grounding (spatial navigation, sensorimotor learning) of animals.
• Limitations: They are subject to model collapse, producing uniform, repetitive outputs, lacking the spontaneous creativity and entropy seen in human (particularly child) cognition.
• Future Directions: The field is now oriented towards distilling this cognitive core, seeking a scalable, adaptable reasoning engine—compact, efficient, and resilient to overfitting—rather than continuing to bloat models with ever more static memory.

This lens sharpens expectations: the way forward is not to mimic biology in its totality, but to pursue the unique strengths and affordances of a digital, disembodied intelligence—a spirit of the datasphere, not a beast evolved in the forest.

Broader Significance

Karpathy’s “ghosts” metaphor crystallises a critical moment in the evolution of AI as a discipline. It signals a turning point: the shift from brute-force memorisation of the internet to intelligent, creative algorithms capable of abstraction, reasoning, and adaptation.

This reframing is shaping not only the strategic priorities of the most advanced labs, but also the philosophical and practical questions underpinning the next decade of AI research and deployment. As AI becomes increasingly present in society, understanding its nature—not as an artificial animal, but as a digital ghost—will be essential to harnessing its strengths and mitigating its limitations.

“AI is so wonderful because there have been a number of seismic shifts where the entire field has suddenly looked a different way. I’ve maybe lived through two or three of those. I still think there will continue to be some because they come with almost surprising regularity.” – Andrej Karpathy – Ex-OpenAI, Ex-Tesla AI

Andrej Karpathy, one of the most recognisable figures in artificial intelligence, has spent his career at the epicentre of the field’s defining moments in both research and large-scale industry deployment.

Karpathy’s background is defined by deep technical expertise and a front-row seat to AI’s rapid evolution. Having completed his PhD at Stanford and held pivotal research positions, he worked alongside Geoffrey Hinton at the University of Toronto during the early surge of deep learning. His career encompasses key roles at Tesla, where he led the Autopilot vision team, and at OpenAI, contributing to some of the world’s most prominent large language models and generative AI systems. This vantage point has allowed him to participate in, and reflect upon, the discipline’s “seismic shifts”.

Karpathy’s narrative has been shaped by three inflection points:

• The emergence of deep neural networks from a niche field to mainstream AI, spearheaded by the success of AlexNet and the subsequent shift of the research community toward neural architectures.
• The drive towards agent-based systems, with early enthusiasm for reinforcement learning (RL) and game-based environments (such as Atari and Go). Karpathy himself was cautious about the utility of games as the true path to intelligence, focusing instead on agents acting within the real digital world.
• The rise of large language models (LLMs)—transformers trained on vast internet datasets, shifting the locus of AI from task-specific systems to general-purpose models with the ability to perform a broad suite of tasks, and in-context learning.

His reflection on these ‘regular’ paradigm shifts arises from lived experience: “I’ve maybe lived through two or three of those. I still think there will continue to be some because they come with almost surprising regularity.” These moments recalibrate assumptions, redirect research priorities, and set new benchmarks for capability. Karpathy’s practical orientation—building “useful things” rather than targeting biological intelligence or pure AGI—shapes his approach to both innovation and scepticism about hype.

Context of the Quote
In his conversation with podcaster Dwarkesh Patel, Karpathy elaborates on the recurring nature of breakthroughs. He contrasts AI’s rapid, transformative leaps with other scientific fields, noting that in machine learning, scaling up data, compute, and novel architectures can yield abrupt improvements—yet each wave often triggers both excessive optimism and later recalibration. A major point he raises is the lack of linearity: the field does not “smoothly” approach AGI, but rather proceeds via discontinuities, often catalysed by new ideas or techniques that were previously out of favour or overlooked.

Karpathy relates how, early in his career, neural networks were a marginal interest and large-scale “representation learning” was only beginning to be considered viable by a minority in the community. With the advent of AlexNet, the landscape shifted overnight, rapidly making previous assumptions obsolete. Later, the pursuit of RL-driven agents led to a phase where entire research agendas were oriented toward gameplay and synthetic environments—another phase later superseded by the transformer revolution and language models. Karpathy reflects candidly on earlier missteps, as well as the discipline’s collective tendency to over- or under-predict the timetable and trajectory of progress.

Leading Theorists and Intellectual Heritage
The AI revolutions Karpathy describes are inseparable from the influential figures and ideas that have shaped each phase:

• Geoffrey Hinton: Hailed as the “godfather of AI”, Hinton was instrumental in deep learning’s breakthrough, advancing techniques for training multilayered neural networks and championing representation learning against prevailing orthodoxy.
• Yann LeCun: Developed convolutional neural networks (CNNs), foundational for computer vision and the 2010s wave of deep learning success.
• Yoshua Bengio: Co-architect of the deep learning movement and a key figure in developing unsupervised and generative models.
• Richard Sutton: Principal proponent of reinforcement learning, Sutton articulated the value of “animal-like” intelligence: learning from direct interaction with environments, reward, and adaptation. Sutton’s perspective frequently informs debates about the relationship between model architectures and living intelligence, encouraging a focus on agents and lifelong learning.

Karpathy’s own stance is partly a pragmatic response to this heritage: rather than pursuing analogues of biological brains, he views the productive path as building digital “ghosts”—entities that learn by imitation and are shaped by patterns in data, rather than evolutionary processes.

Beyond individual theorists, the field’s quantum leaps are rooted in a culture of intellectual rivalry and rapid intellectual cross-pollination:

• The convolutional and recurrent networks of the 2010s pushed the boundaries of what neural networks could do.
• The development and scaling of transformer-based architectures (as in Google’s “Attention is All You Need”) dramatically changed both natural language processing and the structure of the field itself.
• The introduction of algorithms for in-context learning and large-scale unsupervised pre-training marked a break with hand-crafted representation engineering.

The Architecture of Progress: Seismic Shifts and Pragmatic Tension
Karpathy’s insight is that these shifts are not just about faster hardware or bigger datasets; they reflect the field’s unique ecology—where new methods can rapidly become dominant and overturn accumulated orthodoxy. The combination of open scientific exchange, rapid deployment, and intense commercialisation creates fertile ground for frequent realignment.

His observation on the “regularity” of shifts also signals a strategic realism: each wave brings both opportunity and risk. New architectures (such as transformers or large reinforcement learning agents) frequently overshoot expectations before their real limitations become clear. Karpathy remains measured on both promise and limitation—anticipating continued progress, but cautioning against overpredictions and hype cycles that fail to reckon with the “march of nines” needed to reach true reliability and impact.

Closing Perspective
The context of Karpathy’s quote is an AI ecosystem that advances not through steady accretion, but in leaps—each driven by conceptual, technical, and organisational realignments. As such, understanding progress in AI demands both technical literacy and historical awareness: the sharp pivots that have marked past decades are likely to recur, with equally profound effects on how intelligence is conceived, built, and deployed.

“What I think we have to do going forward…is figure out ways to remove some of the knowledge and to keep what I call this cognitive core. It’s this intelligent entity that is stripped from knowledge but contains the algorithms and contains the magic of intelligence and problem-solving and the strategies of it and all this stuff.” – Andrej Karpathy – Ex-OpenAI, Ex-Tesla AI

Andrej Karpathy’s observation about the need to “strip away knowledge whilst retaining the cognitive core” represents one of the most penetrating insights into contemporary artificial intelligence development. Speaking on Dwarkesh Patel’s podcast in October 2025, Karpathy—formerly a leading figure at both OpenAI and Tesla’s Autopilot programme—articulated a fundamental tension at the heart of modern AI: the current generation of large language models have become prodigious memorisers, yet this very capability may be constraining their potential for genuine intelligence.

The Paradox of Pre-training

To comprehend Karpathy’s thesis, one must first understand the architecture of contemporary AI systems. Large language models are trained on vast corpora—often 15 trillion tokens or more—through a process called pre-training. During this phase, models learn to predict the next token in a sequence, effectively compressing the entire internet into their neural networks. Karpathy describes this compressed representation as only “0.07 bits per token” for a model like Llama 3 70B, highlighting the extraordinary degree of compression occurring.

This compression serves two distinct functions, which Karpathy carefully delineates. First, models accumulate factual knowledge—the content of Wikipedia articles, the specifics of historical events, the details of scientific papers. Second, and more crucially, they develop what Karpathy terms “algorithmic patterns”—the capacity for in-context learning, the ability to recognise and complete patterns, the fundamental mechanisms of reasoning itself.

The problem, as Karpathy sees it, is that contemporary models have become too adept at the former whilst the latter remains the true seat of intelligence. When a model can regurgitate passages verbatim or recite obscure facts, it demonstrates remarkable memory. But this same capability creates what he calls a “distraction”—the model becomes reliant on its hazy recollections of training data rather than developing robust reasoning algorithms that could operate independently of specific factual knowledge.

The Cognitive Core Concept

Karpathy’s proposed solution is to isolate and preserve what he terms the “cognitive core”—an intelligent entity stripped of encyclopaedic knowledge but retaining the fundamental algorithms of problem-solving, the strategies of thought, and what he describes as “the magic of intelligence.” This concept represents a profound shift in how we conceptualise artificial intelligence.

Consider the analogy to human cognition. Humans are remarkably poor memorisers compared to AI systems. Present a human with a random sequence of numbers, and they’ll struggle after seven or eight digits. Yet this apparent limitation forces humans to develop robust pattern-recognition capabilities and abstract reasoning skills. We’re compelled to “see the forest for the trees” precisely because we cannot memorise every individual tree.

Karpathy suggests that AI systems would benefit from similar constraints. A model with less memory but stronger reasoning capabilities would be forced to look up factual information whilst maintaining sophisticated algorithms for processing that information. Such a system would more closely resemble human intelligence—not in its limitations, but in the way those limitations drive the development of generalisable cognitive strategies.

The implications extend beyond mere technical architecture. Karpathy envisions cognitive cores as compact as one billion parameters—potentially even smaller—that could operate as genuine reasoning engines rather than glorified databases. These systems would “know that they don’t know” when confronted with factual questions, prompting them to retrieve information whilst applying sophisticated analysis. The result would be AI that thinks more than it remembers, that reasons rather than recites.

From Evolution to Engineering: The Path Not Taken

Karpathy’s perspective on AI development diverges sharply from what he calls the “Richard Sutton viewpoint”—the notion that we should build AI systems analogous to biological intelligence, learning from scratch through reinforcement learning in the manner of animals. Instead, Karpathy argues we’re building what he evocatively describes as “ghosts” or “spirit entities”—ethereal intelligences that emerge from imitating human-generated text rather than evolving through environmental interaction.

This distinction illuminates a crucial divergence in AI philosophy. Biological intelligence, as embodied in animals, emerges from evolution operating over millions of years, with vast amounts of capability “baked in” to neural circuitry. A zebra foal runs within minutes of birth not through reinforcement learning but through evolutionary encoding. Humans similarly arrive with substantial cognitive machinery pre-installed, with lifetime learning representing maturation and refinement rather than learning from first principles.

By contrast, contemporary AI systems learn through what Karpathy terms “crappy evolution”—pre-training on internet documents serves as a compressed, accelerated alternative to evolutionary optimisation. This process creates entities fundamentally different from biological intelligence, optimised for different tasks through different mechanisms. The current approach imitates the products of human intelligence (text, code, conversations) rather than replicating the developmental process that creates intelligence.

The Limits of Current Learning Paradigms

Karpathy’s critique extends to reinforcement learning, which he describes with characteristic bluntness as “terrible.” His concerns illuminate deep problems in how AI systems currently learn from experience. In reinforcement learning, a model generates hundreds of solution attempts, and those that arrive at correct answers have every intermediate step up-weighted, whilst failed attempts are down-weighted. Karpathy calls this “sucking supervision through a straw”—extracting minimal learning signal from vast amounts of computational work.

The fundamental issue is noise. When a solution works, not every step along the way was necessarily correct or optimal. The model may have taken wrong turns, pursued dead ends, or stumbled upon the answer despite flawed reasoning. Yet reinforcement learning broadcasts the final reward across the entire trajectory, reinforcing both good and bad reasoning indiscriminately. Karpathy notes that “you may have gone down the wrong alleys until you arrived at the right solution,” yet every mistaken step gets marked as something to do more of.

Humans, by contrast, engage in sophisticated post-hoc analysis. After solving a problem, we reflect on which approaches worked, which didn’t, and why. We don’t simply label an entire problem-solving session as “good” or “bad”—we dissect our reasoning, identify productive and unproductive strategies, and refine our approach. Current AI systems lack this reflective capacity entirely.

This limitation connects to broader questions about how AI systems might achieve continual learning—the ability to genuinely learn from ongoing experience rather than requiring massive retraining. Karpathy suggests that humans engage in a nightly “distillation phase” during sleep, processing the day’s experiences and integrating insights into long-term knowledge. AI systems have no equivalent mechanism. They simply restart from the same state each time, unable to evolve based on individual experiences.

Model Collapse and the Entropy Problem

A subtle but critical concern in Karpathy’s analysis is what he terms “model collapse”—the tendency of AI systems to produce outputs that occupy “a very tiny manifold of the possible space of thoughts.” Ask ChatGPT to tell a joke repeatedly, and you’ll receive the same three jokes. Request reflection on a topic multiple times, and you’ll observe striking similarity across responses. The models are “silently collapsed,” lacking the entropy and diversity that characterises human thought.

This phenomenon creates profound challenges for synthetic data generation, a technique labs use to create additional training material. If models generate training data for themselves or subsequent models, this collapsed distribution gradually dominates the training corpus. Training on one’s own outputs creates a dangerous feedback loop—each generation becomes less diverse, more stereotyped, more “collapsed” than the last. Karpathy suggests this may not even be a solvable problem, noting that humans similarly “collapse over time,” becoming more rigid and less creative as they age, revisiting the same thoughts and patterns with decreasing learning rates.

The contrast with children is illuminating. Young minds, not yet “overfitted” to the world, produce shocking, creative, unexpected responses precisely because they haven’t collapsed into standard patterns of thought. This freshness, this maintenance of high entropy in cognitive processes, may be essential to genuine intelligence. Yet our current training paradigms actively work against it, rewarding convergence towards common patterns and penalising deviation.

The Decade of Agents: Why Progress Takes Time

When Karpathy states this will be “the decade of agents” rather than “the year of agents,” he draws on hard-won experience from five years leading Tesla’s Autopilot programme. His insights into why artificial intelligence deployment takes far longer than demonstrations suggest carry particular weight given this background.

The central concept is what Karpathy calls “the march of nines.” Getting something to work 90% of the time—the level typically showcased in demonstrations—represents merely the first nine in “99.9%.” Each additional nine requires equivalent effort. During his tenure at Tesla, the team progressed through perhaps two or three nines over five years. More crucially, numerous nines remain before self-driving cars achieve true autonomy at scale.

This pattern isn’t unique to autonomous vehicles. Karpathy argues it applies across safety-critical domains, including software engineering. When code errors can leak millions of Social Security numbers or create critical security vulnerabilities, the cost of failure becomes prohibitively high. The demo-to-product gap widens dramatically. What works impressively in controlled conditions fails in countless edge cases when confronting reality’s full complexity.

Waymo’s experience illustrates this challenge. Despite providing “perfect drives” as early as 2014, the company still operates limited deployments requiring elaborate teleoperation infrastructure and supervision. Humans haven’t been removed; they’ve been rendered invisible, beaming in remotely to handle edge cases. The technology lives in a “pulled-back future”—functional but not yet economical, capable but not yet scalable.

Contemporary AI agents face analogous challenges. Whilst Claude and GPT-5 Pro demonstrate remarkable capabilities, they remain what Karpathy characterises as “elementary school students”—savants with perfect memory but lacking robust reasoning across all necessary dimensions. They’re “cognitively deficient” in ways users intuitively recognise even if they can’t articulate precisely what’s missing.

The Software Engineering Puzzle

Perhaps no domain better illustrates the puzzling contours of current AI capabilities than software engineering. Karpathy notes, somewhat ruefully, that whilst these systems were meant to enable “any economically valuable task,” API revenue remains “dominated by coding.” This supposedly general intelligence overwhelmingly excels at one specific domain.

This concentration isn’t accidental. Code enjoys unique properties that make it ideal for current AI architectures. Software development has always operated through text—terminals, editors, version control systems all manipulate textual representations. LLMs, trained on internet text, encounter code as a native format. Moreover, decades of infrastructure exist for handling code textually: diff tools for showing changes, IDEs for navigation, testing frameworks for verification.

Contrast this with domains lacking such infrastructure. Creating presentations involves spatial arrangement and visual design—there’s no “diff” for slides that elegantly shows modifications. Many knowledge work tasks involve physical documents, in-person interactions, or tacit knowledge that resists textual representation. These domains haven’t been pre-optimised for AI interaction in the way software development has.

Yet even in coding, Karpathy remains sceptical of current capabilities for cutting-edge work. When building nanoChat, a repository implementing a complete ChatGPT clone in simplified form, he found AI tools valuable for autocomplete and handling familiar patterns but inadequate for novel architectural decisions. The models kept trying to impose standard approaches when he deliberately chose non-standard implementations. They couldn’t comprehend his custom solutions, constantly suggesting deprecated APIs and bloating code with unnecessary defensive programming.

This points to a deeper truth: current models excel at reproducing common patterns from their training data but struggle with code “that has never been written before”—precisely the domain of frontier AI research itself. The recursive self-improvement that some forecast, where AI systems rapidly enhance their own capabilities, founders on this limitation. Models can accelerate work within established paradigms but cannot yet pioneer truly novel approaches.

The Trajectory of Intelligence Explosion

Karpathy’s perspective on potential intelligence explosions diverges sharply from both pessimistic and optimistic extremes. He sees AI not as a discrete, alien technology but as a continuation of computing’s evolution—part of an ongoing automation trend stretching back through compilers, high-level programming languages, and computer-aided design tools. From this view, the “intelligence explosion” has already been occurring for decades, visible in the exponential GDP growth curve that represents accumulated automation across countless domains.

This framing leads to counterintuitive predictions. Rather than expecting AI to suddenly accelerate economic growth from 2% annually to 20%, Karpathy suggests it will enable continued progress along the existing exponential trajectory. Just as computers, the internet, and mobile phones transformed society without producing visible discontinuities in aggregate growth statistics, AI will diffuse gradually across industries, maintaining rather than disrupting established growth patterns.

This gradualism doesn’t imply insignificance. The compounding effects of sustained exponential growth produce extraordinary transformation over time. But it does suggest that simple extrapolations from impressive demonstrations to imminent superintelligence misunderstand how technology integrates into society. There will be no discrete moment when “AGI” arrives and everything changes. Instead, we’ll experience continuous advancement in capabilities, continuous expansion of automation, and continuous adaptation of society to new technological possibilities.

The analogy to the Industrial Revolution proves instructive. That transformation didn’t occur through a single breakthrough but through cascading improvements across multiple technologies and practices, gradually shifting society from 0.2% annual growth to 2%. Similarly, AI’s impact will emerge through countless incremental deployments, each automating specific tasks, enabling new workflows, and creating feedback loops that accelerate subsequent progress.

The Human Element: Education in an AI Future

Karpathy’s work on Eureka, his educational initiative, reveals his deepest concerns about AI’s trajectory. He fears not that AI will fail but that “humanity gets disempowered by it,” relegated to the sidelines like the portly, passive citizens of WALL-E. His solution lies in radically reimagining education around the principle that “pre-AGI education is useful; post-AGI education is fun.”

The analogy to fitness culture illuminates this vision. Nobody needs physical strength to manipulate heavy objects—we have machines for that. Yet gyms proliferate because exercise serves intrinsic human needs: health, aesthetics, the satisfaction of physical mastery. Similarly, even in a world where AI handles most cognitive labour, humans will pursue learning for its inherent rewards: the pleasure of understanding, the status of expertise, the deep satisfaction of mental cultivation.

But achieving this vision requires solving a technical problem: making learning genuinely easy and rewarding. Currently, most people abandon learning because they encounter material that’s too difficult or too trivial, bouncing between frustration and boredom. Karpathy describes the experience of working with an expert language tutor who maintained a perfect calibration—always presenting challenges at the edge of current capability, never boring, never overwhelming. This created a state where “I was the only constraint to learning,” with knowledge delivery perfectly optimised.

Replicating this experience at scale represents what Karpathy sees as education’s great technical challenge. Current AI tutors, despite their sophistication, remain far from this standard. They can answer questions but cannot probe understanding, identify gaps, or sequence material to create optimal learning trajectories. The capability exists in exceptional human tutors; the challenge lies in encoding it algorithmically.

Yet Karpathy sees this challenge as tractable. Just as AI has transformed coding through autocomplete and code generation, it will eventually transform education through personalised, responsive tutoring. When learning becomes “trivial”—not in the sense of requiring no effort but in the sense of encountering no artificial obstacles—humans will pursue it enthusiastically. Not everyone will become an expert in everything, but the ceiling on human capability will rise dramatically as the floor on accessibility descends.

The Physics of Understanding: Karpathy’s Pedagogical Philosophy

Karpathy’s approach to teaching reveals principles applicable far beyond AI. His background in physics instilled what he describes as finding “first-order terms”—identifying the essential, dominant factors in any system whilst recognising that second and third-order effects exist but matter less. This habit of abstraction, of seeing spherical cows where others see only messy complexity, enables the creation of minimal, illustrative examples that capture phenomena’s essence.

MicroGrad exemplifies this approach perfectly. In 100 lines of Python, Karpathy implements backpropagation—the fundamental algorithm underlying all neural network training. Everything else in modern deep learning frameworks, he notes, is “just efficiency”—optimisations for speed, memory management, numerical stability. But the intellectual core, the actual mechanism by which networks learn, fits in 100 comprehensible lines. This distillation makes the previously arcane accessible.

The broader principle involves “untangling knowledge”—reorganising understanding so each concept depends only on what precedes it. This creates “ramps to knowledge” where learners never encounter gaps or leaps that would require them to take claims on faith. The famous transformer tutorial embodies this, beginning with a simple bigram model (literally a lookup table) and progressively adding components, each motivated by solving a specific limitation of what came before.

This approach contrasts sharply with the standard academic practice of presenting solutions before establishing problems, or introducing abstractions before concrete examples. Karpathy sees such approaches as, in his words, “a dick move”—they rob learners of the opportunity to grapple with challenges themselves, to develop intuition about what solutions might work, and to appreciate why particular approaches succeed where alternatives fail. The pedagogical crime isn’t challenging students; it’s presenting answers without first establishing questions.

Leading Theorists: The Intellectual Lineage

Richard Sutton and the Bitter Lesson

Richard Sutton, a pioneering reinforcement learning researcher, articulated what became known as “the bitter lesson”—the observation that simple, scalable methods leveraging computation consistently outperform approaches incorporating human knowledge or structural assumptions. His perspective suggests that the path to artificial general intelligence lies through learning algorithms powerful enough to discover structure from experience, much as evolution discovered biological intelligence.

Sutton’s famous assertion that “if you got to the squirrel, you’d be most of the way to AGI” reflects this view. Animal intelligence, in his framework, represents the core achievement—the fundamental learning algorithms that enable organisms to navigate environments, solve problems, and adapt to challenges. Human language and culture, whilst impressive, represent relatively minor additions to this foundation.

Karpathy respectfully dissents. His “we’re building ghosts, not animals” formulation captures the divergence: current AI systems don’t replicate the learning processes that create biological intelligence. They imitate the products of human intelligence (text, code, reasoning traces) rather than replicating its developmental origins. This distinction matters profoundly for predicting AI’s trajectory and understanding its capabilities and limitations.

Geoffrey Hinton and the Neural Network Renaissance

Geoffrey Hinton, often termed the “godfather of AI,” pioneered the neural network approaches that underpin contemporary systems. His persistence through decades when neural networks were unfashionable, his development of backpropagation techniques, and his later work on capsule networks and other architectures established the foundation for today’s large language models.

Karpathy studied directly under Hinton at the University of Toronto, experiencing firsthand the intellectual ferment as deep learning began its ascent to dominance. Hinton’s influence appears throughout Karpathy’s thinking—the emphasis on learning from data rather than hand-crafted rules, the focus on representation learning, the conviction that scale and simplicity often trump elaborate architectural innovations.

Yet Karpathy’s view extends beyond his mentor’s. Where Hinton focused primarily on perception (particularly computer vision), Karpathy grapples with the full scope of intelligence—reasoning, planning, continual learning, multi-agent interaction. His work synthesises Hinton’s foundational insights with broader questions about cognitive architecture and the nature of understanding itself.

Yann LeCun and Convolutional Networks

Yann LeCun’s development of convolutional neural networks in 1989 represented the first successful application of gradient descent to real-world pattern recognition. His work on handwritten digit recognition established core principles: the power of hierarchical feature learning, the importance of translation invariance, the value of specialised architectures for specific domains.

Karpathy’s reconstruction of LeCun’s 1989 network, time-travelling 33 years of algorithmic improvements, reveals his appreciation for this lineage. He found that pure algorithmic advances—modern optimisers, better architectures, regularisation techniques—could halve error rates. But achieving further gains required more data and more computation. This trinity—algorithms, data, compute—advances in lockstep, with no single factor dominating.

This lesson shapes Karpathy’s predictions about AI’s future. He expects continued progress across all three dimensions, with the next decade bringing better algorithms, vaster datasets, more powerful hardware, and more efficient software. But no breakthrough in any single dimension will produce discontinuous acceleration. Progress emerges from the intersection of many incremental improvements.

The Broader Intellectual Context

The debate Karpathy engages extends beyond specific individuals to fundamental questions about intelligence itself. Does intelligence arise primarily from general learning algorithms (the Sutton view) or from accumulated structure and innate mechanisms (the evolutionary perspective)? Can we build intelligence by imitating its products (the current LLM approach) or must we replicate its developmental processes? Will artificial intelligence remain fundamentally tool-like, augmenting human capability, or evolve into genuinely autonomous agents pursuing their own goals?

These questions connect to century-old debates in psychology and cognitive science between behaviourists emphasising learning and nativists emphasising innate structure. They echo discussions in evolutionary biology about the relative roles of genetic determination and developmental plasticity. They parallel arguments in philosophy of mind about whether intelligence requires embodiment or can exist as pure information processing.

Karpathy’s position threads between extremes. He acknowledges both the power of learning from data and the necessity of architectural structure. He recognises both the distinctiveness of AI systems and their illuminating analogies to biological intelligence. He balances optimism about AI’s potential with realism about current limitations and the difficulty of translating demonstrations into robust, deployed systems.

The Cognitive Core in Context: A New Paradigm for Intelligence

The concept of a cognitive core stripped of factual knowledge represents more than a technical proposal—it’s a reconceptualisation of what intelligence fundamentally is. Rather than viewing intelligence as encompassing both reasoning algorithms and accumulated knowledge, Karpathy proposes treating these as separate, with reasoning capability as the essence and factual knowledge as external resources to be accessed rather than internalised.

This separation mirrors certain aspects of human cognition whilst diverging in others. Humans do maintain a distinction between knowing how to think and knowing specific facts—we can reason about novel situations without direct experience, applying general problem-solving strategies learned in one domain to challenges in another. Yet our factual knowledge isn’t purely external; it shapes the very structure of our reasoning, creating rich semantic networks that enable rapid, intuitive judgement.

The proposal to strip AI systems down to cognitive cores involves accepting tradeoffs. Such systems would need to perform external lookups for factual information, introducing latency and dependency on knowledge bases. They would lack the pattern-matching capabilities that arise from vast memorisation, potentially missing connections between superficially unrelated domains. They might struggle with tasks requiring seamless integration of many small facts, where lookup costs would dominate processing time.

Yet the gains could prove transformative. A genuine cognitive core—compact, efficient, focused on algorithmic reasoning rather than fact retrieval—could operate in settings where current models fail. Edge deployment becomes feasible when models don’t require storing terabytes of parameters. Personalisation becomes practical when core reasoning engines can be fine-tuned or adapted without retraining on entire knowledge corpora. Interpretability improves when reasoning processes aren’t obscured by retrieval of memorised patterns.

Most profoundly, genuine cognitive cores might avoid the collapse and loss of entropy that plagues current models. Freed from the burden of maintaining consistency with vast memorised datasets, such systems could explore more diverse solution spaces, generate more varied outputs, and maintain the creative flexibility that characterises human cognition at its best.

Implications for the Decade Ahead

Karpathy’s decade-long timeline for agentic AI reflects hard-earned wisdom about technology deployment. His experience with autonomous vehicles taught him that impressive demonstrations represent merely the beginning of a long productisation journey. Each additional “nine” of reliability—moving from 90% to 99% to 99.9% accuracy—requires comparable effort. Safety-critical domains demand many nines before deployment becomes acceptable.

This reality shapes expectations for AI’s economic impact. Rather than sudden disruption, we’ll witness gradual diffusion across domains with varying characteristics. Tasks that are repetitive, well-defined, purely digital, and allowing high error rates will automate first. Call centre work exemplifies this profile—short interaction horizons, clear success criteria, limited context requirements, tolerance for occasional failures that human supervisors can catch.

More complex knowledge work will resist automation longer. Radiologists, consultants, accountants—professionals whose work involves lengthy timescales, subtle judgements, extensive context, and high costs of error—will see AI augmentation before replacement. The pattern will resemble Waymo’s current state: AI handling routine cases whilst humans supervise, intervene in edge cases, and maintain ultimate responsibility.

This graduated deployment creates an “autonomy slider”—a continuous spectrum from pure human operation through various degrees of AI assistance to eventual full automation. Most jobs won’t flip discretely from human to machine. Instead, they’ll slide along this spectrum as AI capabilities improve and organisations develop confidence in delegation. This process will unfold over years or decades, not months.

The economic implications differ from both optimistic and pessimistic extremes. We won’t see overnight mass unemployment—the gradual nature of deployment, the persistence of edge cases requiring human judgement, and society’s adaptation through creating new roles all mitigate disruption. But neither will we see disappointing underutilisation—the compound effect of many small automations across countless tasks will produce genuine transformation.

The Path Forward: Research Priorities

Karpathy’s analysis suggests several critical research directions for developing robust, capable AI systems. First, developing methods to isolate cognitive cores from memorised knowledge whilst maintaining reasoning capability. This might involve novel training objectives that penalise rote memorisation whilst rewarding generalisation, or architectural innovations that separate knowledge storage from reasoning mechanisms.

Second, creating effective continual learning systems that can distil experience into lasting improvements without catastrophic forgetting or model collapse. This requires moving beyond simple fine-tuning toward something more akin to the reflection and consolidation humans perform during sleep—identifying patterns in experience, extracting lessons, and integrating insights whilst maintaining diversity.

Third, advancing beyond current reinforcement learning to richer forms of learning from experience. Rather than broadcasting sparse reward signals across entire trajectories, systems need sophisticated credit assignment that identifies which reasoning steps contributed to success and which didn’t. This might involve explicit review processes where models analyse their own problem-solving attempts, or meta-learning approaches that learn how to learn from experience.

Fourth, developing multi-agent systems with genuine culture—shared knowledge bases that agents collectively maintain and evolve, self-play mechanisms that drive capability improvement through competition, and organisational structures that enable collaboration without centralized control. Current systems remain fundamentally solitary; genuine agent economies will require breakthroughs in coordination and communication.

Fifth, and perhaps most ambitiously, maintaining entropy in AI systems—preventing the collapse toward stereotyped outputs that currently plagues even frontier models. This might involve explicit diversity penalties, adversarial training to prevent convergence, or inspiration from biological systems that maintain variation through mechanisms like mutation and recombination.

Conclusion: Intelligence as Engineering Challenge

Andrej Karpathy’s vision of the cognitive core represents a mature perspective on artificial intelligence—neither breathlessly optimistic about imminent superintelligence nor dismissively pessimistic about current limitations. He sees AI as an engineering challenge rather than a mystical threshold, requiring patient work across multiple dimensions rather than awaiting a single breakthrough.

This perspective derives from direct experience with the messy reality of deploying AI systems at scale. Self-driving cars that work perfectly in demonstrations still require years of refinement before handling edge cases reliably. Coding agents that generate impressive solutions for common problems still struggle with novel architectural challenges. Educational AI that answers questions adequately still falls far short of expert human tutors’ adaptive responsiveness.

Yet within these limitations lies genuine progress. Models continue improving along multiple dimensions simultaneously. Infrastructure for deploying and managing AI systems grows more sophisticated. Understanding of these systems’ capabilities and constraints becomes more nuanced. The path forward is visible, even if it stretches further than optimists anticipated.

The concept of stripping knowledge to reveal the cognitive core captures this mature vision perfectly. Rather than pursuing ever-larger models memorising ever-more data, we might achieve more capable intelligence through subtraction—removing the crutch of memorisation to force development of robust reasoning algorithms. Like humans compelled to abstract and generalise because we cannot remember everything, AI systems might benefit from similar constraints.

This vision offers hope not for sudden transformation but for steady progress—the kind that compounds over decades into revolutionary change. It suggests that the hard technical problems of intelligence remain tractable whilst acknowledging their genuine difficulty. Most importantly, it positions humans not as passive observers of AI’s ascent but as active participants in shaping its development and ensuring its integration enhances rather than diminishes human flourishing.

The decade ahead will test these ideas. We’ll discover whether cognitive cores can be effectively isolated, whether continual learning mechanisms can be made robust, whether the demo-to-product gap can be bridged across diverse domains. The answers will shape not just the trajectory of AI technology but the future of human society in an increasingly automated world. Karpathy’s contribution lies in framing these questions with clarity, drawing on hard-won experience to guide expectations, and reminding us that the most profound challenges often yield to patient, disciplined engineering rather than waiting for miraculous breakthroughs.