Due Diligence

“AI is so wonderful because there have been a number of seismic shifts where the entire field has suddenly looked a different way. I’ve maybe lived through two or three of those. I still think there will continue to be some because they come with almost surprising regularity.” – Andrej Karpathy – Ex-OpenAI, Ex-Tesla AI

Andrej Karpathy, one of the most recognisable figures in artificial intelligence, has spent his career at the epicentre of the field’s defining moments in both research and large-scale industry deployment.

Karpathy’s background is defined by deep technical expertise and a front-row seat to AI’s rapid evolution. Having completed his PhD at Stanford and held pivotal research positions, he worked alongside Geoffrey Hinton at the University of Toronto during the early surge of deep learning. His career encompasses key roles at Tesla, where he led the Autopilot vision team, and at OpenAI, contributing to some of the world’s most prominent large language models and generative AI systems. This vantage point has allowed him to participate in, and reflect upon, the discipline’s “seismic shifts”.

Karpathy’s narrative has been shaped by three inflection points:

• The emergence of deep neural networks from a niche field to mainstream AI, spearheaded by the success of AlexNet and the subsequent shift of the research community toward neural architectures.
• The drive towards agent-based systems, with early enthusiasm for reinforcement learning (RL) and game-based environments (such as Atari and Go). Karpathy himself was cautious about the utility of games as the true path to intelligence, focusing instead on agents acting within the real digital world.
• The rise of large language models (LLMs)—transformers trained on vast internet datasets, shifting the locus of AI from task-specific systems to general-purpose models with the ability to perform a broad suite of tasks, and in-context learning.

His reflection on these ‘regular’ paradigm shifts arises from lived experience: “I’ve maybe lived through two or three of those. I still think there will continue to be some because they come with almost surprising regularity.” These moments recalibrate assumptions, redirect research priorities, and set new benchmarks for capability. Karpathy’s practical orientation—building “useful things” rather than targeting biological intelligence or pure AGI—shapes his approach to both innovation and scepticism about hype.

Context of the Quote
In his conversation with podcaster Dwarkesh Patel, Karpathy elaborates on the recurring nature of breakthroughs. He contrasts AI’s rapid, transformative leaps with other scientific fields, noting that in machine learning, scaling up data, compute, and novel architectures can yield abrupt improvements—yet each wave often triggers both excessive optimism and later recalibration. A major point he raises is the lack of linearity: the field does not “smoothly” approach AGI, but rather proceeds via discontinuities, often catalysed by new ideas or techniques that were previously out of favour or overlooked.

Karpathy relates how, early in his career, neural networks were a marginal interest and large-scale “representation learning” was only beginning to be considered viable by a minority in the community. With the advent of AlexNet, the landscape shifted overnight, rapidly making previous assumptions obsolete. Later, the pursuit of RL-driven agents led to a phase where entire research agendas were oriented toward gameplay and synthetic environments—another phase later superseded by the transformer revolution and language models. Karpathy reflects candidly on earlier missteps, as well as the discipline’s collective tendency to over- or under-predict the timetable and trajectory of progress.

Leading Theorists and Intellectual Heritage
The AI revolutions Karpathy describes are inseparable from the influential figures and ideas that have shaped each phase:

• Geoffrey Hinton: Hailed as the “godfather of AI”, Hinton was instrumental in deep learning’s breakthrough, advancing techniques for training multilayered neural networks and championing representation learning against prevailing orthodoxy.
• Yann LeCun: Developed convolutional neural networks (CNNs), foundational for computer vision and the 2010s wave of deep learning success.
• Yoshua Bengio: Co-architect of the deep learning movement and a key figure in developing unsupervised and generative models.
• Richard Sutton: Principal proponent of reinforcement learning, Sutton articulated the value of “animal-like” intelligence: learning from direct interaction with environments, reward, and adaptation. Sutton’s perspective frequently informs debates about the relationship between model architectures and living intelligence, encouraging a focus on agents and lifelong learning.

Karpathy’s own stance is partly a pragmatic response to this heritage: rather than pursuing analogues of biological brains, he views the productive path as building digital “ghosts”—entities that learn by imitation and are shaped by patterns in data, rather than evolutionary processes.

Beyond individual theorists, the field’s quantum leaps are rooted in a culture of intellectual rivalry and rapid intellectual cross-pollination:

• The convolutional and recurrent networks of the 2010s pushed the boundaries of what neural networks could do.
• The development and scaling of transformer-based architectures (as in Google’s “Attention is All You Need”) dramatically changed both natural language processing and the structure of the field itself.
• The introduction of algorithms for in-context learning and large-scale unsupervised pre-training marked a break with hand-crafted representation engineering.

The Architecture of Progress: Seismic Shifts and Pragmatic Tension
Karpathy’s insight is that these shifts are not just about faster hardware or bigger datasets; they reflect the field’s unique ecology—where new methods can rapidly become dominant and overturn accumulated orthodoxy. The combination of open scientific exchange, rapid deployment, and intense commercialisation creates fertile ground for frequent realignment.

His observation on the “regularity” of shifts also signals a strategic realism: each wave brings both opportunity and risk. New architectures (such as transformers or large reinforcement learning agents) frequently overshoot expectations before their real limitations become clear. Karpathy remains measured on both promise and limitation—anticipating continued progress, but cautioning against overpredictions and hype cycles that fail to reckon with the “march of nines” needed to reach true reliability and impact.

Closing Perspective
The context of Karpathy’s quote is an AI ecosystem that advances not through steady accretion, but in leaps—each driven by conceptual, technical, and organisational realignments. As such, understanding progress in AI demands both technical literacy and historical awareness: the sharp pivots that have marked past decades are likely to recur, with equally profound effects on how intelligence is conceived, built, and deployed.