A History of Artificial Intelligence: From Turing's Vision to the Transformer Era

Introduction

Artificial intelligence — the field concerned with building machines that can think, reason, and learn — has a history spanning more than seven decades. Its trajectory has been far from smooth: punctuated by periods of extraordinary optimism, followed by disappointment and funding droughts known as "AI winters," and then by revolutions in hardware and data that enabled capabilities previously considered impossible. Understanding this history illuminates why modern AI systems work the way they do, and what challenges remain.

The study of AI history is not merely academic. The choices made in the 1950s about how to represent knowledge — as symbolic rules versus numerical weights — echo in modern debates about interpretability. The failures of the 1970s foreshadowed current challenges with brittleness and out-of-distribution generalization. The successes of expert systems in narrow domains anticipated the fine-tuning paradigm. And the pattern of boom-and-bust that characterized early AI investment provides context for evaluating current claims about AI's near-term trajectory.

This document traces the major milestones from Alan Turing's foundational 1950 paper through the transformer revolution that began in 2017 and continues to reshape the field today. Each era produced lasting ideas, even when the technology of the time could not fully realize them.

The Turing Test and Early Foundations (1936–1955)

The intellectual foundations of AI predate the computer itself. Alan Turing, the British mathematician who laid theoretical groundwork for modern computing with his 1936 paper "On Computable Numbers," asked a deceptively simple question in his 1950 paper "Computing Machinery and Intelligence": Can machines think? Rather than defining machine thought philosophically, Turing proposed an operational test — the "Imitation Game," later called the Turing Test. In it, a human interrogator communicates via text with two participants, one human and one machine. If the interrogator cannot reliably distinguish the machine from the human, the machine is said to have demonstrated human-level conversational intelligence.

The Turing Test set the agenda for AI research for decades. It framed intelligence as behavior: not an internal property of a system, but a relationship between a system and the tasks it performs. This behavioral perspective would influence researchers who built chatbots, natural language processing systems, and eventually the large language models of the 2020s.

In 1943, Warren McCulloch and Walter Pitts published a model of how biological neurons might compute logical functions — the first formal mathematical model of a neural network. Claude Shannon developed information theory in 1948, providing the mathematical language to describe computation and communication. By 1950, the stage was set for the formal birth of the field.

The Dartmouth Conference and the First Wave (1956–1969)

Artificial intelligence as a named discipline was born at the 1956 Dartmouth Summer Research Project on Artificial Intelligence, organized by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon. McCarthy coined the term "artificial intelligence" for the proposal. The organizers held the optimistic view that "every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it."

The decade that followed produced remarkable results. Allen Newell and Herbert Simon developed the Logic Theorist (1956), which could prove mathematical theorems, and the General Problem Solver (1957), an attempt at a universal reasoning program. Simon and Newell famously predicted that within ten years, a computer would be the world's chess champion and would prove a significant new mathematical theorem. These predictions proved premature by several decades — chess would not fall until Kasparov vs. Deep Blue in 1997, and the mathematical theorem milestone came even later with automated proof systems.

Early AI programs operated through symbolic reasoning: representing knowledge as explicit rules and logical relationships, and applying those rules via search. LISP, the programming language developed by McCarthy in 1958, became the primary vehicle for AI research for the next three decades. Its list-processing capabilities made it ideal for manipulating symbolic structures. Systems like ELIZA (1966), developed by Joseph Weizenbaum at MIT, simulated conversation by pattern matching on user input — creating the illusion of understanding without genuine comprehension. Weizenbaum was disturbed by how readily people attributed emotional depth to what was, in his view, a trivial program.

The period also saw early work on machine learning. Frank Rosenblatt developed the Perceptron in 1957, an algorithm that could learn to classify simple visual inputs by adjusting weights based on errors. The Perceptron captured public imagination — the New York Times reported it as "the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence." Such breathless coverage set expectations that the technology would not meet for another sixty years.

The First AI Winter (1970–1980)

The optimism of the 1950s and 1960s collided with hard reality in the early 1970s. Computers were far too slow and memory-limited to scale symbolic AI systems to real-world complexity. Natural language understanding turned out to be incomparably harder than early researchers assumed — human language is ambiguous, context-dependent, and deeply embedded in physical and social experience that computers lacked entirely. No one had adequately grappled with what it would mean to give a machine common sense.

Two influential reports marked the turn: the 1966 ALPAC report concluded that machine translation was not cost-effective and recommended cutting government funding; the 1973 Lighthill Report in the United Kingdom reached similarly skeptical conclusions about AI in general, arguing that no important application had been demonstrated. Funding from DARPA and other government agencies was sharply reduced. Academic AI research contracted significantly, and many researchers moved to adjacent fields.

This period, retrospectively called the first AI winter, was not a complete stagnation. Expert systems — programs that encoded the knowledge of human specialists as explicit rules — were quietly proving useful in narrow domains. DENDRAL (1965–1969) identified chemical compounds from mass spectrometry data with accuracy comparable to specialist chemists. MYCIN (1972–1980) diagnosed bacterial infections and recommended antibiotics with accuracy comparable to expert physicians, though it was never deployed clinically due to liability concerns. These systems demonstrated that AI could add real value in bounded, well-defined domains, even if they could not generalize.

Marvin Minsky and Seymour Papert's 1969 book "Perceptrons" had delivered a significant blow to neural network research by proving mathematically that single-layer perceptrons could not solve problems that were not linearly separable — including the simple XOR function. While their critique applied only to single-layer networks (not multi-layer ones), it effectively defunded neural network research for over a decade.

The Expert Systems Era and the Second AI Wave (1980–1987)

Expert systems experienced a commercial boom in the early 1980s. XCON (also called R1), deployed by Digital Equipment Corporation in 1980, configured minicomputer orders and reportedly saved DEC $40 million annually by the mid-1980s. Companies like Intellicorp, Teknowledge, and Carnegie Group commercialized expert system development environments. The AI industry grew to a billion-dollar sector.

Japan's Fifth Generation Computer Project, announced in 1982, aimed to build massively parallel computers running logic-based AI languages by 1991. The project alarmed Western governments, prompting DARPA to dramatically increase AI funding in the United States and the UK to fund the Alvey Programme. Expert systems were considered strategic national technology.

The backpropagation algorithm, though first described in the 1970s, gained renewed attention through David Rumelhart, Geoffrey Hinton, and Ronald Williams' influential 1986 paper demonstrating its effectiveness for training multi-layer neural networks. Backpropagation showed that the limitations Minsky and Papert had identified for single-layer networks did not apply to deeper architectures. Neural networks could, in principle, learn arbitrary functions given sufficient data and layers. This would prove to be one of the most important theoretical results in the history of AI — though its full impact would not be felt for another two decades.

The Second AI Winter (Late 1980s–1990s)

The expert systems boom collapsed in the late 1980s. The systems proved expensive to build and maintain. Adding new knowledge required significant manual effort from domain experts. Systems were brittle — they failed in unpredictable ways when presented with situations not covered by their rules. The hardware they required (specialized LISP machines from companies like Symbolics and Lisp Machines Inc.) was expensive and could not compete with increasingly capable general-purpose workstations from Sun and others.

Japan's Fifth Generation project failed to meet its ambitious goals and wound down without producing the revolutionary systems it had promised. DARPA and other funders drastically cut AI funding once again. By 1993, the LISP machine market had collapsed entirely.

Neural network research, too, hit a wall. Despite the theoretical promise of backpropagation, training deep networks in practice was difficult. The vanishing gradient problem meant that gradients became exponentially smaller as they were propagated backward through many layers, making it nearly impossible to learn useful representations in the early layers of a deep network. Computational resources were insufficient for the large datasets that deep networks required. The field settled into a quieter period of incremental progress.

The Rise of Machine Learning (1990s–2000s)

The second AI winter drove AI research toward more rigorous mathematical foundations. Statistical machine learning — using probabilistic models and optimization theory rather than hand-coded rules — began to dominate. The field increasingly looked like applied statistics and optimization, which helped it attract more rigorous theoretical treatment.

Support Vector Machines (SVMs), developed by Vapnik and others in the 1990s, provided powerful classification algorithms with strong theoretical guarantees. Random forests and gradient boosting methods gave practitioners tools that were robust, interpretable, and effective on tabular data. These methods dominated competitive machine learning throughout the 2000s and into the early 2010s — Kaggle competitions were frequently won by gradient boosting variants.

IBM's Deep Blue defeated world chess champion Garry Kasparov in 1997, though through brute-force search augmented with human chess knowledge rather than learned intelligence. IBM's Watson won Jeopardy! in 2011 using sophisticated information retrieval and statistical NLP — again, not through learned general intelligence but through a clever combination of existing techniques. These victories generated enormous public attention but obscured the distinction between narrow performance and general intelligence.

The Deep Learning Revolution (2012 and Beyond)

The watershed moment came at the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton submitted AlexNet, a deep convolutional neural network trained on GPUs. AlexNet achieved a top-5 error rate of 15.3%, compared to 26.2% for the second-best entry. The gap was so large that it caused immediate reassessment across computer vision.

AlexNet's success rested on several convergent developments: the availability of ImageNet, a database of 14 million labeled images assembled by Fei-Fei Li and colleagues; the programmability of modern GPUs for massively parallel computation; the ReLU activation function which mitigated the vanishing gradient problem; and dropout regularization which reduced overfitting. These ingredients had all existed separately; AlexNet demonstrated their combination.

Deep learning spread rapidly to speech recognition, natural language processing, and other domains. Google's deep learning efforts improved speech recognition dramatically. DeepMind's AlphaGo defeated world Go champion Lee Sedol in 2016, a milestone many researchers had believed was at least a decade away — Go's enormous branching factor had seemed to preclude brute-force approaches, and AlphaGo's use of deep reinforcement learning represented a genuinely new technique.

Word Vectors and Early NLP (2013–2017)

Natural language processing lagged computer vision in adopting deep learning, partly because language has discrete symbolic structure that resisted purely numeric treatment. Word2Vec, published by Tomas Mikolov and colleagues at Google in 2013, was a turning point. Word2Vec showed that simple neural networks trained to predict words from context (or context from words) learned dense vector representations that captured semantic relationships: the famous example being that vector("king") - vector("man") + vector("woman") ≈ vector("queen").

GloVe (Global Vectors for Word Representation), published by Pennington, Socher, and Manning at Stanford in 2014, improved on Word2Vec's training stability. These pretrained word vectors became foundational infrastructure — models for sentiment analysis, named entity recognition, and machine translation all improved by initializing with pretrained word embeddings rather than random vectors.

Recurrent neural networks (RNNs), particularly Long Short-Term Memory (LSTM) networks developed by Hochreiter and Schmidhuber in 1997, became the standard architecture for sequence modeling. LSTMs maintain a "cell state" that can persist information across long sequences, addressing the vanishing gradient problem in the temporal dimension. They powered the best machine translation systems, speech recognition engines, and text generation models of the mid-2010s.

The Transformer Architecture (2017)

In 2017, a team at Google Brain published "Attention Is All You Need," introducing the Transformer architecture. The Transformer dispensed with recurrence entirely, replacing it with a self-attention mechanism that could directly model relationships between any two positions in a sequence, regardless of distance.

Self-attention computes, for each token in a sequence, a weighted sum of all other tokens' representations, where the weights are determined by a learned compatibility function between tokens. This allows the model to "attend" to relevant context anywhere in the sequence simultaneously — unlike RNNs, which must pass information through intermediate states sequentially.

The implications were profound. Transformers were highly parallelizable, enabling much more efficient training on modern hardware. They scaled gracefully with both model size and training data. And they proved extremely general — applicable not just to text but to images, audio, protein sequences, code, and any modality that could be tokenized.

BERT and GPT-1 (2018–2019)

Google's BERT (Bidirectional Encoder Representations from Transformers), published in 2018, demonstrated that pretraining a Transformer on massive text corpora and then fine-tuning on specific tasks produced state-of-the-art results across a wide range of NLP benchmarks. BERT's bidirectional pretraining — masking random words and predicting them from both left and right context — created richly contextual word representations.

OpenAI's GPT-1 (Generative Pretrained Transformer), also published in 2018, took a different approach: autoregressive pretraining on text (predicting the next word), then fine-tuning. GPT-1 was modest in scale — 117 million parameters — but demonstrated the principle. GPT-2 (2019) scaled this to 1.5 billion parameters and generated text of such surprising fluency and coherence that OpenAI initially declined to release the full model, citing concerns about potential misuse for generating disinformation.

GPT-3 and Few-Shot Learning (2020)

GPT-3, published by OpenAI in May 2020, contained 175 billion parameters — more than 100 times larger than GPT-2. The scale jump produced qualitatively different behavior. Most strikingly, GPT-3 demonstrated powerful few-shot and zero-shot capabilities: given a few examples of a task in the prompt, it could perform that task on new inputs without any weight updates. This was "in-context learning" — learning from examples provided at inference time, not during training.

GPT-3 could translate between languages, write code, answer trivia questions, perform arithmetic, summarize text, and generate coherent long-form prose — all from a single pretrained model, steered only by the prompt. The implication was that scale, combined with sufficient training data, could produce a form of general-purpose language understanding that had eluded the field for decades.

The GPT-3 paper noted that many capabilities seemed to emerge abruptly at scale — present in the 175B model but absent in smaller versions. This "emergent capabilities" phenomenon would become a major topic of research in subsequent years.

ChatGPT and the Current Era (2022–Present)

OpenAI's ChatGPT, released in November 2022, applied Reinforcement Learning from Human Feedback (RLHF) to GPT-3.5, aligning the model's outputs to be more helpful, harmless, and honest in conversation. RLHF fine-tuned the model to produce responses that human raters preferred, transforming a powerful but raw text predictor into a capable conversational assistant.

ChatGPT reached 100 million users within two months of launch — the fastest adoption of any consumer technology product to that point. It demonstrated to a broad public that large language models could hold coherent multi-turn conversations, answer complex questions, write and debug code, draft essays and emails, and assist with a remarkable range of cognitive tasks.

GPT-4 (March 2023) was multimodal — capable of processing images as well as text — and showed further improvements in reasoning, instruction-following, and reliability. Anthropic released Claude (April 2023), Google released Bard (later Gemini), and Meta released LLaMA, opening up open-weight model development. The field entered a period of intense competition, rapid capability gains, and equally rapid scaling of deployment.

Context windows expanded dramatically: from GPT-3's 4,096 tokens (roughly 3,000 words) to Claude's 200,000 tokens (roughly 150,000 words) and Gemini 1.5 Pro's 1 million tokens. These long-context models enabled new applications: analyzing entire codebases, summarizing book-length documents, maintaining context across very long conversations.

The challenges ahead include: reliably grounding model outputs in factual sources (reducing "hallucinations"), extending reasoning capabilities to complex multi-step tasks, ensuring safety and alignment as capabilities scale, reducing inference cost to enable broader access, and understanding why these systems work as well as they do — the mechanistic understanding of large language models remains surprisingly incomplete.

The Scaling Hypothesis and Emergent Capabilities

One of the most consequential ideas to emerge from the deep learning era is the "scaling hypothesis" — the proposition that many AI capabilities will continue improving predictably as model size, dataset size, and compute increase, even without algorithmic innovations. This hypothesis, associated most prominently with GPT-3 and subsequent work, suggested that the path to more capable AI was largely a matter of resources rather than fundamental breakthroughs.

The OpenAI team's 2020 scaling laws paper by Kaplan et al. showed that language model loss followed smooth power laws as a function of compute, dataset size, and number of parameters. Critically, these relationships held across many orders of magnitude without apparent saturation. If you doubled the compute budget, you could predict roughly how much better your model would get.

Emergent capabilities complicated the scaling story in interesting ways. Some behaviors — multi-step reasoning, code generation, solving novel analogical problems — appeared to arise abruptly at certain scales, being essentially absent in smaller models and then suddenly present in larger ones. Wei et al. (2022) documented over 100 such emergent capabilities. Critics noted that many apparent "emergences" were artifacts of the metrics used — smooth underlying improvements looked abrupt when measured by discontinuous metrics like exact-match accuracy.

The "chinchilla" scaling laws, published by Hoffmann et al. at DeepMind in 2022, refined the earlier understanding. The team argued that most large models were significantly under-trained relative to their size: given a fixed compute budget, it was better to train a smaller model on more data than a larger model on less data. Their chinchilla model (70B parameters, trained on 1.4 trillion tokens) outperformed GPT-3 (175B parameters, trained on 300 billion tokens) on most benchmarks. This insight reshaped how subsequent models like Llama 2 and Claude were trained.

Retrieval-Augmented Generation and Grounding

One of the central limitations of large language models is that their knowledge is frozen at training time. A model trained through 2021 cannot answer questions about events in 2024. More troublingly, models "hallucinate" — confidently asserting facts that are simply untrue, a consequence of the autoregressive prediction process generating plausible-sounding continuations without genuine fact retrieval.

Retrieval-Augmented Generation (RAG), introduced by Lewis et al. at Facebook AI in 2020, offered a partial solution. Rather than relying solely on knowledge encoded in weights during pretraining, a RAG system retrieves relevant documents from an external database at inference time and provides them to the model as context. The model is then asked to generate a response grounded in the retrieved material.

RAG architectures have several advantages: they can be updated without retraining (just update the retrieval database), they can cite sources, and they reduce hallucination by anchoring responses in retrieved text. They also directly interface with the context window considerations covered in this lesson: retrieved documents must fit within the model's available context alongside the query and generation.

Vector databases — systems like Chroma, Pinecone, Weaviate, and Milvus — emerged as infrastructure for RAG, enabling fast approximate nearest-neighbor search over millions of document embeddings. The combination of pretrained language models and vector retrieval became a standard architecture for knowledge-intensive applications.

Instruction Tuning and RLHF

Raw pretrained language models are powerful but difficult to direct. A model pretrained to predict the next word will, given the prompt "How do I write good code?", attempt to complete a plausible continuation of that text — which might be more training data examples rather than a helpful response. Bridging the gap between pretraining and deployment required significant additional work.

Instruction fine-tuning, demonstrated by the InstructGPT paper (Ouyang et al., 2022) and the FLAN series from Google, showed that fine-tuning on a curated dataset of instruction-response pairs dramatically improved a model's ability to follow user requests. The model learned that instructions should be followed, questions should be answered, and the register of conversation is different from the register of raw text.

Reinforcement Learning from Human Feedback (RLHF) went further. A separate "reward model" was trained to predict which of two model responses human raters preferred. The language model was then fine-tuned using reinforcement learning to maximize this reward signal — essentially, learning to produce responses that humans found more helpful, accurate, and appropriate. RLHF was central to ChatGPT's conversational capability and to the development of Claude.

Constitutional AI (CAI), developed by Anthropic, offered an alternative approach to alignment that reduced dependence on human ratings. Rather than training a reward model from human preferences alone, CAI specified a set of principles (a "constitution") and used the model itself to provide AI feedback on adherence to those principles. This self-critique loop could scale more efficiently than gathering human ratings for every comparison.

Open-Weight Models and Democratization

Until 2023, the most capable large language models were proprietary — accessible only through APIs from companies like OpenAI, Anthropic, and Google. The release of Meta's LLaMA model family changed this dynamic. LLaMA (February 2023), though initially released only for research use, was widely shared and became the foundation for hundreds of community fine-tunes and derivatives.

LLaMA 2 (July 2023) was released more permissively, enabling commercial use. Instruction-tuned derivatives like Alpaca, Vicuna, and Mistral demonstrated that smaller open-weight models (7B–70B parameters) could match or approach GPT-3.5 class performance on many tasks, particularly after instruction fine-tuning.

The open-weight ecosystem enabled academic research, local deployment (running models on personal hardware), fine-tuning for specialized domains, and innovation in inference optimization. Techniques like quantization (reducing numerical precision from 32-bit or 16-bit floats to 8-bit or 4-bit integers), knowledge distillation (training small models to mimic large ones), and speculative decoding enabled capable models to run on consumer hardware. Ollama, LM Studio, and similar tools made local LLM deployment accessible to non-specialists.

Multimodality and the Expansion Beyond Text

The Transformer architecture's generality made it natural to extend beyond text. Vision Transformer (ViT), published by Dosovitskiy et al. in 2020, divided images into patches and treated them as sequences, achieving competitive performance with convolutional networks on image classification. DALL-E, Stable Diffusion, and Midjourney applied diffusion models conditioned on text to image generation, enabling the creation of photorealistic images from natural language descriptions.

GPT-4V (Vision) extended GPT-4 to accept images as input. Claude 3 Sonnet and Opus could analyze charts, diagrams, photographs, and documents. Gemini 1.5 Pro accepted images, video, audio, and text simultaneously. The integration of modalities enabled new classes of applications: automatic accessibility descriptions, medical image analysis assistance, document understanding, and multimodal search.

Audio understanding — speech recognition, speaker identification, tone analysis — followed similar patterns. OpenAI's Whisper, released in 2022, provided near-human speech recognition across many languages. GPT-4o (Omni) integrated real-time audio processing, enabling voice conversations with low latency.

Conclusion

The history of AI is a history of cycles: ambition outpacing capability, followed by disillusionment, followed by technical breakthroughs that vindicated parts of the original vision while revealing new difficulties. What is different about the current era is the breadth of capability now demonstrably available from a single class of system — the large language model. Whether this represents the beginning of a path to general intelligence, or a powerful but ultimately limited narrow technology, remains one of the central open questions in science.

The practical challenges ahead are significant: reliably grounding model outputs in factual sources (reducing hallucinations), extending reasoning to complex multi-step tasks, ensuring safety and alignment as capabilities scale, reducing inference cost to enable broader access, and building mechanistic understanding of why these systems work. The context window — the amount of information a model can process at once — remains a fundamental constraint, driving the research described in this lesson on chunking, summarization, and retrieval strategies.

What is not in question is that the field has changed permanently. Every domain that involves producing or processing language — which is to say, nearly every domain of human intellectual activity — is being reshaped by systems whose capabilities would have seemed like science fiction to the researchers who gathered at Dartmouth in 1956. The journey from Turing's thought experiment to trillion-parameter multimodal systems took less than a human lifetime — and the pace of change shows no sign of slowing.