Where Did AI Come From?

The figure is about thirty centimetres tall and carved from a mammoth’s tusk. It stands upright on two legs, with a mostly human body and a lion’s head. It was made about forty thousand years ago. It is in the Museum Ulm in southern Germany, which describes it as the oldest known depiction of a mythical creature. The museum is closed for rebuilding, and the figure goes on show again in early 2027.^{1, 3}

It came out of the ground in pieces. On 25 August 1939, the last day of a dig at the Stadel cave in the Swabian Jura, the excavators lifted out fragments of worked ivory.^{2, 3} A week later Germany invaded Poland, and the fragments went into storage. For thirty years nobody saw a figure in them. In December 1969 the archaeologist Joachim Hahn, taking an inventory of the pieces in the museum’s store, noticed carving on them and fitted more than two hundred together.^{2, 3} Fragments added in 1982 showed that the head was a cat’s.² New excavations at the cave between 2009 and 2013 found more, and the figure is now assembled from over three hundred pieces.³

We know what it is made of, how old it is and where it lay. We do not know what it meant. Its maker may have thought of it as a god, an ancestor, a character in a story, or a person dressed in a lion’s skin. Even its sex cannot be settled. As the magazine Archaeology put it, “neither the gender of the animal nor of its human parts is discernible.”²

About forty thousand years later, around 1250, late in the Chola period, an unnamed caster in South India made a bronze of Narasimha, Vishnu in his man-lion form. Narasimha is one of the ten main avatars of Vishnu, and for devotees his form recalls the defeat of a demon who seemed impossible to beat.⁴ This bronze leaves the fight out. Narasimha sits with his legs crossed, held in place by a yoga band, his four arms spread in balance. His upper right hand holds a flaming discus. The face above the mane is set in concentration. Two posts rise on either side of him, and they once held a halo. The Cleveland Museum of Art, which owns the figure, reads it as royal power joined to yogic discipline.⁴

Neither figure is an early form of artificial intelligence, and nothing connects either one to a computer by descent. What they share with artificial intelligence is older. For at least forty thousand years, people have given shape to minds that are not simply human. The Lion Man’s maker left no account of what that mind was. Narasimha’s makers left a full one: his name, his story, his posture and the way he is read are written down and still practised.^{4, 5, 6} Both were made to be looked at, not to act. The wish that a made thing might answer back came much later, and for a long time it stayed a story. When it became a programme of work, the steps were taken by named people at known dates, and each carried a decision about what intelligence is.

The made thing that answers

In the tenth book of Ovid’s Metamorphoses, a poem he was still revising when he was banished to the Black Sea coast in 8 CE, the sculptor Pygmalion carves a woman “out of snow-white ivory” and falls in love with what he has made.⁷ At the festival of Venus he prays, shyly, for a wife like his ivory girl. When he gets home and touches the statue, the ivory softens under his fingers and grows warm. She is alive, and she looks back at him.

Ovid does not give her a name. She was called Galatea much later, and the name was first given wide circulation by Rousseau’s Pygmalion, probably written in 1762.⁸ The story records, in Ovid’s Rome, the wish for a made thing that answers back. A goddess grants it, and the statue comes to life as what her maker prayed for.

Nineteen centuries later the story was still being carved. Rodin made a model of the scene in 1889, and it was carved in marble, not ivory, about twenty years later. In the version now in New York, the sculptor sits against the stone with his head at the woman’s waist, while she leans away from him with her lower legs still part of the rough block.⁹

The made things that did answer were engineered. Around 1206, an engineer from upper Mesopotamia, Ibn al-Razzaz al-Jazari, finished a book of mechanical devices for the ruler Nasir al-Din. It covers clocks, trick vessels for drinking parties, fountains, machines for raising water and devices for washing.¹⁰ One of them is a basin for washing the hands. Water pours from the beak of a peacock, and a small figure comes out of a door to offer soap.

The painted leaf is not from 1206. It comes from a copy made in Damascus in 1315, one of fifteen copies of the book that survive.¹⁰ The museum that owns it is careful on a further point: nobody knows for certain that the peacock basin was ever built.¹⁰ Al-Jazari worked in a tradition of automata that was already old, known in the ancient, Byzantine and Islamic worlds.

Everything the peacock was designed to do was fixed by its maker. Its mechanism would decide when the figure appeared, and it would appear the same way every time. It was meant to respond when someone used it, but nothing in it would change through use. To make a thing that changed through use, someone first had to ask whether a machine could think at all.

Intelligence becomes a research programme

In October 1950 the mathematician Alan Turing published a paper in the philosophy journal Mind. Its first sentence is: “I propose to consider the question, ‘Can machines think?’”¹¹ Turing then set the question aside. If “machine” and “think” were defined by how the words are commonly used, he argued, the answer would have to come from an opinion poll, which was absurd. So “I shall replace the question by another.” The replacement was a game. An interrogator puts written questions to two hidden players, one a person and one a machine, and tries to tell which is which. Turing’s choice was to judge a machine by its conduct, by what it said, and to leave aside what went on inside it.

Near the end of the paper Turing made a second suggestion: “Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child’s?“¹¹ Build something like a child’s mind, and then educate it.

Five years later four researchers made a different bet. On 31 August 1955, John McCarthy, Marvin Minsky, Nathaniel Rochester and Claude Shannon dated a proposal for “a 2 month, 10 man study of artificial intelligence” at Dartmouth College in the summer of 1956.¹² The proposal rested on one conjecture: that “every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.” Like Turing’s game, it was a decision about what intelligence is.

For the next thirty years, much of the field wrote descriptions by hand. Allen Newell and Herbert Simon built the Logic Theorist, a program that proved theorems in logic, which they date to 1955. Two decades later they stated the idea behind it as a hypothesis: “A physical symbol system has the necessary and sufficient means for general intelligent action.”¹³

From 1965, Edward Feigenbaum and his colleagues at Stanford built DENDRAL, a program that worked out the structure of chemical compounds from the readings of laboratory instruments such as a mass spectrometer. Feigenbaum described how they were made. A “knowledge engineer” worked intensively with a human expert, drew out what the expert knew, and wrote it into the program as rules.¹⁴

Each of these choices met its limit. In 1972 the mathematician James Lighthill wrote a survey of the field for Britain’s Science Research Council, published the following year. Lighthill concluded that no part of the field had yet delivered the major impact it had promised.¹⁵ He singled out one general cause: the field had not reckoned with the combinatorial explosion. A method that worked on a small problem failed on a large one, because the number of possibilities to check grew much faster than the problem did.

The expert systems had their turn in the 1980s. By one standard textbook’s account, the AI industry grew from a few million dollars in 1980 to billions of dollars in 1988.¹⁶ A review written in 1988 found that the business “has been through a terrible period in 1987–88”. The once-leading American start-ups had run into losses and cut staff. Symbolics, the biggest single company in the field on the strength of its expensive Lisp workstations, had suffered “an equally dramatic reversal of fortunes.”¹⁷ The years that followed came to be called the AI winter.¹⁶

The people inside the field had warned of it. In 1984, at AAAI-84, the main American conference in the field, a panel met to discuss “The Dark Ages of AI”. It started from a “deep unease among AI researchers” that expectations of the field were too high, and that this would “eventually result in disaster.”¹⁸

Some had been trying another road all along. In 1958 the psychologist Frank Rosenblatt published the perceptron, a network of simple units whose connections were adjusted by experience rather than written by hand.¹⁹ The work was sponsored by the US Navy’s Office of Naval Research, and in 1960 the Navy released a photograph of the machine built on the idea, the Mark I Perceptron. The caption explains how it learned. Test patterns, such as printed letters, were placed in front of its photocell “eye”. “When the machine incorrectly identifies a pattern or letter, the trainer forces it to respond correctly by means of an electrical control.”²⁰

The adjustment was the machine’s. The letters, the right answers and the judgement of when it was wrong all came from the person standing beside it. It was close to what Turing had proposed in 1950: a child machine, taught like a pupil, partly by punishments and rewards.¹¹

Rules and learning ran side by side for decades, and each fell in and out of favour. In 1969 Minsky and Seymour Papert published Perceptrons, which proved that simple networks of this kind could represent very little. A two-input perceptron could not be trained to tell whether its two inputs were different. The proofs did not cover networks with more layers. Soon afterwards, money for research on neural networks dwindled almost to nothing.^{21, 16} In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams showed how a network with extra layers between input and output could be trained by passing its errors backwards through it. The units in those middle layers, they wrote, “come to represent important features of the task domain”, features that nobody had described to the machine.²²

Networks with many layers took off only in 2011, first in speech recognition and then in recognising objects in images.¹⁶ In 2012 a network built by Alex Krizhevsky, Ilya Sutskever and Hinton was entered in an annual competition to sort photographs into a thousand categories. Its top-five error rate was 15.3 per cent. The next-best entry’s was 26.2 per cent.²³ The systems it beat were built largely on features designed by hand, and similar gains were reported in speech recognition and machine translation.¹⁶

In 2017 eight researchers, most of them at Google, published a design they called the transformer. It was built to translate between languages. Most earlier translation networks read a sentence one word after another, and the transformer dropped that step, “dispensing with recurrence and convolutions entirely”, and relied instead on a mechanism called attention.²⁴ The paper also wrote down what attention costs. It weighs every part of a passage against every other part, so a passage twice as long needs about four times as many comparisons.²⁴ Its authors judged attention faster than reading word by word for sentences of the length used in translation, and suggested that, for very long inputs, each part could be limited to its neighbours.²⁴

The next bet was on size. A language model works on text broken into small units called tokens, and in training it learns to predict the next one. In January 2020 Jared Kaplan, Sam McCandlish and eight colleagues at OpenAI reported that such a model’s prediction loss fell in a smooth, predictable way as three things grew: the model, its training text and the computing spent on it. As the computing budget increased, they wrote, it “should be spent primarily on larger models”.²⁵ In May 2020 Tom Brown and thirty colleagues at OpenAI described GPT-3, a model trained to predict the next token, with 175 billion parameters (the values adjusted in training). That was ten times as many as any earlier language model built the same way. Three-fifths of its training mix came from a filtered copy of Common Crawl, an archive of pages from the web.²⁶ In March 2023 OpenAI described GPT-4 the same way, as a model “pre-trained to predict the next token in a document”.²⁷

Each step was a bet: judge the machine by its conduct, describe intelligence precisely, write the expert’s knowledge into rules, let the machine adjust itself from examples, translate with attention, train ever larger models to predict the next token over ever more text.

Back to the ivory

The Lion Man is still in Ulm, and no account of it survives.

Neither figure carries its maker’s name. The makers of artificial intelligence left theirs, on dated decisions about what intelligence is. Turing’s paper is in Mind. The Dartmouth proposal is a dated typescript with four names on it. Lighthill’s survey and the 1988 trade review each carry a date and their authors’ names, and the Navy’s photograph of the Mark I carries the date of its release.

For the systems in use today the record is thinner. As with the Mark I, people still supply the answers and the judgement. Most of the text a language model learns from was written by people. In March 2022 Long Ouyang and nineteen colleagues at OpenAI described hiring about forty contractors to write examples of good answers and to rank the model’s outputs. The rankings trained a second model to score answers, and the language model was tuned against it.²⁸ A year later, OpenAI’s report on GPT-4 gave “no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.”²⁷ For GPT-4, the details of which text went in, how the model was built and how it was tuned were not published, and silence about the data is common. In December 2025 a Stanford-led index scored thirteen major developers on what they disclose and found them most secretive about training data.²⁹

In October 2026 Nick Cammarata of Goodfire, an interpretability research lab, published a study of the neurons in Qwen3-32B.³⁰ Its makers had released the model’s weights for anyone to download.³¹ Goodfire’s preliminary audits found that most of the model’s neurons seem to have one main job. Cammarata calls this a working hypothesis.³⁰

Two days later the researcher Leo Gao agreed that neurons had been too quickly written off as a jumble of unrelated jobs. But Gao said this was “nowhere near a sufficient argument that we understand most of what neurons are doing”.³² Researchers outside the company that built the model have started to read what its neurons do, and because the weights are public, anyone with the skill can check their reading.

Sources

[1] Museum Ulm, Archaeology collection, museumulm.de/en/collections/archaeology/. About 40,000 years old; carved from mammoth ivory; described by the museum as the oldest known depiction of a mythical creature (the museum’s framing). The caves of the Swabian Jura became a UNESCO World Heritage Site in 2017. The page (read 24 October 2026) states that the museum “is currently closed due to rebuilding measures” and that the Lion Man is “on display again at the Ulm Museum starting January 16, 2027”.

[2] Jarrett A. Lobell, “New Life for the Lion Man,” Archaeology, March/April 2012, archaeology.org/issues/march-april-2012/features/new-life-for-the-lion-man/. Fragments found on 25 August 1939, a week before the war began and before the fieldwork could be completed; three decades in storage until Joachim Hahn began an inventory and pieced together more than 200 fragments; fragments added in 1982 showed a cat’s head; “neither the gender of the animal nor of its human parts is discernible.” Archaeology gives the later excavations as 2008–2011; this essay follows the Museum Ulm folder [3] for those dates.

[3] Museum Ulm, Der Löwenmensch im Museum Ulm / The Lion Man at the Museum Ulm, bilingual visitor folder, undated (it refers to the 2017 UNESCO listing as past). Fragments recovered in August 1939, on the last day of the excavations at the Stadel cave; carving noticed in December 1969 and the splinters fitted into a figure of about 30 cm; first professional restoration in 1988; known to the public as the “Löwenmensch” (Lion Man) since 1994; further ivory pieces found in the State Office for Cultural Heritage Baden-Württemberg excavations from 2009 to 2013; the figure now assembled from more than 300 pieces; described as “the oldest effigy of a mythical creature”. The folder says the carving “attracted the attention of an archeologist” in December 1969 and does not name him.

[4] Cleveland Museum of Art, Yoga Narashimha, Vishnu in his Man-Lion Avatar, 1973.187, clevelandart.org/art/1973.187. South India, late Chola period, c. 1250; bronze, 57 × 31 × 23 cm. Description: one of the ten main avatars of Vishnu; “for devotees, the man-lion form recalls a myth that describes his superhuman defeat of a seemingly invincible demonic foe”; the form “embodies royal power and yogic discipline”; the posts on either side held a halo.

[5] T. A. Gopinatha Rao, Elements of Hindu Iconography, vol. 1, part 1 (Madras: Law Printing House, 1914), pp. 149–150, read from the Internet Archive scan. Summarising the image-making texts, Rao gives the single seated image of Narasimha (Girija- or Kevala-Narasimha) as “seated on a padmasana in the utkutika posture, the forelegs being maintained in the required position by the yoga-patta belt going round them and the back of the body”, with two or four arms; if four, the upper right holding the chakra; “This form of the image of Narasimha is often designated as the Yoga-Narasimha.” He gives the story from the Padma Purana, “substantially in agreement with what is found in the Vishnu Purana”. That the form is still worshipped: see [6].

[6] Arulmigu Lakshmi Narasimmaswamy Temple, Sholinghur, Ranipet district, Tamil Nadu (temple code TM001502), official site maintained by the Hindu Religious and Charitable Endowments Department, Government of Tamil Nadu, sholinghurnarasimmar.hrce.tn.gov.in, read 25 October 2026. “Narasimha sits in a Yogic posture in a shrine facing East and is called Yoga Narasimha Swamy”; the temple is “praised by the Thirumangai Alwar, Pei Alwar, Nam Alwars in their pasuram”; “The Friday Thirumanjanam of the Periyamalai weekly is at 7.00 am”.

[7] Ovid, Metamorphoses X.243–297, translated by A. S. Kline, poetryintranslation.com. Pygmalion carves “snow-white ivory”; the festival of Venus; the ivory softens at his touch. Ovid does not name the statue. Date: Kline’s Ovid page (poetryintranslation.com/PITBR/Latin/Ovhome.php) gives the banishment as 8 AD, to Tomis on the Black Sea coast; in Tristia I.VII (tr. A. S. Kline, 2003) Ovid calls the Metamorphoses “the work cut short by [its] author’s sad flight”, “rough and still unfinished”.

[8] Cited at second hand. Meyer Reinhold, “The Naming of Pygmalion’s Animated Statue,” The Classical Journal 66, no. 4 (1971): 316–319, as reported in Wikipedia, “Galatea (mythological statue)” (read 24 October 2026): no surviving ancient text names the statue, and the name “Galatea” was first given wide circulation by Rousseau’s Pygmalion. The article is behind a paywall (JSTOR) and was not read; no open copy was found. Rousseau probably wrote the text in 1762; it was first performed in Lyon in 1770 (Wikipedia, “Pygmalion (Rousseau)”, read 25 October 2026).

[9] The Metropolitan Museum of Art, Auguste Rodin, Pygmalion and Galatea, 10.31, metmuseum.org/art/collection/search/191292. Marble; modelled 1889, carved c. 1908–9; 97.2 × 88.9 × 76.2 cm. Open Access, CC0.

[10] Cleveland Museum of Art, Peacock-shaped Hand Washing Device: Illustration from The Book of Knowledge of Ingenious Mechanical Devices (Automata) of Ibn al-Razza al-Jazari, 1945.383.a, clevelandart.org/art/1945.383.a. A leaf “from a 1315 Syrian copy” of the book, “written in 1206”; Damascus, Mamluk period; 15 surviving manuscript copies; al-Jazari, “an engineer from upper Mesopotamia”, was “in the service of King Nasrī al-Dīn”; the book’s devices include “clocks, trick vessels for drinking sessions, devices for washing, fountains, water-raising machines, and measuring instruments”; the automata tradition was “known in the ancient, Islamic, and Byzantine worlds”; the peacock’s mouth “acts as a spigot” and the figure “emerged from a small door and offers soap”; “We do not know with certainty that al-Jazarī’s device was ever actually constructed.” The museum does not name the ruler’s dynasty or city, and neither is given here.

[11] A. M. Turing, “Computing Machinery and Intelligence,” Mind 59, no. 236 (October 1950): 433–460. §1: “I propose to consider the question, ‘Can machines think?’” and “Instead of attempting such a definition I shall replace the question by another.” §7: “Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child’s?”

[12] J. McCarthy, M. L. Minsky, N. Rochester and C. E. Shannon, “A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence,” typescript dated 31 August 1955; reprinted in AI Magazine 27, no. 4 (2006), doi:10.1609/aimag.v27i4.1904. Read from the typescript scan. “August 31, 1955”; “a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College”.

[13] Allen Newell and Herbert A. Simon, “Computer Science as Empirical Inquiry: Symbols and Search,” Communications of the ACM 19, no. 3 (1976): 113–126. The Logic Theorist dated by the authors to 1955; the physical symbol system hypothesis as quoted.

[14] Edward A. Feigenbaum, “The Art of Artificial Intelligence: Themes and Case Studies of Knowledge Engineering,” Proceedings of IJCAI-77 (1977): 1014–1029. DENDRAL “begun in 1965”, a project with the Stanford Mass Spectrometry Laboratory, working from instrument data that included “data from a mass spectrometer”; the knowledge engineer “works intensively with an expert to acquire domain-specific knowledge and organize it for use by a program”, with the expert’s rules represented “in the situation => action form”.

[15] James Lighthill, “Artificial Intelligence: A General Survey,” in Artificial Intelligence: A Paper Symposium (Science Research Council, 1973); written July 1972. The conclusion is paraphrased from “In no part of the field have the discoveries made so far produced the major impact that was then promised.” Lighthill singles out “one rather general cause for the disappointments that have been experienced: failure to recognise the implications of the combinatorial explosion,” the explosive growth in possible combinations as a knowledge base grows.

[16] Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 4th ed. (Pearson, 2020), chapter 1, read from the publisher’s sample chapter (aima.cs.berkeley.edu/newchap01.pdf). Secondary. §1.3.3, p. 22: Perceptrons “proved that, although perceptrons (a simple form of neural network) could be shown to learn anything they were capable of representing, they could represent very little”; “a two-input perceptron could not be trained to recognize when its two inputs were different”; “their results did not apply to more complex, multilayer networks”, yet “research funding for neural-net research soon dwindled to almost nothing”; back-propagation brought “an enormous resurgence in neural-net research in the late 1980s”. §1.3.4, p. 24: the industry “boomed from a few million dollars in 1980 to billions of dollars in 1988”; “soon after that came a period called the ‘AI winter.’” §1.3.8, p. 26: “It was not until 2011, however, that deep learning methods really took off. This occurred first in speech recognition and then in visual object recognition”; the 2012 ImageNet system “demonstrated a dramatic improvement over previous systems, which were based largely on handcrafted features”; “Similar gains have been reported in speech recognition, machine translation, medical diagnosis, and game playing.”

[17] T. Johnson, J. Hewett, C. Guilfoyle and J. Jeffcoate, “Expert systems: the second wave,” The Knowledge Engineering Review 3, no. 3 (1988): 177–181, doi:10.1017/S0269888900004446. Abstract: “The expert systems business has been through a terrible period in 1987–88”; in the USA, the leading start-ups “have all run into losses and cut back staff”; Symbolics, “whose high priced Lisp workstations have made it the biggest single company in artificial intelligence,” “has hit an equally dramatic reversal of fortunes.”

[18] Drew McDermott, M. Mitchell Waldrop, B. Chandrasekaran, John McDermott and Roger Schank, “The Dark Ages of AI: A Panel Discussion at AAAI-84,” AI Magazine 6, no. 3 (1985): 122–134. Abstract: the panel discussed the “deep unease among AI researchers who have been around more than the last four years or so … that perhaps expectations about AI are too high, and that this will eventually result in disaster.” A forecast made in 1984, not evidence of what followed.

[19] F. Rosenblatt, “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain,” Psychological Review 65, no. 6 (1958): 386–408. “The theory has been developed for a hypothetical nervous system, or machine, called a perceptron”; for such a system to learn, “it must be possible to modify the A-units or their connections”. A footnote records the work’s sponsorship by the Office of Naval Research.

[20] US Navy photograph 330-PSA-80-60 (USN 710739), “Experimental Machine Able To Identify Letters of Alphabet Announced By Navy,” released 24 June 1960; the caption gives the work as “sponsored by the Office of Naval Research with the assistance from Rome Air Development Center”; National Museum of the U.S. Navy, via Wikimedia Commons, public domain. Caption quoted as above.

[21] Marvin Minsky and Seymour Papert, Perceptrons: An Introduction to Computational Geometry (MIT Press, 1969). Cited as the book; its results are described here as in [16].

[22] David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, “Learning representations by back-propagating errors,” Nature 323 (9 October 1986): 533–536. Hidden units “come to represent important features of the task domain.”

[23] Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25 (2012). ILSVRC-2012: winning top-5 test error of 15.3%, against 26.2% for the second-best entry.

[24] Ashish Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems 30 (2017), arXiv:1706.03762. Built for machine translation, “dispensing with recurrence and convolutions entirely”. Table 1 gives the per-layer complexity of self-attention as O(n²·d), where n is the sequence length and d the representation dimension. Section 4: self-attention layers are faster than recurrent layers “when the sequence length n is smaller than the representation dimensionality d, which is most often the case with sentence representations” in machine translation; “for tasks involving very long sequences, self-attention could be restricted to considering only a neighborhood of size r”.

[25] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei, “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 23 January 2020 (OpenAI; Kaplan also Johns Hopkins). Abstract: “The loss scales as a power-law with model size, dataset size, and the amount of compute used for training.” Section 1: “As the computational budget C increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size.”

[26] Tom B. Brown and 30 co-authors (OpenAI), “Language Models are Few-Shot Learners,” arXiv:2005.14165, first posted 28 May 2020; Advances in Neural Information Processing Systems 33 (2020). Abstract: GPT-3 is “an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model.” Table 2.2: filtered Common Crawl, 410 billion tokens, 60% of the training mix.

[27] OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 15 March 2023. Section 2: GPT-4 is “a Transformer-style model pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers”, then fine-tuned with reinforcement learning from human feedback; “Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.” OpenAI says it plans to make further details “available to additional third parties”.

[28] Long Ouyang and 19 co-authors (OpenAI), “Training language models to follow instructions with human feedback,” arXiv:2203.02155, 4 March 2022. Section 1: “We first hire a team of 40 contractors to label our data”; “a dataset of human-written demonstrations of the desired output behavior”; “a dataset of human-labeled comparisons between outputs from our models”, used to train a reward model “to predict which model output our labelers would prefer”. Section 3.4: “a team of about 40 contractors on Upwork and through ScaleAI.”

[29] Alexander Wan, Kevin Klyman, Sayash Kapoor, Nestor Maslej, Shayne Longpre, Betty Xiong, Percy Liang and Rishi Bommasani, “The 2025 Foundation Model Transparency Index,” arXiv:2512.10169, 11 December 2025 (Stanford, with Berkeley, Princeton and MIT). Thirteen companies scored, OpenAI among them; “the average score out of 100 fell from 58 in 2024 to 40 in 2025. Companies are most opaque about their training data and training compute …”; §5.1: “Data properties is the lowest scoring subdomain, with companies scoring just 15% on average. Eight companies score none of the 5 indicators in this subdomain, failing to disclose the data size, language composition, or domain composition, or to provide external access to the data or instructions for replicating the data.”

[30] Nick Cammarata (Goodfire), with contributions from Curt Tigges and Tarun Gogineni, “Neural Networks Are Already Clean,” already-clean.goodfire.com, 15 October 2026, read 25 October 2026. Described on the page as “An informal research essay and working hypothesis about neural networks.” Summary: “We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.” Section 1: “In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood”; on his earlier work at OpenAI (Bills, Cammarata, Mossing et al., 2023): “about 1.7% of neurons interpretable by our most generous measure.” followed by “With better tools and modern models, the situation looks quite different.” A neuron’s poles are its positive and negative sides; the page notes the share varies with the threshold set for weak activations. On its gallery of 1,000 randomly chosen neuron studies (Qwen3-32B and Qwen3.6-35B-A3B): “What we see is about 90% of neurons having basically one job.” The 1.7% figure is from an older model and a different method. Goodfire, Company page, goodfire.com/company, read 25 October 2026: “Goodfire is an AI interpretability research lab”; “We are headquartered in San Francisco”.

[31] Qwen team (Alibaba), Qwen3-32B model card, huggingface.co/Qwen/Qwen3-32B, read 25 October 2026. “License: apache-2.0”; the weights are published for download. Qwen3 Technical Report published 14 May 2025. Qwen organisation card, huggingface.co/Qwen, read 25 October 2026: “the large language model family built by Alibaba Cloud”.

[32] Leo Gao, “Neural networks are not already clean,” LessWrong, 17 October 2026, lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean, read 25 October 2026. Agrees that “people have been too quick to write off neurons as polysemantic” and that “we should update on it. But it’s nowhere near a sufficient argument that we understand most of what neurons are doing, and these results do not show that interpretability is solved, or even that it is vastly easier than previously anticipated.” Closing section: “I’m hopeful that it’s possible to disentangle everything, figure out how computation-in-superposition works, and ultimately find clean, understandable circuits in networks.”

Image credits. Lion Man: Dagmar Hollmann, Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Loewenmensch1.jpg), licensed CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), resized and unaltered otherwise. Narasimha and the al-Jazari leaf: Cleveland Museum of Art, CC0. Rodin: The Metropolitan Museum of Art, CC0. Mark I Perceptron: National Museum of the U.S. Navy, public domain.

The artworks in this essay were found through Open Museum (except the photograph of the Lion Man).