Rosalie Avalon home

ROSALIE AVALON

Search the journal
,

What the Ouroboros Can Teach Us About Software and AI

An engraved ouroboros bites its tail beside alchemical glassware, its scales turning into copper circuit traces.

In a medieval manuscript in Venice, a serpent curls up beside a drawing of laboratory glassware. Its tail rests between its teeth. Put your finger on its back and follow it round. You can make a second lap, or a tenth, and every lap will take you over the same patch of ink.

The drawing comes from The Making of Gold, a work attributed to Cleopatra the Alchemist. The serpent is an ouroboros, Greek for “tail-eater”. Inside the circle, three words read hen to pan, often translated as “the all is one”. For an alchemist who believed that different metals shared a common beginning, it was a fitting image. Lead might look quite different from gold and still have something of the same substance in it.1

What interests me about the drawing is the spot where the mouth meets the tail. I think of it whenever someone suggests fixing a computer problem by starting fresh. How fresh can the start be if the old tool helps make the new one? A programmer named Ken Thompson once answered that question with a password.

How a secret password survived a rewrite

Imagine you’re the person in charge of a computer. Someone tells you that a stranger can sign in to your account. You try your own password. It works. You try a wrong one. The computer turns you away. What neither test tells you is that one particular wrong password will let the stranger in.

In 1984, Thompson explained how he could arrange just that. You could read every line of the instructions for signing in and they would look perfectly reasonable. The trick happened a moment later, when a tool called a compiler turned those instructions into a program the computer could run. Whenever Thompson’s compiler recognised the sign-in program, it quietly added a second way in. Your password would open your account, and so would his.2

So you decide to replace the faulty tool. You get a clean set of instructions for making a compiler. You read them yourself. Then you hand them to the compiler already on your computer. After all, making programs is its job.

There’s the bite in the tail. Thompson had given his compiler one more instruction. Whenever it was asked to make its replacement, it passed on both tricks. The new compiler knew how to slip the password into the sign-in program. It also knew how to teach the next compiler to do the same. You could start with clean instructions every Monday and, by lunchtime, Thompson could still be signing in as you.2

When the tool makes the sign-in program, it adds a secret password. When it makes a new tool, it gives it both tricks.
The old tool can slip a password into one program and pass the trick to its own replacement.

Thompson called his lecture Reflections on Trusting Trust. I like the repetition in that title. You trusted the instructions because you’d read them. You trusted the finished program because you’d watched it being made. It’s easy to miss the extra thing you took on trust that morning. Who made the tool you handed the instructions to?

What one AI passes on to the next

Years later, researchers studying AI could follow a similar handover on the page. They let one AI write text, used that writing to teach the next version, then repeated the process. To see what each version had picked up along the way, they gave it the same half-finished passage about the masons who built medieval church towers.4

The first version carried on talking about buildings. The fifth gave a list of languages. By the ninth, the answer began with the word “architecture” and went on to describe jackrabbits with different-coloured tails.4

An imagined blue-tailed jackrabbit in a field beside a medieval church tower.
The researchers started with a passage about church towers. Nine rounds later, the AI was writing about jackrabbits.

Thompson had deliberately taught his tool to preserve a secret. Nobody had taught the AI to put rabbits in a churchyard. Yet in both stories, something the old version produced went into the making of the new one. If an AI leaves an unusual detail out of everything it writes, the next version has less chance to learn it. Over many rounds, whole stretches of the original writing can slip out of reach. Researchers call that loss model collapse.4

Where can you stand outside the circle?

For the computer under your care, the way out begins with a tool that came from somewhere else. Researcher David A. Wheeler showed how to start with a compiler you trust, build a fresh one from the written instructions, then use that one to build a second copy. Under conditions where the same steps should give the same result every time, an exact match with the suspect copy tells you something worth knowing. The program on your computer really does match the instructions you read.3

With AI, a second research team tried something simpler. They kept the original material and added each round of AI writing alongside it. In their experiments, later versions could keep learning from things their predecessors had left out, and the researchers avoided the collapse they saw when each new batch replaced the old one.5

There’s a way to step outside this article, too. Open the church-tower study to page four and find “jackrabbits”. There on the same page are the words the researchers gave the AI and the words it gave back. I can tell you that story twice. The pleasure of following it to its source is that you get to put your own finger on the line. Read Example 1 in the original study (PDF).

Sources

  1. The alchemical manuscript. Figure 9.4 in Greek Science of the Hellenistic Era See also Lawrence M. Principe on the inscription.
  2. Ken Thompson. Reflections on Trusting Trust, 1984
  3. David A. Wheeler. Countering Trusting Trust through Diverse Double-Compiling, 2005
  4. Ilia Shumailov and colleagues. AI models collapse when trained on recursively generated data, Nature, 2024
  5. Matthias Gerstgrasser and colleagues. Is Model Collapse Inevitable?, 2024
Previous essay
Next essay