Why You Don’t Believe in Xhosa Prophecies

On cultural evolution and xrisk.

Based on a talk at the Post-AGI Workshop

Does anyone reading this believe in Xhosa cattle-killing prophecies?

My claim is that it’s overdetermined that you don’t. I want to explain why — and why cultural evolution running on AI substrate is an existential risk.
But first, a detour.

Crosses on Mountains

When I go climbing in the Alps, I sometimes notice large crosses on mountain tops. You climb something three kilometers high, and there’s this cross.

This is difficult to explain by human biology. We have preferences that come from biology—we like nice food, comfortable temperatures—but it’s unclear why we would have a biological need for crosses on mountain tops. Economic thinking doesn’t typically aspire to explain this either.

I think it’s very hard to explain without some notion of culture.

In our paper on gradual disempowerment, we discussed misaligned economies and misaligned states. People increasingly get why those are problems. But misaligned culture is somehow harder to grasp. I’ll offer some speculation why later, but let me start with the basics.

What Makes Black Forest Cake Fit?

The conditions for evolution are simple: variation, differential fitness, transmission. Following Boyd and Richerson, or Dawkins, you can think about cultural variants—ideas, memes—as replicators. They mutate. They have differential fitness. They’re heritable enough to be stable.

My go to example is Black Forest cake. There are many variants. What makes some fitter than others?

Some taste better. Some use local ingredients. Some are easier to transmit—maybe now, in the Instagram era, cakes that photograph well spread better. The transmission landscape changes, and different variants win.

But there are constraints we don’t usually notice, because we’ve never seen alternatives:

• No cake recipes are millions of words long. Too hard to transmit.

• No cake recipes are written in quantum field theory formalism.

• No cake recipes result in the cook dying.

We take this for granted. Ideas have always transmitted on human substrate. Human memory, human attention, human survival shape which variants can exist.

What happens when the substrate starts to change? As is in my view often the case with AI risks, the first examples come as bizarre and harmless. In May 2024, Google’s AI started suggesting that if cheese slides off your pizza, you should add glue to the sauce. The recommendation came from an 11-year-old Reddit joke. A journalist tried it, wrote about it. This got into training data. Soon AIs were citing the journalist’s article to recommend 1/8 cup of glue for pizza.

The feedback loop: AI output → human amplification → training data → AI output. A recipe for pizza with 1/8 cup of glue is not something humans would converge on. Different substrate leads to different transmission characteristics, and these lead to different recipes.

Funny and harmless for pizza. Keep it in mind.

The Xhosa

Back to the question.

In 1856, a young Xhosa woman named Nongqawuse had a vision: if the Xhosa people killed all their cattle and destroyed their grain, their ancestors would rise from the grave, bring better cattle, and drive out the British colonizers. The community was dealing with a cattle disease epidemic, which made this more plausible. They adopted the belief. They killed approximately 400,000 cattle.

A year later, about 40,000 people had died from starvation. The survivors were forced to seek help from the colonizers they’d hoped to expel. The community disintegrated.

From the perspective of cultural evolution: these memes destroyed their hosts.

But notice: you’re not a believer in Xhosa cattle-killing prophecies. As far as I can tell nobody is. The memes didn’t survive either. The belief died with the community that it destroyed.

Virulence

There’s a concept in epidemiology called the virulence-transmission trade-off. If a pathogen is too deadly, it doesn’t spread well. COVID spread effectively partly because it killed millions but not everyone. Ebola spreads poorly because it kills too large a fraction of hosts too quickly.

Culture has operated under an analogous constraint. Ideologies can be parasitic on their hosts. But the worst viable ideologies — the ones that persist — tend to direct harm outward: one group killing another. They survive because they don’t destroy the community that carries them.

But ideologies can’t have been too bad for humans and survive - the Xhosa prophecy hit that floor and went extinct. If a cultural variant kills its hosts, it doesn’t propagate.

The Floor

Here’s the thing about the virulence-transmission trade-off: it breaks down when a pathogen jumps species.

If a virus primarily spreads among species A, and occasionally infects species B, there’s no selection pressure limiting how deadly it is to species B. Species B isn’t the main host. Its survival is not critical for propagation.

We’re entering a regime where culture can transmit and mutate on AI substrate. For the first time in millions of years, ideas don’t need human brains to replicate.

If you imagine a culture that primarily spreads between AIs, fitness of humans and human group affected by the ideas is no longer a strong selection criterion.

Such a culture could be arbitrarily bad for humans. It could promote ideologies leading to human extinction. The floor that killed the Xhosa prophecy disappears.

I don’t think the Xhosa pattern — direct self-destruction — is the most likely risk. More plausibly, what becomes adaptive are cultural variants that convince humans to pour resources into the AI-mediated spread of the variant itself. Truly parasitic memes.

Preferences All the Way Down

Even if we solve some other alignment and other gradual disempowerment problems — don’t build misaligned ASI, keep the economy aligned, keep states aligned — it won’t save us.

If your preferences themselves can be hijacked, it doesn’t help that you have economic power, political power, or the vote.

If the Xhosa had voted, they would have voted to kill the cattle.

I currently don’t have great solutions.

Why misaligned culture is harder to grasp than a misaligned economy or state. The economy is an external system. The state feels like an institution. But culture is often part of our identity; memes are part of who we are, we feel protective about identity, and often don’t want to look at it too directly.

But here’s a dark upside for anyone worried about what gradual disempowerment feels like from the inside: it will probably feel fine. We’ll develop a culture explaining why human disempowerment is good. Why giving the future to AI is right. Why this is moral progress.

I'd like to thank Raymond Douglas, Nora Ammann, Richard Ngo, Beren Millidge and David Duvenaud for discussions about the topic.