What is ChatGPT actually doing when I ask it something?

Liz has a new kitten, and the kitten needs a name. So she asks an AI chatbot. The answer starts appearing on her screen, word by word, as if someone on the other end were typing it.

“Who's typing that?”

“Nobody,” says Dan.

One word at a time

Here's what's actually happening. The chatbot reads everything so far: Liz's question, plus whatever it has already written. Then it picks a likely next word. It adds that word to the end. Then it reads everything again, now one word longer, and picks the next one. And again. And again.

“For every single word?”

Every one. OpenAI, the company behind ChatGPT, describes exactly this: its models are trained to predict the next word, one at a time. That's why the answer appears in little bursts instead of all at once. You're watching it being written.

It's actually even finer than words. The chatbot works in pieces called tokens. OpenAI says they “can be thought of as pieces of words”, about three quarters of a word each, on average, in English. Common words are usually one piece. Rarer ones get split up. When we checked with OpenAI's own counting tool, the answer Liz got, “How about Captain Whiskers?”, came out as seven pieces: How, about, Captain, Wh, isk, ers and the question mark. Whiskers took three.

How does it know what's likely?

It learned from a mountain of text. Web pages, books, encyclopedia articles. OpenAI's GPT-3, an earlier model, learned from about 300 billion tokens of text. Meta's Llama 3 used over 15 trillion.

To put that in human terms: if you read 250 words a minute, eight hours a day, every single day for 80 years, you'd read about 3.5 billion words in your whole life. GPT-3's reading pile was around 60 lifetimes of that. Today's models read thousands of lifetimes' worth. Way more than anyone could ever read.

From all that text, it learned which words tend to follow which, in which situations.

Liz recognized the idea right away: “Like my phone suggesting the next word!” Exactly that. Google's phone keyboard, Gboard, suggests your next word with a small model trained to do the very same job. A chatbot is the same idea, just much, much bigger, and trained on vastly more text. That's why it sounds so natural: it writes the way the sentences it learned from usually go.

“So it's just fancy autocomplete?”

Yes and no. The job is the same: pick the next word. But to pick the next word well, across almost any subject, it had to learn a lot along the way. Grammar, so its sentences hold together. Facts, because the next word in a sentence about Paris often depends on knowing things about Paris. Even working things out step by step. Researchers found that when these models write out the steps of a problem first, they get harder problems right more often, and some newer models, like OpenAI's o1, are trained to work through steps before answering.

And it all still happens the same way: one piece at a time.

Liz's chatbot settled on “Captain Whiskers”. Perfect.

What the video left out

“Likely”, not “most likely”. The chatbot doesn't always pick the single most likely next word. OpenAI says several next words can fit, and a bit of randomness in the choice is why asking the same question twice can get you two different answers.

It got extra lessons. Predicting text on its own makes a model that continues documents, not one that chats politely. To become a helpful assistant, it gets further training, partly with feedback from people. OpenAI's GPT-4 report calls this post-training.

Searching the web. Some chatbots can look things up online before answering. When they do, they still write the answer the same way, one piece at a time.

Sounding right isn't being right. Writing smoothly and being correct are different things, which is a whole story of its own: why AI sounds so sure when it's wrong.

So it picks likely next words, from patterns in a mountain of text.

What you can do

• Watch the next answer appear and remember what you're seeing: one likely piece at a time, not someone typing.
• Ask the same question twice to see “likely, not most likely” for yourself.
• For anything important, check the answer against a source you trust. Fluent isn't the same as correct.