How a language model reads

Language is statistics

Behind language lies a complete set of statistics. Every word in a given context carries a statistic with it. In essence, that is the whole idea behind language models.

From text to tokens

But there are more regularities to discover in language. Characters carry a statistic too: in most Western languages “e” is the most used character, unlike “x” or “q”.

How often letters occur in Dutch

Some patterns of characters also occur more often, such as “ing”. A word like “wandering” can be split into two parts: wander + ing. Both parts carry a certain meaning. Dividing language into tokens means a text contains fewer tokens than characters, which in turn makes the model more efficient.

One pattern, many words

A vocabulary like an alphabet

The complete set of tokens is a model’s vocabulary, and it works like an alphabet: you use a limited number of letters to form an endless number of words. In the same way a model can build many different words from a relatively small set of tokens.

Another advantage of tokens is that the model can recognise unknown words. Often these are words from everyday speech, like hangry, or jokes like chillax.

Unknown words, known pieces

The process of splitting the original text into these tokens is called tokenization.

100 tokens is about 75 words

As a rule of thumb, a token is about three quarters of the length of a word. That means 100 tokens roughly correspond to 75 words.

Most models are optimised for English; for Dutch the ratio is less favourable. Using Dutch therefore costs more tokens, and so more money.

Same meaning, two languages

Now you know what a token is. The next question is what it costs: input and output, caching, and the tokens you don’t see.