What are tokens?
Tokens are the small units of text that AI language models work with. A token can be a whole word, part of a word, a number or a punctuation mark. Models read your prompt as tokens and write their answer one token at a time.
A common rule of thumb is that one English token is about four characters, or roughly three quarters of a word. Common words are often a single token, while rare words, long compounds and unusual brand names are split into several pieces. Dutch and many other languages often need more tokens than English for the same amount of text. Each model uses its own tokenizer, so the same sentence can produce a different count in ChatGPT, Claude or Gemini.
Tokens are the unit behind almost every limit in AI: the size of the context window, the maximum length of an answer and the price of using a model through an API.
Why tokens matter for AI visibility
Token limits decide how much of your content a model can take in. When an assistant searches the web, it cannot pass every page it finds to the model in full. Retrieval systems often split pages into chunks of a set number of tokens and pass on only the most relevant ones. A key fact buried under a long introduction may never make it into the answer.
Tokens also explain why AI answers vary. A large language model picks each next token from a range of likely options, so the same prompt can produce different wording, different brand lists and a different order from one day to the next. That is why a single test tells you little and AI answer volatility is worth tracking.
How to write with tokens in mind
Lead with the answer. Put the key fact in the first sentence of a section, so it survives chunking and truncation.
Keep passages focused. One idea per paragraph makes each chunk self-contained and easier to match to a question.
Cut filler. Boilerplate, repeated navigation text and long intros take up space without adding meaning.
Spell your brand name one way. Variations turn into different token sequences, so stick to one version everywhere.
Measure answers over time. Because output varies, judge visibility on repeated runs, not on one screenshot.
Trackbase runs the same prompts every day across 11 AI models, so you see trends instead of one-off answers. See how prompt tracking works.
Examples of tokens in practice
A common English word such as "visibility" may be a single token, while an invented brand name such as "Zwoblix" is split into several pieces.
By the rule of thumb, a 1,500-word English buying guide is roughly 2,000 tokens, and a retrieval system may pass only a few chunks of it to the model.
A B2B payroll software vendor asks ChatGPT the same question on three days and gets three shortlists in a different order, because each answer is generated token by token.
A marketing agency pays for API usage per million tokens when it runs large batches of prompts through a model.
Frequently asked questions
How many words is 1,000 tokens?
By a common rule of thumb, 1,000 tokens is about 750 English words, since one token averages around four characters. The exact number depends on the model's tokenizer and the language. Dutch and other languages with long compound words often need more tokens for the same number of words, so use the provider's token counter when precision matters.
Why do AI models have token limits?
Every token a model reads or writes costs computing power and memory, so providers limit how many tokens fit in one request (the context window) and how long an answer can be. Very long inputs also make it harder for the model to use every detail well. For content, this means concise, well-structured pages have an advantage.
Do tokens affect whether my brand is mentioned?
Indirectly. Tokens set how much retrieved content fits into the model's input, so clear passages with the answer up front are more likely to be used than facts buried deep in a long page. Because answers are generated token by token with some randomness, your brand can also appear in one answer and be missing from the next, which is why repeated tracking matters.