AI & the web
Token
A unit into which a language model divides text for processing, such as a word, part of a word or punctuation.
In everyday use
A long word may occupy several tokens.
What to keep in mind
Token counts depend on the model's tokenizer, not just word count.
Also searched as: LLM token
Further reading
Original explanation and example by Perspectism. Further reading may cover the wider subject, rather than this exact wording. Edition updated 7 October 2026.
More in ai & the web
Page updated .