Token — The basic unit of text that AI models process. A token is roughly 4 characters or 0.75 words in English.
Tokens are the fundamental units that LLMs use to process text. Rather than processing characters or words directly, models split text into tokens—which can be whole words, parts of words, or punctuation marks. The number of tokens affects both the cost of using an AI model and what fits within its context window. For AI coding tools, tokens are used to measure how much code the model can read and generate in a single session.
A token is the basic unit an AI model uses to process text. Roughly speaking, 1 token ≈ 4 characters ≈ 0.75 words in English. AI models have limits on how many tokens they can process at once.
Tokens determine how much code an AI can read and generate. More tokens = larger context window = the AI can understand more of your codebase. Token limits also affect pricing for API-based tools.
Context Window
The maximum amount of text (measured in tokens) that an AI model can process at one time, including both input and output.
Large Language Model (LLM)
A type of AI model trained on massive amounts of text data, capable of understanding and generating human-like text across a wide range of tasks.
Fine-tuning
The process of further training a pre-trained AI model on a specific dataset to improve its performance on a particular task or domain.
EXPLORE MORE
Explore our complete glossary of AI, LLM, and system prompt terminology.
View Full Glossary