What Are AI Tokens? Why Words Do Not Predict Usage
The tokens used by a sentence depend on how the model segments it. Word and character counts alone cannot give an exact token count, and the same text may produce different counts with different models.

Tokens are the units that the model processes.
A token is a small unit representing text for a language model. It can be a whole word, part of a word, punctuation, or a unit containing spaces. Token boundaries need not match the words we read.
The component that divides text is called Tokenizer and the splitting process is called Tokenization After splitting, each unit is converted into a numeric code or token ID using the tokenizer's vocabulary to prepare model input. The documentation Hugging Face on Encoding distinguishes these two steps.
A token ID identifies a unit; it is not a unit count. A large numeric ID does not mean that many tokens were used. Count the actual units and check which special components the tool includes.
The word Token in this article refers to AI processing units. Token, which is used to refer to digital assets, has another meaning. When purchasing an AI package or using a service quota, you must look directly at the conditions of that service.
Why can one word have multiple tokens?
Tokenizers have different rules and vocabularies. The method called Subword Tokenization Subword segmentation combines existing pieces into many word forms without storing every word as one unit
See the example from Hugging Face on Subword Tokenization below. The word tokenization is split into token and ization so one word uses more than one token in this example.
Do not infer from this image that every tokenizer splits this sentence identically or that long words always use more tokens than short ones. Results depend on segmentation and vocabulary. Hugging Face algorithm documentation Explains approaches including BPE, Unigram, and WordPiece, which do not all follow identical rules
Can Thai token usage be inferred from words or characters?
The Thai language does not have spaces between every word. Dividing the text by space does not yet tell us the number of Thai words. And even more, it cannot be used to replace the number of tokens of the model. If you want to know the message วันนี้อยากเรียนรู้เรื่อง AI Check token counts using tools supporting that model
Word counting, character counting, and token counting answer different questions. The word count tool is suitable for checking the length of writing, while the Tokenizer is used to check how text is divided according to the rules of the model. Therefore, one number cannot be used directly as a substitute for another number.
English-based estimates, such as average characters per token, may help gauge some tasks but are not fixed rates for all Thai text. Do not use them to calculate actual costs or conclude that Thai always uses more tokens than English without checking the model and the text being compared.
Incoming, Outgoing Tokens and Context Window
AI conversations involve both information sent to the model and information it generates. The term Input Tokens means the unit of the input data, while Output Tokens means output data units. Reporting boundaries for additional components, such as internal reasoning or tools, depend on the service and model.
Incoming data can include system commands, documents, and chat history sent by the system. For example: Gemini documentation Count text, conversation history, system instructions, and tools separately. A short follow-up after a long conversation does not necessarily contain only the latest message.
Context Window is the amount of information a model can process together, usually stated in tokens. Input, output, and other budget rules vary by model. For long documents, check the model's limits in Context Window manual and long document A larger text capacity does not guarantee correct answers about every part
For API services, token counts may contribute to charges, with different input and output rates and additional conditions. Consumer chat-app subscriptions depend on their own quotas and terms; do not automatically substitute API pricing formulas.
See how spaces and punctuation affect token counts
The following exercise is KrobClick's proposed observation method, not experimental results or measured token counts. Use nonsensitive text, choose a counter allowing model selection, and record the model or tokenizer version first.
- Count the initial text
วันนี้อยากเรียนรู้เรื่อง AIthen record the displayed count, and segmentation output if available - Change only the punctuation to
วันนี้อยากเรียนรู้เรื่อง AI!then recount with the same model - Return to the initial prompt and change only the final word to
วันนี้อยากเรียนรู้เรื่องปัญญาประดิษฐ์Record results separately from the previous round. - For cross-model comparisons, keep the input text identical character for character and label each result with its model
Observe whether segmentation boundaries change and how much the count differs. Do not assume adding one punctuation mark must add one token: segmentation rules may also change surrounding units.
Check whether the tool counts plain text alone or includes chat formatting and special tokens. Different counting scopes can give different results for the same visible text. If the service does not disclose the scope, state that limitation. A number written by AI is not evidence from the token-counting system.
For development, counting before submission helps check input size, while the data usage is API-reported usage after processing, offering checks on actual consumption. Advance input counts do not include the unknown response tokens. For service-specific steps, read Gemini's guide to counting tokens before sending and after responding
Write concisely while preserving key information
When choosing what to send to AI, remove repeated instructions, select the relevant parts of documents, and specify the answer format. Understanding tokens helps organize that input, but conditions and exceptions that affect the answer still need to stay.
For a hypothetical refund-policy review, you might use this prompt:
Using the attached policy, explain who can request refunds, within how many days, and with which exceptions. Identify supporting sections. If unspecified, say no information was found.
Provide both the main conditions and relevant exceptions, then check the answer against the original. Removing exceptions to save tokens may leave insufficient information even with a shorter request. Before sending real data, check service permissions and policies, especially for personal documents or customer data.
Sources used for verification
- Hugging Face LLM Course: Tokenizers
- Hugging Face Transformers: Tokenization algorithms
- Google AI for Developers: Understand and count tokens
Sources reviewed October 23, 2026, Thailand time. Segmentation images are Hugging Face examples. Thai exercises and prompts are KrobClick proposals, not model tests or actual API calls. Tokenization, limits, and billing depend on the model and service.