The Token Count a Prompt Cleaner Shows Is Fake — the 0.25 Approximation Trap
When a prompt-cleaning tool shows the number dropping before and after cleaning — "~tokens: 1,234 → 987" — it's easy to believe your API cost or context usage actually fell by that much. But this number isn't the output of a real tokenizer — it's an approximation, character count multiplied by a fixed ratio. This guide explains exactly what that approximation is and why it's especially unreliable for Korean.
1. The approximation baked into the actual source code
Open modoohub.com's prompt cleaner (prompt-cleaner.html) and the function that computes token count looks like this:
function estTokens(s){return Math.ceil(s.length*0.25);}It takes the input string's length × 0.25, rounds up, and shows that directly as "~tokens." It never calls a real tokenizer library.
So it's a simple linear ratio of one token per four characters. The tool's own FAQ also states that "it does not run a real tokenizer and uses only the approximation character count × 0.25," so this is an intended simplification, not a hidden bug.
2. Where does "one token per four characters" come from?
OpenAI's tiktoken and Claude's tokenizer are BPE (Byte-Pair Encoding)-family algorithms that use a vocabulary trained on large amounts of text to merge frequently occurring byte/character combinations into single tokens. Tokenizing English prose this way yields, on average, about one token per four characters — a widely known empirical figure. This tool's 0.25 coefficient is that English-based rule of thumb taken directly.
3. Why it goes so wrong for Korean in particular
A BPE vocabulary reflects the language distribution of its training data. Most general-purpose tokenizers have an overwhelmingly high English share in their training corpus, so common English word/prefix/suffix combinations are often bundled whole into a single token. Korean, by contrast, is relatively under-optimized in the vocabulary, so individual syllables or the byte sequences that make up syllables often get split into multiple tokens. As a result, for the same character count, the actual token count of Korean text often comes out far higher than the "character count × 0.25" formula predicts — this approximation systematically under-estimates real token consumption for Korean.
4. What happens in practice
Clean a prompt with this tool and then judge, from the on-screen "tokens saved" alone, that "there's now room in the context window" or "API cost dropped by this much," and — especially for a Korean prompt — you can be off from the real value. The savings percentage is also computed with the same approximation for the before and after values, so it inherits the same error. The cleaning features themselves (whitespace tidying, blank-line handling, tab conversion, etc.) genuinely reduce character count and are valid, but using the token number on screen as the basis for actual billing/context calculations is risky.
5. If you need an accurate token count
- OpenAI-family models: run the real encoding with the tiktoken library (or an online tool that wraps it) to check the token count.
- Claude-family models: use the token count feature the Anthropic API/SDK provides.
- If comparison is the goal: actually tokenizing the before text and the after text with the methods above and directly comparing the difference is the only reliable approach.
This site's AI token counter is also worth a look, but with any tool the important habit is to first check whether it uses a character-count approximation in the browser or calls a real tokenizer.
Frequently Asked Questions
Q. Should I not trust the prompt cleaner's token figure at all?
A. It's not entirely meaningless. The directional signal that "character count went down" between before and after cleaning is valid. But don't trust the absolute value or the ratio for Korean — when you need an exact number, run a real tokenizer separately.
Q. Why doesn't the tool use a real tokenizer from the start?
A. Embedding a library like tiktoken in the browser adds size, and tokenizers differ by model (OpenAI/Claude/Gemini), so you'd have to choose which model to count by. Lightweight tools often pick an approximation over that complexity.
Q. Can I trust this approximation for an English prompt?
A. English is the reference language the 0.25 coefficient derives from, so the error is relatively small. But text with lots of code blocks, special characters, or repetitive patterns can still deviate from the simple ratio even in English.
Q. Is the savings percentage useless for judging cleaning effectiveness?
A. The savings rate is computed with the same 0.25 approximation, so its absolute accuracy is low, but it's usable for relative comparison between options ("turning this option on raised the savings rate"). To confirm actual API cost reduction, put the before and after text through a real tokenizer.