Multilingual UI, But It Can't Analyze Chinese or Japanese — The N-Gram Analyzer's Regex Language Bias
The N-gram Analyzer's interface itself is translated into four languages — Korean, English, Chinese, and Japanese. Switch the language button to 中文 or 日本語 and the menus and instructions display fine in that language. But paste an actual Chinese or Japanese sentence into that same screen and run the analysis, and the results contain not a single Han character or kana. It's a real-world example of how "the UI is multilingual" and "the text-processing logic is multilingual" are two completely different layers.
1. Two things "multilingual support" usually gets bundled into
For a web tool, "multilingual support" usually gets bundled into two separate things. One is translating interface strings — buttons, labels, instructions. The other is processing the user's actual input data (text) according to that language's grammar and writing system. The first is done once you fill in a language-keyed dictionary of strings in an _i18n object; the second requires every regex, tokenization rule, and character range to have actually been written with that language in mind. The fact that these two layers don't automatically move together is the core of this case.
2. What the regex actually looks like
The N-gram Analyzer's "remove punctuation" option is on by default, and in the code it's applied as text.replace(/[^\w\s가-힣]/g, ' '). Read literally, this regex says: "replace any character that is not a Latin letter/digit/underscore (\w), whitespace (\s), or a precomposed Korean Hangul syllable (가-힣) with a space." In other words, the only character classes explicitly on the allow-list are the Latin alphabet and Hangul — Chinese Han characters (the CJK Unified Ideographs block) and Japanese hiragana/katakana appear nowhere on that list. Any character not on the allow-list is swapped for a space with no exception handling, so a word made up entirely of Han characters or kana disappears without a trace before tokenization even begins.
3. What actually happens when you feed it CJK text
Here's what happens when a sentence mixing Korean, Chinese, and Japanese is run through the analyzer with the default setting (remove punctuation on).
| Original input | Text remaining after the regex |
|---|---|
| 这是一个测试。日本語のテストです。한국어 테스트입니다. | 한국어 테스트입니다 |
4. Why Korean got the special exception
In JavaScript regex, \w doesn't mean the full Unicode range — it means the ASCII range only (a-z, A-Z, 0-9, _). Since this tool originally launched as a Korean service and treated Korean-content analysis as the top-priority requirement, it looks like the "Hangul is getting stripped out" problem was the one bug caught during development, so the 가-힣 range was explicitly added to fix it. Later, when the UI was translated into Chinese and Japanese as well, the interface layer expanded — but the processing-logic layer, the regex, never caught up with that expansion. In other words, the asymmetry where "Hangul works but Han characters and kana don't" isn't really a one-off mistake — it's the classic hole that opens up when a regex written around a single language gets reused as-is for other languages.
5. Does turning off "remove punctuation" fix it?
Unchecking "remove punctuation" does stop this regex from being applied at all, so Han characters and kana do survive. But the trade-off is that real punctuation marks — periods, commas, question marks — stay embedded in the tokens and get mixed into the N-gram results. For example, a token like "테스트。" would get tallied as a separate entry from "테스트", muddying the frequency counts. On top of that, if the original text contains double quotes while this option is off, you'd also run into a separate bug: the CSV export code doesn't escape double quotes inside fields, which breaks the column structure of the exported CSV. In the end, there's no workaround in the current version for handling Chinese or Japanese text other than turning the option off — and even that isn't a complete fix.
6. How to avoid language bias like this in practice
If you ever need to write your own text-preprocessing regex, it's safer to use Unicode script properties (\p{Script=Han}, \p{Script=Hiragana}, \p{Script=Katakana}, etc., which require the u flag) instead of listing out specific character ranges one by one. But if you're in the position of using an already-shipped tool like this one where you can't edit the regex yourself, the realistic workaround is to adjust the options to match the language you're analyzing and get in the habit of eyeballing the results afterward. If all you need is keyword frequency, the Word Frequency Counter or Keyword Extractor may give you similar results with simpler logic, and if you also want stopword removal, the Stopword Remover is worth a look too. That said, those tools may also have language-specific quirks baked into their internal logic, so when working with multilingual text, actually eyeballing the results is always the safest check.
Frequently Asked Questions
Q. Does this mean the N-gram Analyzer doesn't support Chinese and Japanese at all?
A. It supports UI translation, but not text-processing logic under the default settings. Turning off "remove punctuation" does let the characters survive, but that comes with the side effect of punctuation marks getting mixed in, so it's hard to call it full support.
Q. Why does the text disappear silently instead of throwing an error?
A. Because regex replacement just swaps unmatched characters for spaces — it doesn't throw an exception. From the program's point of view, that's normal behavior, so there's no reason to raise a warning, which is exactly why users have no way to tell on their own why the results came back empty.
Q. Does English or other European-language text get processed without issue?
A. Basic ASCII letters (a-z, A-Z) are included in \w, so they're processed normally. However, accented Latin characters like é, ü, or ñ fall outside the \w range and get replaced with spaces and disappear, just like Han characters and kana.
Q. Could similar language bias exist in other text tools?
A. Yes, it's possible. A regex that hardcodes specific character ranges can reproduce the same type of problem for any language outside that range, so when choosing a tool for multilingual text, it's safer to actually test it with real text in that language and check the output yourself.