3 Encoding Traps That Make CSV Diff Flag Every Row as Changed
"The data is clearly identical, but the CSV diff tool marks every row as 'changed'" — in practice this almost never means the data actually changed. It's a text-encoding mismatch happening before the comparison even starts. MODOO HUB's CSV Diff Checker matches rows by treating the first column as a key, but it does not automatically detect or correct character encoding, BOM, or delimiter problems that exist before parsing even begins. This guide breaks down those three traps by root cause.
1. Trap 1: EUC-KR/CP949 vs UTF-8 Encoding Mismatch
CSV files produced in Korea are often saved with EUC-KR or CP949 as the default encoding when exported from Excel as "CSV (Comma delimited)." Paste such a file into an editor or browser that interprets it as UTF-8 and the Korean text turns into garbled characters (mojibake). If one of two CSV files is saved as UTF-8 and the other as EUC-KR, then even though the underlying Korean value is the same, the byte sequences are completely different — so from the diff logic's point of view, the two strings were never the same value to begin with. The result: nearly every cell containing Korean text, such as names or addresses, gets flagged as "changed."
2. Trap 2: Presence or Absence of a UTF-8 BOM (Byte Order Mark)
Windows Notepad and some versions of Excel prepend an invisible 3-byte BOM (U+FEFF) to the start of a file when saving as UTF-8. Compare a file that has this BOM against one that doesn't, and the very first cell value (usually the first column name in the header, or the first cell of the first row) ends up with one extra invisible character attached only in one file. Visually, id and id look identical, but the actual strings are "id" and "id" — different from each other. This causes the common bug where the first column, whether it's a header or data, is always treated as "different."
3. Trap 3: Delimiter Mismatch
Excel in European and Korean locales sometimes defaults to semicolon (;) as the CSV delimiter, while US-locale Excel uses comma (,). The CSV Diff Checker lets you manually pick comma, tab, or semicolon at the top of the screen — but if the two files were actually saved with different delimiters and you select the same delimiter for both, one file's columns never get split at all, and the entire line is read as a single cell value. Once that happens, matching by the first "column" as a key effectively treats every single row as a distinct, unmatched value.
| Cause | Symptom | Fix |
|---|---|---|
| EUC-KR vs UTF-8 | Every Korean-text cell shows as different | Re-save both files as UTF-8 before comparing |
| Presence of a UTF-8 BOM | Only the first column always shows as different | Standardize on UTF-8 without BOM |
| Delimiter mismatch (,/;/tab) | Nearly every row shows as "changed" | Check each file's actual delimiter and select the same one in the tool |
4. A 3-Minute Checklist Before You Compare
- Open both files in Notepad, VS Code, or similar, and match the encoding label shown in the status bar first (UTF-8, UTF-8 with BOM, ANSI/CP949, etc.).
- Check whether the very first character of the first line starts with an invisible character, and confirm the first header column name matches exactly what you expect.
- Open the file and visually confirm whether the actual delimiter is a comma, semicolon, or tab, then set the tool's delimiter option to match.
If the values still come out different after fixing all three of these, that's when you're looking at a real data change. Until the encoding is aligned, don't trust the CSV Diff Checker's output as proof that "the data actually changed."
Frequently Asked Questions
Q. Does the tool automatically detect encoding or BOM and warn me?
No. Since it works by pasting text, it only receives and compares strings the browser has already decoded — it does not automatically detect or warn about the original file's encoding or BOM. You need to standardize the encoding yourself before comparing.
Q. If only the first column keeps showing as different, what should I suspect first?
The BOM (Byte Order Mark). If every other column looks fine but the first column keeps showing "changed," there's a very high chance one of the two files has a UTF-8 BOM and the other doesn't.
Q. What happens if I select the wrong delimiter?
The entire line fails to split and is read as a single value, so the value used as the first column (the key) effectively becomes the whole row's text. As a result, nearly every row fails to match and shows up as added/deleted instead.
Q. How are values that contain a comma inside a cell handled?
A quoted field (e.g. "value,with,commas") is parsed correctly — the parser ignores delimiters that appear inside quotes. This only works, though, when the delimiter itself has been selected correctly in the first place.