What Counts as a Duplicate Line? Case, Whitespace, Punctuation, and Hidden Characters

Two lines can look the same to a person and still be different to a duplicate-removal tool. The key question is not whether the lines seem equivalent. It is what rule the tool uses to decide that two prepared lines match.

Mycelgrid Remove Duplicate Lines performs textual line matching. By default, it trims leading and trailing whitespace and compares lines without case sensitivity. It does not automatically ignore punctuation, internal spacing, invisible characters, or differences in meaning.

Start with the matching rule

Consider these two lines:

Apple
apple

With the Tool’s default case-insensitive matching, they belong to the same matching group. If Match case-sensitively is enabled, they are different because capitalization becomes part of the key.

This is a matching choice, not a judgment about which mode is universally correct. Case can be irrelevant in an ordinary name list but meaningful in identifiers, code, or other structured values.

“Ignore surrounding spaces” changes the result too

The current option label says Ignore surrounding spaces, but the implementation uses JavaScript String.trim(). In practical terms, it removes leading and trailing whitespace from each line before matching.

That option is not comparison-only. The trimmed line is also the value that can survive into the result.

For example, a source line with extra leading or trailing whitespace can match its trimmed counterpart, and the surviving output will use the trimmed form. If boundary whitespace carries meaning in your data, review the result rather than treating this as a harmless comparison setting.

Internal spacing is different

Trimming the boundaries does not collapse spacing inside a line.

New York
New  York

Those prepared values can remain different because the internal spacing differs. If inconsistent internal whitespace is the real problem, normalize that intentionally before or after deduplication with a tool suited to that job.

Punctuation still matters

These are different text values:

example.com
example.com,
example.com/

A person may understand them as closely related, but the duplicate-line Tool does not strip punctuation to force a match. That restraint matters for URLs, filenames, identifiers, code, and other text where punctuation can carry information.

Hidden and Unicode differences can prevent a match

Text can contain characters that are hard to see. Two lines that render similarly may contain different Unicode characters or invisible characters that are not removed by the selected preparation rules.

The Tool does not promise universal Unicode normalization or semantic equivalence. Its default case-insensitive key uses the browser’s toLocaleLowerCase() behavior, so locale-sensitive casing details should not be generalized into a universal case-folding guarantee.

Empty-line removal is a separate operation

Remove empty lines is enabled by default, but it is not the same operation as deduplication. After optional trimming, prepared lines that are exactly empty can be discarded before duplicate matching.

That is why the Tool reports Duplicates removed and Empty lines removed separately. One answers “which repeated values were removed?” The other answers “which empty prepared lines were discarded?”

Textual duplicates are not semantic duplicates

A deterministic line deduplicator does not decide that two differently written values mean the same thing.

USA
United States

or:

Acme Corp.
Acme Corporation

may refer to the same real-world entity, but they are not the same prepared text. Entity resolution, fuzzy matching, contact merging, and AI-assisted semantic matching are different tasks.

Which representation survives?

Matching decides which lines belong to the same group. The survivor rule decides which representation is kept.

By default, Mycelgrid keeps the first matching occurrence. If Keep last occurrence is enabled, the last matching occurrence survives instead. With case-insensitive matching, that can change the capitalization you see in the final list. With trimming enabled, the survivor has already had leading and trailing whitespace removed.

For a fuller explanation of how keep-first, keep-last, and optional sorting affect survivor order, read How to Remove Duplicate Lines Without Changing Their Order.

A reliable review sequence

When the distinctions in a list matter:

1. Decide whether capitalization should distinguish values.
2. Decide whether leading and trailing whitespace is disposable.
3. Check whether punctuation or internal spacing may carry meaning.
4. Keep semantic/entity cleanup separate from textual deduplication.
5. Remove duplicates and compare the generated result with the source.
6. Copy or download the result only after the surviving representations look correct.

The goal is not merely to make a list shorter. It is to define equality narrowly enough that the Tool removes the repetitions you intend without silently merging distinctions it does not understand.