Text and data cleanup tool
Remove Duplicate Lines
Remove repeated lines from pasted text while preserving the original order or applying the matching rules you choose.
Paste lines
Use one item, keyword, URL, email, or value per line.
Unique lines
Review the cleaned list before copying or downloading it.
Your text is processed directly in your browser by this tool. This version does not upload the lines you enter to Mycelgrid for processing.
How to remove duplicate lines
- Paste one item per line.
- Choose how spaces, capitalization, empty lines, and repeated items should be handled.
- Select Remove duplicates.
- Review the resulting list and statistics.
- Copy the unique lines or download them as a TXT file.
Examples
Clean a keyword list
Remove repeated search terms before importing the final list into a spreadsheet or research document.
Deduplicate URLs or domains
Keep one copy of each URL while preserving the order in which the first occurrence appeared.
Clean copied AI output
Remove repeated bullets or list items from generated text before saving or sharing the result.
Prepare spreadsheet values
Paste a column of values, remove duplicates, and copy the unique list back into a spreadsheet.
Frequently asked questions
Does the tool preserve the original order?
Yes. By default, it keeps the first occurrence of each line and preserves the order of those first occurrences.
Can I keep the last occurrence instead?
Yes. Select “Keep last occurrence.” The final list will retain the last copy of each repeated line.
Does capitalization matter?
Only when “Match case-sensitively” is selected. Otherwise, Apple and apple are treated as duplicates.
What does ignore surrounding spaces do?
It compares lines after trimming spaces from the beginning and end. The kept result also uses the trimmed version.
Can it remove empty lines?
Yes. “Remove empty lines” is enabled by default and reports how many empty lines were removed.
Is my list uploaded?
This version processes the pasted lines directly in your browser and does not upload them to Mycelgrid for processing.
Method and limitations
The tool splits the input at line breaks, optionally trims each line, optionally removes empty lines, and creates a matching key for each remaining line. Matching keys are case-insensitive unless case-sensitive matching is enabled.
By default, the first occurrence is kept. When “Keep last occurrence” is enabled, the tool scans from the end and restores the surviving items to their original sequence. Optional sorting happens after duplicate removal.
Lines that look similar but contain different punctuation, invisible Unicode characters, or internal spacing may not be treated as duplicates. Review sensitive lists before using them in production systems.
Guides
Learn how matching rules determine which lines count as duplicates and how survivor choices affect order.