38 Open-source Free Duplicate File Finder and Removal Tools

38 Open-source Free Duplicate File Finder and Removal Tools
Photo by Unseen Studio / Unsplash

Duplicate file removal is the process of identifying and removing duplicate files from your computer. Duplicate files are identical files that occupy unnecessary disk space and can slow down your computer.

There are several reasons why you might need to remove duplicate files from your computer. Firstly, duplicate files consume valuable disk space on your computer. This can lead to your computer running slowly or even crashing. Secondly, duplicate files can make it difficult to find the files you need. This can be frustrating and time-consuming. Finally, duplicate files can also make it difficult to backup your files. Backing up duplicate files is a waste of time and resources.

Advantages of Cleaning Your Duplicate Files

1. Free Up Disk Space

By removing duplicate files, you can free up valuable disk space on your computer. This can help improve the performance of your computer and make it run faster.

2. Improve Computer Performance

Removing duplicate files can help improve the performance of your computer. By freeing up disk space, your computer can run more smoothly and quickly.

3. Easier File Management

Having duplicate files on your computer can make it difficult to find the files you need. By removing duplicate files, you can make it easier to manage your files and find the files you need quickly and easily.

4. Save Time

By removing duplicate files, you can save time. This is because you don't have to spend time searching for files or backing up unnecessary files.

5. Better Backup Management

Removing duplicate files can help you better manage your backups. This is because you don't have to backup unnecessary files, which can save you time and resources.

Caution

While duplicating files can be useful in some cases, it is important to be cautious when doing so. Duplicating files can lead to confusion and make it difficult to manage your files. It is important to keep track of which files are duplicates and which files are original. Additionally, it is important to regularly clean up duplicate files to ensure that they are not taking up unnecessary space on your computer.


In the following list, we offer you the best open-source free duplicate file finders and removals that use different methods and algorithms to find, locate and manage your duplicate files.

1- Duplicate Files Finder (Windows, Linux)

Duplicate Files Finder is an open-source, free cross-platform application for finding and removing duplicate files by deleting, creating hardlinks or creating symbolic links. A special algorithm minimizes the amount of data read from disk, so the program is very fast.

Platforms: Windows, Linux

Duplicate Files Finder homepage
Duplicate Files Finder
Download Duplicate Files Finder for free. Duplicate Files Finder is a cross-platform application for finding and removing duplicate files by deleting, creating hardlinks or creating symbolic links. A special algorithm minimizes the amount of data read from disk, so the program is very fast.


2- DuplicaErase (Windows)

DuplicaErase is a free and open-source lightweight app that helps you to identify duplicate files, determine the amount of storage occupied by these files, and seamlessly remove them. This will aid you to up valuable disk space by eliminating redundant copies and optimizing your storage usage.

The app is a free Libre open-source solution (FLOSS). It is released under the GNU General Public License version 3.0 (GPLv3).

DuplicaErase
Download DuplicaErase for free. Introducing a lightweight duplicate file remover tool for Windows. With this software, you can efficiently identify duplicate files, determine the amount of storage occupied by these files, and seamlessly remove them.

3- fclones

fclones is a command line utility that identifies groups of identical files and gets rid of the file copies you no longer need. It comes with plenty of configuration options for controlling the search scope and offers many ways of removing duplicates. For maximum flexibility, it integrates well with other Unix utilities like find and it speaks JSON, so you have a lot of control over the search and cleanup process.

fclones treats your data seriously. You can inspect and modify the list of duplicate files before removing them. There is also a --dry-run option that can tell you exactly what changes on the file system would be made.

fclones is written in Rust, so it runs on Windows, Linux, and macOS.


With fclone you can:

  • Identify groups of identical files
  • Select multiple files
  • Remove duplicate files and clean your deskspace

fclones supports parallel processing capability in all I/O and CPU heavy stages, automatic tuning of parallelism and access strategy based on device type (SSD vs HDD), and dozens of file formats.

License: MIT.

GitHub - pkolaczk/fclones: Efficient Duplicate File Finder
Efficient Duplicate File Finder. Contribute to pkolaczk/fclones development by creating an account on GitHub.

4- Duplicate File Remover

This free app allows you to detect and remove duplicate files on your computer.

Duplicate File Remover
Download Duplicate File Remover for free. Detect and remove duplicate files on your computer. Detect and remove duplicate files on your computer. By this free tool, you can help your computer work faster by removing unnecessary files.

5- DupTerminator (Windows)

DupTerminator is a free Windows program to search, delete, move, and rename duplicate file by comparing MD5 hash. The program is free and open source software.

DupTerminator features:

  • Sort groups of duplicate by column.
  • Select files in a specific folder, in a folder in the groups that have all folders from the selected group, by date, by name length, by specified name.
  • Rename files, moving files.
  • Delete to recycle bin.
  • Load / Save the list of duplicates.
  • Change program font, font and color of list duplicates.
  • English, Russian, Portuguese language in xml.

License: GNU General Public License version 3.0 (GPLv3).

DupTerminator
Download DupTerminator for free. DupTerminator will scan your directories for any duplicate files using. DupTerminator - a program to search, delete, move, and rename duplicate file by comparing MD5 hash. The program is free and open source software.


6- DupMaster (macOS)

DupMaster is a simple free app that locate and help you clean macOS from duplicated files. It is an open-source project that is released under the GPL3.0 License.

You can set your search settings, choose your scanning algorithm between file content, or filename/ size. Also, you can select the file size range for your search.

DupMaster
Download DupMaster for free. Get rid of duplicate files on your Mac. A Powerful Duplicate File Finder and Cleaner for Mac OS X.

7- Duplicate File Finder 2.2 (Windows)

Duplicate File Finder app is a free tool that searches a specific folder and its sub-folders for any duplicate files. Once the tool detects duplicate files, the user can decide to delete them, freeing up a significant amount of hard drive space.

The app is released under the Creative Commons Attribution License.

Features

Full folder and folder-subdirectory file search

  • Smart progress bar
  • Can detect any type of file
  • Ability to open file location of selected file
  • Ability to delete files that you choose
Duplicate File Finder 2.2
Download Duplicate File Finder 2.2 for free. A program used to find duplicate files on your computer. The Duplicate File Finder by Autosofted is a free tool used to search a specific folder for any duplicate files that the folder and it’s sub-folders may have. After detecting the duplicate files, th…


8- GUI for fclones

This app offers a free GUI for the clones command-line tool to find and delete duplicated files.

GitHub - pkolaczk/fclones-gui: Interactive duplicate file finder and remover
Interactive duplicate file finder and remover. Contribute to pkolaczk/fclones-gui development by creating an account on GitHub.

9- MediaFileDuplicateFinder

Media File Duplicate Finder apps helps you find duplicated video and image files based on similarity.

You can easily find duplicates which have different resolution, frame rate, watermark, and video file tags. It's a cross-platform program that supports Windows, Linux, and macOS.

Features:

  • Finds duplicate videos / images based on similarities
  • Fast scanning speed
  • Cross-platform: Windows, Linux and macOS
  • Windows Installer (MSI package), which includes FFmpeg binaries.
  • Play/Open Videos and Images from the list of duplicate files.
  • Optionally select duplicate files with lower resolution for deletion.


MediaFileDuplicateFinder
Download MediaFileDuplicateFinder for free. MediaFileDuplicateFinder finds duplicated video, image, & audio files. Media File Duplicate Finder finds duplicated video and image files based on similarity. Finds duplicates which have different resolution, frame rate, watermark, and video file tags.

10- GDuplicateFinder

GDuplicateFinder is a free cross-platform application that can search for duplicate files not only on local drives but also on network shares, including Windows and Linux shares using the VFS library. Unlike other duplicate finders, such as Easy Duplicate File Finder, Auslogics Duplicate File Finder, MindGems Fast Duplicate File Finder, Duplicate Cleaner, and DupFiles, GDuplicateFinder offers this functionality for free.

Using the power of Groovy facilities and GPars to process in parallel, GDuplicateFinder makes it easy and user-friendly to get rid of the duplicates that you've always wanted to dispose of.

Features

  • Supported network shares (SMB, SSH, Amazon S3, FTP)
  • Manage network drives(add, edit, remove)
  • Turn on/off network drives
  • Save/load duplicates results session
  • Stop, save, load and resume a search for duplicates
  • Open duplicate files with system's default editor
  • Internationalization (Spanish, English, Portuguese, Traditional Chinese, Simplified Chinese)
  • Receive update notifications
GDuplicateFinder
Download GDuplicateFinder for free. GDuplicateFinder - A FREE Groovy way to find file duplicates! Unlike other duplicate finders such as Easy Duplicate File Finder, Auslogics Duplicate File Finder, MindGems Fast Duplicate File Finder, Duplicate Cleaner, DupFiles and others, GDuplicateFinder is a FRE…

11- DupFinder (Windows)

  • Free Duplicate File Finder and CleanUp Tool for Windows© XP/Vista/7/8

This app (DupFinder), allows you to find duplicated files in a snap, and clean up your disk from duplicated files with almost no effort. The app is released under the GNU General Public License version 2.0 (GPLv2).

DupFinder
Download DupFinder for free. Free Duplicate File Finder and CleanUp Tool for Windows© XP/Vista/7/8. AFDP can help you free up hard drive space by detecting and removing duplicate files. Duplicate files have exactly the same contents regardless of their name, date, time and location.

12- Duplicate-File-Remover

Duplicate File Remover is a command-line utility that recursively scans the set directory to find exact duplicate files inside all the subdirectories. The files so found can be listed in an output file. If required the duplicates can also be removed, thereby preserving a single unique file.

Its current features include:

  1. Recursive scan of the set directory.
  2. Generate list of duplicate files.
  3. Scan all files or filter based on file extension.

Platforms: Windows, Linux, macOS.

GitHub - vishal-wadhwa/Duplicate-File-Remover: Command line utility to remove exact duplicate files.
Command line utility to remove exact duplicate files. - GitHub - vishal-wadhwa/Duplicate-File-Remover: Command line utility to remove exact duplicate files.


13- dupe_eraser

A command-line tool which automate the deletion of duplicate files.

GitHub - NicolasBizzozzero/dupe_eraser: A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash.
A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash. - GitHub - NicolasBizzozzero/dupe_eraser: A command-line tool which automate the deletion…

14- dude (Windows, Linux)

Duplicates Detector is a GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.

Features

  • Scanning for duplicate files in multiple designated folders (up to 8). Optional "Cross paths" mode
  • Optional command line parameters to start scanning immediately or integrate Dude with your favorite file manager
  • Two synchronized panels
  • duplicates groups
  • directory of selected file
  • Two stage processing
  • interactive marking of files with multiple criteria
  • taking action on marked files (Move to Trash/Recycle Bin, delete, hard-link or soft-link)
  • support for regular expressions or simplified (wildcards) syntax
  • searching for duplicates based on the hash of the file content. Different filenames or extensions do not affect the search results
  • support for Linux and Windows
GitHub - PJDude/dude: Duplicates Detector is a GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.
Duplicates Detector is a GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for effic…

15- jdupes (Linux/ Unix)

jdupes is a command-line program for identifying and taking actions upon duplicate files such as deleting, hard linking, symlinking, and block-level deduplication (also known as "dedupe" or "reflink"). It is faster than most other duplicate scanners. It prioritizes data safety over performance while also giving expert users access to advanced (and sometimes dangerous) features.

GitHub - jbruchon/jdupes: A powerful duplicate file finder and an enhanced fork of ‘fdupes’.
A powerful duplicate file finder and an enhanced fork of ‘fdupes’. - GitHub - jbruchon/jdupes: A powerful duplicate file finder and an enhanced fork of ‘fdupes’.

16- Deduplicator

Deduplicator is a command-line tool written in Rust, that allows you to find, filter, sort & delete duplicate files recursively.

GitHub - sreedevk/deduplicator: Filter, Sort & Delete Duplicate Files Recursively
Filter, Sort & Delete Duplicate Files Recursively. Contribute to sreedevk/deduplicator development by creating an account on GitHub.


17- Findm

Findm is a Python script that searches for duplicate files in a specified directory. It identifies duplicates by computing a hash for each file while also considering other factors.

This script can find duplicate images, documents, videos, and audio files. However, it cannot identify similar audio files with different bitrates as duplicates because different bitrates result in different checksums.

GitHub - sameera-madushan/Findm: Findm is a python script to find duplicate file copies in a given directory.
Findm is a python script to find duplicate file copies in a given directory. - GitHub - sameera-madushan/Findm: Findm is a python script to find duplicate file copies in a given directory.


18- Yet Another Duplicate File Remover (Windows)

Another duplicate file finder/remover. But this one is ULTRA FAST, safe, easy to use and of course free and open source. It is an open-source project that uses GNU General Public License version 3.0 (GPLv3).

Yet Another Duplicate File Remover
Download Yet Another Duplicate File Remover for free. Another duplicate file finder/remover. But this one is ULTRA FAST, safe, easy to use and of course free and open source.


19- pyCleaner

  1. https://github.com/MK-Ware/pyCleaner

This is a simple yet powerful duplicate file removal script, written in Python. Use it with caution.

Features

  • Simple, user-friendly GUI.
  • Cross-platform, It works for Windows, Linux, and macOS.
  • Secure file deletion with multiple passes.
  • Scan directories or whole drives for duplicated files.
  • Choice between secure wiping or normal deletion for dealing with duplicated files.
  • Uses md5 hashing and file size to detect duplicate files.
  • No realistic chance of false positives (see below).
  • Available as a pre compiled Windows binary, or a Python script for Linux in the releases tab.
GitHub - MK-Ware/pyCleaner: Securely wipe files or folders and clean duplicated files
Securely wipe files or folders and clean duplicated files - GitHub - MK-Ware/pyCleaner: Securely wipe files or folders and clean duplicated files

20- File Duplication Checker

This is a command line application that quickly checks to see if files are duplicates of each other. It takes a directory as an argument and scans over that entire directory tree; (it searches files in all subfolders as well as the parent folder). The program detects duplicate files even if they have different names or are in different (sub-)directories.

The app is written in Java, so it can be installed and run on Windows, Linux, and macOS.

GitHub - Gartham/file-duplicate-checker: Program to scan and search for file duplicates. (~300MB/s)
Program to scan and search for file duplicates. (~300MB/s) - GitHub - Gartham/file-duplicate-checker: Program to scan and search for file duplicates. (~300MB/s)


30- samanlainen

Yet another command-line app, written in Rust to find duplicate files.

GitHub - raspi/samanlainen: Delete duplicate files
Delete duplicate files. Contribute to raspi/samanlainen development by creating an account on GitHub.

31- FineSame

A Python tool to find duplicate files and directories. It reports duplicate directories as well, using a Merkle tree for directory hash calculation.

GitHub - elcorto/findsame: Find duplicate files and directories based on file hashes.
Find duplicate files and directories based on file hashes. - GitHub - elcorto/findsame: Find duplicate files and directories based on file hashes.


32- Hashdex

Hashdex is a file indexer based on content hashes to quickly find duplicate files on your system. I created this tool because I always forget which pictures from my phone I already uploaded to my Dropbox account. Now I can just upload all pictures to an uploads directory check these files against the index and only organize the remaining files.

GitHub - jeff-99/hashdex: A file indexer based on content hashes to quickly find duplicate files on your system.
A file indexer based on content hashes to quickly find duplicate files on your system. - GitHub - jeff-99/hashdex: A file indexer based on content hashes to quickly find duplicate files on your sys…

33- duplicate-file-finder

This is a simple app to scan through all the files in a given directory, then list out all the duplicate files according to their MD5 hash values. Very quickly find files with duplicate content, and provides the option to delete duplicates.

It comes with two modes:

  • Quick finder: quickly finds the duplicates using message digest function
  • Memory saves: Finds duplicates by limiting the size of buffer used by the same message digest function.
duplicate-file-finder/screenshots/Screenshot4.png at master · maanavshah/duplicate-file-finder
A Java application with GUI(Swing) to find all the duplicate files in a given directory and all its sub-directories, using the SHA-512 hash function - maanavshah/duplicate-file-finder


34- Easy Duplicae

This Python script compares the files in a folder with md5 checksums and deletes duplicate files or moves them to the desired folder.

GitHub - cobanov/easy-duplicate: Compares the files in a folder with md5 checksums and deletes duplicate files or moves them to the desired folder.
Compares the files in a folder with md5 checksums and deletes duplicate files or moves them to the desired folder. - GitHub - cobanov/easy-duplicate: Compares the files in a folder with md5 checksu…


35- Polllux

Pollux is a simple program written in C that finds duplicate files. It is named after one of the twin stars in the constellation Gemini (literally, "twins"), its twin being Castor.

Pollux scans directories recursively and creates a binary tree of filenames and their sizes. When files are encountered with the exact same size, Pollux calculates their respective SHA-256 message digests and compares them. If they are the same, the files are duplicate files; otherwise, an array pointed to by the node in the tree is created, where all same-sized files are kept, along with their message digests for quick comparison.

GitHub - g-hannah/Pollux: Simple program to find duplicate files
Simple program to find duplicate files. Contribute to g-hannah/Pollux development by creating an account on GitHub.


36- Duplicates Cleaner

Simple script to delete duplicated files from specific directory. Detecting the duplicated files usually done by comparing the files hashes.

GitHub - 0aub/duplicates-cleaner: simple script to delete duplicated files from specific directory
simple script to delete duplicated files from specific directory - GitHub - 0aub/duplicates-cleaner: simple script to delete duplicated files from specific directory

37- DuplicateFileRemover (Java)

This open-source command-line removes all duplicate files (of any format .) recursively inside directories. It is built with Java, so it can run on Linux, Windows, and macOS.

GitHub - jjsridharan/DuplicateFileRemover: Removes all duplicate files (of any format *.*) recursively inside folders. 100% Accurate.
Removes all duplicate files (of any format *.*) recursively inside folders. 100% Accurate. - GitHub - jjsridharan/DuplicateFileRemover: Removes all duplicate files (of any format *.*) recursively…

38- Czkawka

  1. https://github.com/qarmin/czkawka

Czkawka (tch•kav•ka (IPA: [ˈʧ̑kafka]), "hiccup" in Polish) is a simple, fast and free app to remove unnecessary files from your computer. It is written in memory-safe Rust.

Features:

  • Amazingly fast - due to using more or less advanced algorithms and multithreading
  • Free, Open Source without ads
  • Multiplatform - works on Linux, Windows, macOS, FreeBSD and many more
  • Cache support - second and further scans should be much faster than the first one
  • CLI frontend - for easy automation
  • GUI frontend - uses GTK 4 framework and looks similar to FSlint
  • No spying - Czkawka does not have access to the Internet, nor does it collect any user information or statistics
  • Multilingual - support multiple languages like Polish, English or Italian

Included tools:

  • Duplicates - Finds duplicates based on file name, size or hash
  • Empty Folders - Finds empty folders with the help of an advanced algorithm
  • Big Files - Finds the provided number of the biggest files in given location
  • Empty Files - Looks for empty files across the drive
  • Temporary Files - Finds temporary files
  • Similar Images - Finds images which are not exactly the same (different resolution, watermarks)
  • Similar Videos - Looks for visually similar videos
  • Same Music - Searches for music with the same artist, album etc.
  • Invalid Symbolic Links - Shows symbolic links which point to non-existent files/directories
  • Broken Files - Finds files that are invalid or corrupted
  • Bad Extensions - Lists files whose content not match with their extension

Supported Platforms:

  • Windows: 7,8.1, 10, 11
  • Linux: (Ubuntu, Fedora, Alpine Linux, Debian, Solus, and Linux Mint)
  • macOS: 10.15+

There are several GUI apps that use the command-line tool under the hood.

Read more