How to delete duplicated files based on pattern (see examples below) using a script or a Bash command

Question

My question is simple :

I have video files named as follows :

xxxx_yyy_720_3800.mp4
xxxx_yyy_720_8000.mp4

where yyy can vary in length (e.g yyyyyy or more y)

As I'm not used in shell scripts I'd like to delete automatically (in a specified folder)

If a file with the name xxx_yyy_720_3800.mp4 exists and if the file xxx_yyy_720_8000.mp4 exists, delete xxx_yyy_720_3800.mp4
If there is only a file xxx_yyy_720_8000.mp4 and not a file xxx_yyy_720_3800.mp4, do nothing.

Any help would be greatly appreciated.

use fdups http://askubuntu.com/questions/177346/how-to-use-fdupes — Panther, Jul 14 '15 at 16:27
Is it only in xxxx_yyy_720_3800.mp4 the the number of y may vary or is it also in xxxx_yyy_720_8000.mp4? If so, how do those things relate (i.e. should any file matching xxxx_yyy_720_3800.mp4 be deleted regardless of the number of y in xxxx_yyy_720_8000.mp4)? — kos, Jul 14 '15 at 17:15
the number of yyy can vary from one set of files to another but i'd like to delete files files with the same yyy number and if the couple of files xxx_yyy_720_8000.mp4 and xxx_yyy_720_3800.mp4 exist the xxx_yyy_720_8000.mp4 has to be deleted — Laurent06000, Jul 18 '15 at 18:33
Contrary to the proposition of using fdupes, fslint or DupeGuru my question is to complicated to use those tools — Laurent06000, Jul 18 '15 at 18:46
As I mentioned the number of yyy can vary but I want to delete xxx_yyyy_720_8000.mp4 only if the file xxx_yyyy_720_3800.mp4 exists with the strictly same yyyy pattern. — Laurent06000, Aug 01 '15 at 13:54

A.B. · Answer 1 · 2015-07-18T18:37:07.023

Using `find` and `gawk`

Install gawk
```
sudo apt-get install gawk
```
Go into your folder or replace the . after the find command with your foldername, eg: find ~/my_video_duplicates f -iname …

Test the command

The command below shows only the remove candidates

find . -type f -iname "*_8000.mp4" -print0 | \
    while read -d $'\0' file; do \
        gawk -F_ '{ \
            a=gensub(/\_8000\./, "_3800.", "g" , $0); \
            system("if [ -f \""a"\" ]; then echo \""a"\" will be deleted; fi")}' <<< "$file";\
    done

Check again, if you are in the right folder or replace the . after the find command with your foldername, eg: find ~/my_video_duplicates f -iname …

If you sure, run the command below

find . -type f -iname "*_8000.mp4" -print0 | \
    while read -d $'\0' file; do \
        gawk -F_ '{ \
            a=gensub(/\_8000\./, "_3800.", "g" , $0); \
            system("if [ -f \""a"\" ]; then rm \""a"\"; fi")}' <<< "$file";\
    done

Example

The starting situation

% ls -og
total 3
-rw-rw-r-- 1 0 Jul 14 19:37 xxxx_yyy_720_3800.mp4
-rw-rw-r-- 1 0 Jul 14 19:20 xxxx_yyy_720_8000.mp4
-rw-rw-r-- 1 0 Jul 14 19:21 aaaa_yyy_720_8000.mp4

The dry run

% find . -type f -iname "*_8000.mp4" -print0 | \
    while read -d $'\0' file; do \
        gawk -F_ '{ \
            a=gensub(/\_8000\./, "_3800.", "g" , $0); \
            system("if [ -f \""a"\" ]; then echo \""a"\" will be deleted; fi")}' <<< "$file";\
    done
./xxxx_yyy_720_3800.mp4 will be deleted

The removal

% find . -type f -iname "*_8000.mp4" -print0 | \
    while read -d $'\0' file; do \
        gawk -F_ '{ \
            a=gensub(/\_8000\./, "_3800.", "g" , $0); \
            system("if [ -f \""a"\" ]; then rm \""a"\"; fi")}' <<< "$file";\
    done

The final situation

% ls -og
total 2
-rw-rw-r-- 1 0 Jul 14 19:20 xxxx_yyy_720_8000.mp4
-rw-rw-r-- 1 0 Jul 14 19:21 aaaa_yyy_720_8000.mp4

Thank you A.B. but I tried the first of your commands (the one to echo the name of the file to be deleted) in a test folder containing a couple of xxxx_yyy_720_3800.mp4 and a xxxx_yyy_720_8000.mp4 files but it echoes nothing. As I'm not a gawk specialist I'm not able to tell what's wrong — Laurent06000, Jul 14 '15 at 21:08
Nothing or an error message? Can you post an example of the filenames? In my script, I check only the 8000 part — A.B., Jul 15 '15 at 05:18
Nothing is displayed (the file $0 will be deleted is not displayed). No error message. example of a couple of files xxx_yyyy_720_3800.mp4 and xxx_yyyy_720_8000.mp4 in the same test folder. if the yyyy vary from one file to another then the files must NOT be considered as "duplicates" — Laurent06000, Jul 18 '15 at 18:35
@Laurent06000 If your file name ends with*_8000.mp4 as described in your question, then something is displayed. — A.B., Jul 18 '15 at 18:39
Sorry for the mistake but here what I get in a terminal on a testing folder : find . -type f -iname "*_8000.mp4" -print0 | \

while read -d $'\0' file; do \ gawk -F_ '{ \ a=gensub(/\_8000\./, "_3800.", "g" , $0); \ system("if [ -f \""a"\" ]; then echo \""a"\" will be deleted; fi")}' <<< "$file";\ done

./xxx_yyyy_720p_3800.mp4 will be deleted laurent06000@PC-DE-LDP:~/Vidéos$ ./xxxx_yyyy_720_3800.mp4 will be deleted with yyyy stricly the same. I'll try it live on my real forler. Many thanks. Laurent — Laurent06000, Aug 01 '15 at 14:11

Martin Thornton · Answer 2 · 2015-07-18T21:10:18.270

1

Using the `Bash` Shell

[ -f "file" ] checks whether a filename exists, and is an ordinary file (e.g. not a directory or symbolic link)

"${name/%x/y}" replaces the suffix x of $name with y.

Therefore to delete xxx_yyy_720_3800.mp4 only if xxx_yyy_720_8000.mp4 exists, where xxx and yyy are identical in each case, for all *_*_720_3800.mp4 in the current directory:

for name in *_*_720_3800.mp4
do if [ -f "${name/%3800.mp4/8000.mp4}" ]
   then echo "$name"
   fi
done

Change the echo to rm if you are sure this is OK.

edited Jul 18 '15 at 21:10

answered Jul 14 '15 at 18:30

Martin Thornton

5,431

Thank you Martin, but this solution does not appear to be a safe one : keep in mind that only files with yyyy be the same between xxx_yyyy_720_3800.mp4 and xxx_yyyy_720_8000.mp4 files can be considered as "duplicates". – Laurent06000 Jul 18 '15 at 18:42
It is correct. foo_bar_720_3800.mp4 would only be deleted if exactly foo_bar_720_8000.mp4 exists, and foo_baz_720_3800.mp4 would only be deleted if exactly foo_baz_720_8000.mp4 exists. Maybe I confused you with: "${name/%x/y}" replaces the suffix x of $name with y. That x=720_3800.mp4 and that y=720_8000.mp4 Each name is considered separately. – Martin Thornton Jul 18 '15 at 20:38
Sorry for the mistake but your script seems to work on a testing folder. I'll try it live in the real folder. Many thanks. Laurent – Laurent06000 Aug 01 '15 at 14:13

How to delete duplicated files based on pattern (see examples below) using a script or a Bash command

2 Answers2

Using find and gawk

Using the Bash Shell

Using `find` and `gawk`

Using the `Bash` Shell