Mysql: How to clone a table removing duplicate entries

If you have a table in MySql having duplicate entries an you want to remove these duplicate entries you can do that by cloning the table. In MySQL this can be obtained by the following one liner:

CREATE TABLE new table (  `ID` int(11) NOT NULL AUTO_INCREMENT, `col1` int(11) NOT NULL, `col2` varchar(1000) NOT NULL,`col3` varchar(255) NOT NULL,`col4` varchar(30000) NOT NULL, PRIMARY KEY (`ID`)) SELECT DISTINCT * FROM oldtable;

The column definitions are just for demonstration purpose and have to be identical with the original table. By the SELECT DISTINCT query you obtain a copy of the table without duplicate entries.

What to do if you get a bash: /usr/bin/rm: Argument list too long error.

If you have directories which you contain millions of files you might get in trouble when you want to apply an operation on a bunch of files. For example rm *.jpg will not work in that case due to memory limits.

An easy way is you to use find for the job you want to do:

If you want to move all files in a directory which have the extension jpg.gif to gif only  you can use the following command:

> find -name “*.gif.jpg”  -exec mv ‘{}’ jpg \;

Another useful application is to easily remove all files with the extension .jpg

This can be done with the following command:

> find -name “*.json”  -exec rm ‘{}’ \;

In principle you can apply find to any other job you want to perform.