Archiving file duplicates

Finding file duplicates across your network has been possible for a while, using tools like SPACEWatch Storage Suite from Sharpeware.  But what to do once you’ve discovered that there are thousands of them – and yes, they do take up an inordinate amount of your storage?

An interesting new feature has just been added to SPACEWatch’s “Enterprise Edition” – it lets you archive file duplicates.  This means you can choose what type of files to focus on (e.g. pesky audio files), find all the duplicates, and replace them with stubs pointing to just one of the copies – all in one task.  What’s more, once you’re happy with the way it works, you can schedule tasks to run automatically!

The screen shot below shows some file duplicates on a medium sized network, that we found using SPACEWatch.

I’ve expaneded one of the duplicate file sets.  The green icons indicate files that have never been used (i.e. never opened since they were created).  Pink highlights files that haven’t been used for a long time.

Now we can manually carry out what Sharpeware call an “inline” archive, or a traditional archive.  Inline archive de-duplicates the file set by replacing all the files (except one that we choose) with stubs that redirect applications to the one we choose.  Traditional archiving replaces the files with stubs that redirect application to another location.  Both can be automated with the administrator tool that comes with SPACEWatch.

Not only is this an interesting – and easy – way to reduce storage use and waste, but it can be applied across your network.  Duplicates are found wherever they are – not just one PC or server.  In fact the search can be extended to the file attachments in email systems like Exchange and Lotus Domino as well.