What do you want to achieve?

Previously we blogged about how to archive duplicates – and how to automate this, so you don’t have to spend time manually finding and dealing with file duplicates.  How do you find the interesting ones in the first place – the ones that you want to spend time on?  First you need to decide what you are trying to achieve, e.g.

• Clean-up as much storage as possible, as quickly as possible
• Find out which users are generating the most duplicates, and tell them about it
• Check where most file duplicates are being stored – and if I have problems with whole folders being duplicates
• Decide if I am concerned about any type of file, or just ones I hate, like audio or video files.

How accurate should I be?

Having decided what you are looking for, you need to decide how accurately you want to search.  This is a tradeoff between speed and performance – unless you choose a tool where this can be automated to a schedule.  If you want to clean-up across your entire network you need to consider this area carefully.  Scaling up duplicates file search can be a challenge and is highly tool dependent.

A good example of this issue is whether or not to use “checksum” comparison to find duplication.  In this method the checksum of each file is calculated and compared with each other file.  To calculate a checksum the data in each file must be read and a calculation applied.  When files get to larger sizes (e.g. 50MB or more) then calculating a checksum can take minutes, even in the best conditions (when a file is stored on a local filesystem and the PC or server is powerful and not running other tasks).  Some tools let you do a “byte by byte” comparison of files, rather than calculating and comparing checksums.  This takes even longer and arguably gives very little extra accuracy.

So how accurate is a checksum comparison of files?  This depends on the algorithm used.  With a 32 bit checksum algorithm we estimate that the likelihood of getting an incorrect duplication result on a large network will be once every 20 years or so… if you’re cleaning up your storage (rather than carrying out forensic investigation, for example) then you’re probably not too bothered by this margin for error.  In fact, for most purposes you can probably dispense with checksum calculation altogether and just use meta data like file update date, size and file type to clean up.

How do I want to organize results?

You’ve decided what you want to achieve, and how accurate you want to be.  Next its worth thinking about how to organize the results of your analysis.  If you want to find out whose creating most duplication then looking at results by file owner would be useful.  If you want to find duplicated directories then viewing by folder would be best.  If you just want to clean up wasted storage then using file lists would be most easy.

An example of file duplicates organized by file

An example of file duplicates organized by creator

An example of file duplication by folder

The examples above come from our favourite storage management tool, SPACEWatch Storage Suite. Its Duplicates Finder lets you choose from many options to filter where – and what – you look for.  It then uses databases-driven search to give very fast results.  What’s more it will find duplicates across multiple servers, storage appliances – and even email systems.

What to do with the results?

So you’ve got the results you want, organized as you want them.  What next?  With SPACEWatch you can:

• Generate and email a report – e.g. to the owners of the duplicates
• Carry out a file action like copy, move, delete – or more advanced actions like archiving or sending to compressed folders
• Save the results – e.g. to publish on an intranet site, or use in another report
• Save the search – so you can run the exact same search with one click, or automate it to run to a schedule.

Usually, the easiest part of the business case is describing what you want to do – the “end state” or final infrastructure configuration.  This is the “new” stuff, and suppliers are always at hand with white papers and product data sheets to help with the detail.  Conversely, often the most difficult part is building the financial justification – what will be improved as a result of this change, or what cost will be reduced?  The reason this is difficult is because it requires a detailed understanding of your current infrastructure – and no supplier can give you a data sheet covering that!

So what you need for your next storage business case is a painless way to find out what’s out there – and what the impact of a planned change might be.  Step up SPACEWatch Storage Suite.  Using it’s Scenario Planning tool its possible to choose all – or part – of your storage, choose a change scenario like “all files unused for 3 months”, then calculate in seconds the impact of that change.  What’s more, the change impact is calculated by looking at what the historic impact would have been on your storage, then extrapolating this forward.  Add in your own TCO (total cost of ownership) figures and it will provide the financial impact as well – all ready to be pasted into your business case paper.

In the example above we’ve chosen “all files unused for 3 months” as an example and considered what would happen if we applied this across all our storage.  SPACEWatch lets us compare this with a “do nothing” scenario to clearly see the benefit of the change – if users were to continue using storage the way they always have.  This example is interesting – not only is there clear and substantial savings, but it will take me a year or more to get back to the same level of storage use.  Storage growth also becomes more linear, indicating that historically the growth in storage taken up by unused files has actually been increasing!  Definitely time to act – and there’s the evidence to argue your case.

Why not download and try a free trial of SPACEWatch now – it will work with any size of network.

Don’t ask me what that dog was all about!

The rest, as they say, is history.  Now we have a Windows 7 search tool that depends on a heavy-weight indexing system.  Test your knowledge and prove that it really is easy to use (not my experience) … tell me how you start an advanced search?

Complexity isn’t the only issue – Microsoft have increased indexing sophistication because we now create so many more files.  No one worries about having enough disk space – they create backups and copies of copies and never clean up.

And if you use a network file server, who know where that file went that you were working on last month.  Perhaps its in that email I sent?

The future is worse — apparently we’ll all be using cloud file storage more and more in the future.  Cloud services are typically priced on data transfer and file access frequency, rather than just storage space consumed – so consider the challenge (translates as “cost”) of searching your cloud storage – ignoring any performance issues!

Microsoft wimped out of building a “proper” database-based file system into Windows… one day perhaps.  In the meantime you can use software like SPACEWatch Storage Suite to give you instant network-wide file search.

The screen shot below shows the main SPACEWatch File Finder window.  You can see the results of a search on the right, and a “visualisation” pane on the left.  The visualisation pane shows all the systems (servers etc) and users for whom results have been found.  Results can be filtered by clicking on one of these, or entering text into the filter box on the top right.

Note there are other panes available as tabs on the left and right borders of the windows.  These give access to the custom search options, for example.  Custom options include obvious things like file name, size, type, owner etc. but also more unusual options like whether or not the file has ever been used.

SPACEWatch lets you save even the most complex searches and re-use them with one click, or generate reports from them.  Other SPACEWatch users can re-use these searches as well.  Interestingly, how you sort the results is also saved – so, for example, you can set a limit on how many results to show and sort the results by size – and get a dynamic “top x” list.

SPACEWatch does use a proper database to store the file data it collects – so you can be assured that even the most complex searches, on the largest networks, produce results in seconds.  Don’t believe me?  Try downloading one of the free trial versions and see for yourself.

This can sometimes be addressed with Exchange archiving solutions – but keeping local PST archives still gives users the easiest way to search and find what they want in old mailboxes.

So what to do?

A good start is to find out the scale of the problem.  Take a look at the screen shot on the left.  This is from SPACEWatch‘s File Types analysis tool.

This shows how PST files have been found right across the network – I can expand the tree to see how many files, and how much storage they’re consuming.

If I want to investigate further I right click and choose one of the context menu options such as listing the largest ones in a particular area of the network – or the least used.

But what about cleaning up all this potentially wasted space?  There are a number of approaches you can take, all of which SPACEWatch will help with, and many of which can be automated to run to a scheduled routine:

• Send users a list of the PSTs they own and ask them to check if they’re still needed – and remove them if they’re not.
• Archive PST files that haven’t been used for a long time – typically leaving a “stub” that looks like the original file to any application, but redirects the application to the file’s new location (which could be cheap secondary storage).
• Delete unused PST files.
• Leave the PST files where they are, but archive file attachments within them (this is typically what takes up most of the file space): SPACEWatch with it’s Exchange add-on lets you do this on PST files, Exchange mailboxes and public folders.

Perhaps if you follow some of these approaches, you won’t end up with a top twenty file types chart like this!  By the way, someone’s already been cleaning up on this network – you can tell, because “mp3″ only just scrapes in to the top twenty.  Try this with SPACEWatch on your network and I’ll bet its a lot higher!

An interesting new feature has just been added to SPACEWatch’s “Enterprise Edition” – it lets you archive file duplicates.  This means you can choose what type of files to focus on (e.g. pesky audio files), find all the duplicates, and replace them with stubs pointing to just one of the copies – all in one task.  What’s more, once you’re happy with the way it works, you can schedule tasks to run automatically!

The screen shot below shows some file duplicates on a medium sized network, that we found using SPACEWatch.

I’ve expaneded one of the duplicate file sets.  The green icons indicate files that have never been used (i.e. never opened since they were created).  Pink highlights files that haven’t been used for a long time.

Now we can manually carry out what Sharpeware call an “inline” archive, or a traditional archive.  Inline archive de-duplicates the file set by replacing all the files (except one that we choose) with stubs that redirect applications to the one we choose.  Traditional archiving replaces the files with stubs that redirect application to another location.  Both can be automated with the administrator tool that comes with SPACEWatch.

Not only is this an interesting – and easy – way to reduce storage use and waste, but it can be applied across your network.  Duplicates are found wherever they are – not just one PC or server.  In fact the search can be extended to the file attachments in email systems like Exchange and Lotus Domino as well.