The Redeye Complex


Friday, October 30, 2009
My data

Interesting article over on Luminous Landscape about data backup methods for photographers. It can be applied to anyone really, just that photographers and videographers are generating huge files that need to be stored reliably ... somehow and somewhere. Or as I would like to say somewhere and somewhere else. You can never have enough copies of your data.

My day job in the data centre has me thinking of storage, backup and recovery issues every day. So I am versed in all the language, products and such. Still when I read that article, the terms used are so complex that you really have to be versed in the storage world in order to wrap your head around the issues.

Like ...
... the data protection in RAID 5 occurs because a 'check' number is calculated as the data is written and is then stored on a separate disk, if the original data is damaged, the missing or damaged data can be mathematically reconstructed. This is known as a 'rebuild'.
Uh huh. Now the punch line ...
The down side of this is that the system can tolerate losing only one drive, not two, and a rebuild can take a long, long time for a large data set. We recently rebuilt a large array at a television studio, with about 12TB of data, it took a week to rebuild, the entire system was down until that operation completed.
Uhh yeah.

One of many problems is that there is no one product that does the job. Why? Well because no really describes what job needs to be done. All products are "backup" products. Now who doesn't want their to be backed up. But what is really required is to be able to restore the data when needed. This is not an easy task when you think about it.

The issue of the week long storage rebuild has nothing to do with "backing up your data". However when you want to restore your data, and its gonna take a week! Well I didn't think that would be an issue.

The task of being able to restore the data really means that the equipment where your data is stored needs to be tested and monitored constantly to make sure that it still works. And not just the equipment. What really what needs to be done is to restore your data more often than you need to. How often? Are you up for restoring a few terabytes of photographs every week? Every month? Once a year?

The answer of course is that you do need to develop a routine, a workflow that tests your backed up data ... on a regular basis.

If you are not up for that then you can just have faith that where you are storing your data will be there when you need it.

The real answer then is somewhere in between. You need to have some faith in the equipment and media you are using. Something that does RAID5 is a good choice. However and remember that having your data only on a RAID array does not mean it is backed up. You really do need multiple copies of your data. Stored in two different places. Using different methods. And over time you do need to look at your data to make sure that it is still usable.

I read an article somewhere suggesting the best way for all your photographs to be stored reliably is to become famous and let the world's galleries and libraries do it for you. Of course if you are famous you are likely deceased. See. I told you this was a tough problem to solve.

4 comments


Newer Posts Older Posts