ultra high capacity storage

[Mark300SL]

I am in the market for an extremely high capacity storage system - Ideally I want to be able to transport 6-10Tb of data off site at the end of a day and store that data for 7 years (DPA)

I need the cheapest solution for 1 shot storage of say 6Tb (lower end) at a time and ATM I am looking at Tb tape drives as they will also offer a decent physical size for storage.

Anyone else had to do something similar or does anyone have a better solution for high volume storage ?


Cheers


Mark

 

[Sp!ke]

Tape is still the cheapest and easiest backup solution for that kind of volume of data.

I'm not entirely convinced of the reliability of a tape after 7 years unless it has been stored in a controlled environment though.

With terrabyte drives now becoming available for reasonable money, this might be another reasonably cheap failsafe for one off backups.

 

[Spinal]

We still use tapes for our backups; but we also have a RAID array for the day-to-day backups.

As Spike says; it's a good idea to keep the tapes in a controlled environment.

 

[columb]

Do you really need 6-10Tb? Before compression? Daily?
What about incremental backup? Do you create daily 10Tb of data? Whoa!
What about

Raid array + tapes.

You might consider huge raid array with NAS (or if you can afford, SAN ).
It would be really hard to transfer daily 10Tb of data off site...

Cheers
Chris

 

[columb]

I can not see a way to copy 10Tb of data in less than 24h
What about money?

Cheers
Chris

 

[Mark300SL]

Do you really need 6-10Tb? Before compression? Daily?
What about incremental backup? Do you create daily 10Tb of data? Whoa!
What about

Its a vast amount of compressed but high quality video streams from hundreds of very high resolution cameras (think ultimate big brother, system can see a zit on a chav in a crowd!)

It already saves to a 100Tb SAN in rolling storage that drops off the end when the storage is full - however once or twice a week there is going to be a legal requirement to supply the data to plod, and plod have to store it for 7 years.


Money is certainly not the key driver here - but we have to supply the data and it needs to be financially viable even if not cheap.

 

[columb]

Whoa!

Then: to transfer 10Tb of data on 1GB connection you need AROUND 21 hours. ALL THE TIME full bandwidth utilization. You need 1gb connection first How far you need to transfer it away?

And 7 years- do you need to store every copy of 10Tb for 7 years?

Cheers
Chris

 

[Mark300SL]

And 7 years- do you need to store every copy of 10Tb for 7 years?

Transfer will be done locally in the data suite
Yep up to twice a week and all data retained for 7 years - we know its going to cost!

 

[BTB 500]

As mentioned, tape has a very finite life ... even in climate controlled storage. If it's important you would probably want to take 2 copies, test-read everything straight after backup and annually thereafter. Obviously 7 years is quite a long time in IT, you need to know that you'll still have a compatible drive to read the tapes on and that the file format is still supported. Regular test reading will stop any nasty surprises there, as well as making sure the tapes are still OK.

 

[columb]

It will take more than 24h. You need to think about Ethernet protocol and overheads.

I don't know. Giving up
Even if tapes then DLS now is 800GB which means that you need 10-13 tapes for your data. I don't know what's the speeds are now, but I can't imagine writing tapes in less then few hours.
You need as well a huge library.
I would speak to someone from metoffice

And (maybe that sounds stupid) but I would replicate everything in REAL TIME.

Cheers
Chris

 

[Spinal]

I misread your original post then, I thought it was a one-off thing kept for 7 years... I didn't realise you need to transfer 10TB a day!

Conventional tapes might be a little slow then, you may need to invest in quite a few RAID arrays!

Inmarsat (who does satellite data&comms for the US army amongst others) needs to keep copies of all traffic; and sends this once a month over to the states (from their Italian base).

Last time I was there, this was done on hard-drive arrays, and the entire rack of hard drives was shipped monthly across the pond...

Speed-wise, physically moving an array may be your fastest bet...

Michele

 

[columb]

http://www.dlttape.com/products/PerformanceSeries/SDLT600A/index.aspx

In few years time there will be ready solution
Now it is only 300GB but that's what you are looking for!

Cheers
Chris

Btw: I'd love to be involved in that project!!!!!!!!!!!

 

[Colin_b]

I am in the market for an extremely high capacity storage system - Ideally I want to be able to transport 6-10Tb of data off site at the end of a day and store that data for 7 years (DPA)

I need the cheapest solution for 1 shot storage of say 6Tb (lower end) at a time and ATM I am looking at Tb tape drives as they will also offer a decent physical size for storage.

Anyone else had to do something similar or does anyone have a better solution for high volume storage ?

Mark

Tape is an expensive way of doing a stuff. Needs storing in a controlled environment and regenerating every 5 years and test reading the tapes. if the data is valuable enough. Possibly 2 copies...

You need to keep drives and software that will read the tapes so you can still get at the data in 7 years.

So the total of tapes, drives, storage and maintenance works out bloody expensive overall. We were looking at a similar think, but just tape maintainance would have been around 40k a year after 7 years! Decided that it was much cheaper to reduce data volumes and only keep it for 3 years.

Using large SATA disks appears to be cheaper and more reliable than tapes, you might want to investigate that option.

 

[Ted]

I misread your original post then, I thought it was a one-off thing kept for 7 years... I didn't realise you need to transfer 10Gigs a day!


Michele

No Spinal, 10Tb!!!!!!!!!1

 

[columb]

I misread your original post then, I thought it was a one-off thing kept for 7 years... I didn't realise you need to transfer 10Gigs a day!

1Gb is not a problem at all

Cheers
Chris

 

[Colin_b]

1Gb is not a problem at all

Cheers
Chris

And 10,000 times that isn't quite 10,000 times not a problem at all

 

[Mark300SL]

Talking with the customer we are going to deliver the problem back to the police who created it in the first place

Current feeling is we will supply a "portable" 10Gb NAS for them to remove from site that we mirror in real time. They then give it back to us for the next event and we just plug it in again.
8-10hr event on a 10gb core link we can mirror in real time

How they copy the NAS and store it is not my problem, but if they insist on keeping every frame of HD video of 7 years due to data protection then I will work with them, but I've decided we are not going to do all the work for them!

 

[Sp!ke]

Using Lt03 tape technology, real throughput is 160gb an hour over fibre channel in our environment (best case).

You'd need to have two copies of the tapes so you'd need two tape libraries and probably a hot spare. Each library is six figure money and you'd probably need a couple of people managing the task forever.

The cost is huge and task very labour intensive just because of sheer volume of data.

Thinking about this a little differently, this data.. video data you say? What format is it in? An uncompressed AVI can be as high as 1gb per minute whereas an Mpeg4 could be almost lossless and be under 5mb per minute. Are you compressing the video data as much as you can already? Could the 10tb of data not e compressed to a significantly smaller figure before being archived?

Its not my field really but I happen to work for a storage company where there are some very talented storage specialists I could point you at for advice (not sales )

 

[Spinal]

No Spinal, 10Tb!!!!!!!!!1

Typo; now edited

 

[Rory]

I think people talking about Ethernet and SDLT are in the wrong ball-park here - this needs something like a library of LTO4 Fibre Channel drives.
http://h10010.www1.hp.com/wwpc/uk/e...-12324748-12324776.html?jumpid=reg_R1002_UKEN