Yeah, 50-100GB is huge for a file.  A typical imaging session on our
spectral confocal (C1Si) is usually 2-6 GB, depending on spectral
density (up to 32 channels) and if large volumes are acquired.  I hope
you mean 50-100GB per imaging session, or that this is a typo and you
meant MB...  Handling 0.1 TB per file would be ugly.  Our Drobo is
good to 10 TB, but if what you are saying is correct that's only about
100 files/data sets for you.  You may want to have a look at tape
backup if you really are getting files that big.  I'd still recommend
a Drobo or other network attached storage just as a place to stash
files for processing.  Once you are done analyzing/manipulating a data
set move it to tape backup from the hard disk array.  Tapes are really
cheap, hold quite a bit, but are slow, so they are best for offline
archival.  The network attached storage (Drobo, etc) can't hold as
much as a bin full of tapes but are reasonably quick and would allow
you to work on the data.

Links on general tape backup information:

http://findarticles.com/p/articles/mi_m0BRZ/is_8_20/ai_65513193/

http://www.zdnet.com/blog/ou/are-tape-backup-systems-obsolete/267

Craig

On Fri, Jun 18, 2010 at 11:17 AM, Mario <[log in to unmask]> wrote:
> Sylvie,
> When you say "data files that are 50-100Gb," do you mean single files or
> files in aggregate on a daily basis? What does this amount to per month or
> per year? 100 GB  is easy to do especially when collecting video rate image
> sets (~ 20 min uncompressed 16-bit/pixel HD or ~200 x 128 full frame image
> stacks).
> Craig's suggestion using low power drives sounds like a very good option. I
> wonder what you do after that because with daily acquisition of that much
> data you will fill the drive(s) in short order.
> Blue ray storage disks can get you something like 50 gigs for dual layer
> double sided disks, which may be adequate. A couple of years ago Pioneer
> announced a 16 layer disk that could get 400 GB per disk.
> Sylvie, do you or anyone else know whether such a system using relatively
> non-volatile optical data encoding is available yet?
>
> Dear all
>
>
>
> We often generate data files that are 50-100Gb. I suppose the only solution
> for us it to set up a server to save/back up our images.
>
>
>
> Does anyone have any experience with very large amounts of data?
>
>
>
> Med vänlig hälsning / Best regards
>
>
>
> Sylvie
>
>
>
> @@@@@@@@@@@@@@@@@@@@@@@@
>
> Sylvie Le Guyader
>
> Live Cell Imaging Unit
>
> Dept of Biosciences and Nutrition
>
> Karolinska Institutet
>
> Sweden
>
> office: +46 (0)8 608 9240
>
> mobile: +46 (0) 73 733 5008
>
>
>
> --
>
> ________________________________________________________________________________
> Mario M. Moronne, Ph.D.
>
> [log in to unmask]
> [log in to unmask]
>