*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Dear all,

i think this is an important question and also a problem with other microcopy techniques like super-resolution localisation microscopty or SPIM time laps recodings. One aspect is that one might want to reanalise the videos at a later time with an improved tracking algorithm and would need the raw data for this. Lossless compression could help as well as cropping out the important part of the image. But when the amount of data created is just prohibitive one might argue that it should be easer to repeat the experiment than saving all the data. This would mean making sure there will be access to the samples and keeping the microscopes as well as knowledge how to use them... 

 

 best wishes

Andreas

 

-----Original Message-----
From: Dmitry Sokolov <[log in to unmask]>
To: CONFOCALMICROSCOPY <[log in to unmask]>
Sent: Wed, 6 Feb 2013 8:22
Subject: Re: data storage requirements for NIH records - original videos or just tracks?


*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
*****

Hi Tim,

good question! I believe what you are asking relates to the data sampling.

The scientific method is about the reproducibility of the experiments 
under the conditions given.
The Technology of Research deals with the sustainability of human 
activities:
http://confocal-manawatu.pbworks.com/w/page/48682494/Goals%20of%20Technology%20of%20Research
In your case we should probably talk about the sustainability of your 
research.

If the high data sampling rate makes your research unsustainable, it 
will not be practical to confirm your results too.
However, if your secondary data (the tracks) are reproducible, you are safe.

Geological expedition is probably a good analogy with your experiment. I 
believe that its description is satisfactory when based on the GPS data, 
physical samples and photos from the sampling sites. High definition 
satellite real-time movies would be probably useful but still not 
required. The volume and character of data must be adequate to the 
objectives of a problem. Too much of data in your raw images form the 
"noise" that is meant to be "filtered" by your particle tracking 
algorithms. This is the fundamental problem of scientific 
instrumentation as the human/nature interface.

I hope you find it useful.
Other opinions would be highly appreciated.

Published in MIAWiki:
http://confocal-manawatu.pbworks.com/w/page/63370062/Scientific%20Instrumentation%20as%20Human%20-%20Nature%20Interface

Cheers,
Dmitry

*Advanced Knowledge Management*
for *MICROSCOPY *and *Image Analysis *
------------------------------------------------------------------------
*Dmitry Sokolov*, Ph.D.
Mob: *+64 21 063 5382***
[log in to unmask] <mailto:[log in to unmask]>



06.02.2013 8:32, O'Brien III, E. Timothy ?????:
> *****
> To join, leave or search the confocal microscopy listserv, go to:
> http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
> *****
>
> Dear Microscopists-
>
> Our group has begun using a parallel microscope system to study the movement 
of fluorescent beads on cells, or in biofilms, mucus, other biological fluids.  
We then track the bead movements and generate MSD (mean squared displacement) 
curves for each bead.  Each 1 minute video at 60 FPS takes up about a gigabyte 
of data storage.  Meanwhile the tracks (position/ time) might take several kB 
for each bead.  We can take 12 videos simultaneously, so potentially we are 
generating 12 gB/minute, a terabyte every hour and 25 minutes!
>
> We believe that taking an image at the beginning of tracking, and keeping the 
tracking records would be sufficient for us to troubleshoot our data, since we 
can't possibly store the original videos.  This would let us know where the 
beads were at the beginning of the video (on the nucleus?  On the glass?) 
Signatures of "lost beads" or "stuck beads" are easily identified in control 
experiments.
>
> We are also considering other intermediate data reduction-potentially saving 
parts of the videos throughout the timecourse.  But this is going to be 
difficult to implement, and keep track of.  Moreover, the reduction is not 
nearly as high as taking one frame and keeping the tracking results.
>
> What is the community's understanding of the requirements for storing 
"original" data?  Do we need to keep full videos and spend all our budget on 
hard drives, or will just the position/time data and an index frame be enough?
>
> What other solutions does your group use?
>
> Thanks very much!
>
> Tim O'Brien
> Computer Integrated Systems for Microscopy and Manipulation
> UNC Chapel Hill, North Carolina