|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] scientific computing data formats
|
Wow. Thanks for the replies.
>What about CSV isn't working for you?
CSV and gz are working perfectly fine. But to backup this data is a
nightmare because of the small files. Each file ranges from 2k to
800kb (compressed)
How are you going to be using the data?
This data can be used for viewing and doing calculations. The
calculation is basically going to seek thru the file and its fields
and perform calculations.
basically, zcat foo.csv.gz | awk -F, '{ print ($3 + $9) }'
What languages will you be using it from?
C/C++/Perl and Java (yuck)
How much data are you talking about?
As of this morning, 80TB
On Thu, Mar 26, 2009 at 9:43 PM, Michael Leone <turgon@mike-leone.com> wrote:
> Edmond Rodriguez wrote:
>> Isn't size a big factor?
>
> My, what a loaded line ... LOL
>
> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|