|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] scientific computing data formats
|
One word. Hadoop (http://hadoop.apache.org/core/).
If you take the database route, you're going to be stuck with oracle.
Even postgresql is generally capped at 32TB. SQL server will die at
around 16-20TB. Don't even think about sybase or mysql.
You're insane if you stick with csv files. :-)
-Jon Bringhurst
On Fri, Mar 27, 2009 at 12:16 AM, Mag Gam <magawake@gmail.com> wrote:
> Wow. Thanks for the replies.
>
>>What about CSV isn't working for you?
> CSV and gz are working perfectly fine. But to backup this data is a
> nightmare because of the small files. Each file ranges from 2k to
> 800kb (compressed)
>
> How are you going to be using the data?
> This data can be used for viewing and doing calculations. The
> calculation is basically going to seek thru the file and its fields
> and perform calculations.
>
> basically, zcat foo.csv.gz | awk -F, '{ print ($3 + $9) }'
>
> What languages will you be using it from?
> C/C++/Perl and Java (yuck)
>
> How much data are you talking about?
> As of this morning, 80TB
>
>
>
> On Thu, Mar 26, 2009 at 9:43 PM, Michael Leone <turgon@mike-leone.com> wrote:
>> Edmond Rodriguez wrote:
>>> Isn't size a big factor?
>>
>> My, what a loaded line ... LOL
>>
>> ___________________________________________________________________________
>> Philadelphia Linux Users Group -- http://www.phillylinux.org
>> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
>> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>>
> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|