|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] scientific computing data formats
|
I don't think any RDBMS or DBMS can really help.
The UNIX filesystem is pretty fast if you know what you are looking for.
Hadoop is not what we need or want. A lot of our research is propriety
and and I not sure what hadoop can provide that we can't simply code.
On Fri, Mar 27, 2009 at 9:38 AM, Jon Bringhurst <jon@bringhurst.org> wrote:
> One word. Hadoop (http://hadoop.apache.org/core/).
>
> If you take the database route, you're going to be stuck with oracle.
> Even postgresql is generally capped at 32TB. SQL server will die at
> around 16-20TB. Don't even think about sybase or mysql.
>
> You're insane if you stick with csv files. :-)
>
> -Jon Bringhurst
>
> On Fri, Mar 27, 2009 at 12:16 AM, Mag Gam <magawake@gmail.com> wrote:
>> Wow. Thanks for the replies.
>>
>>>What about CSV isn't working for you?
>> CSV and gz are working perfectly fine. But to backup this data is a
>> nightmare because of the small files. Each file ranges from 2k to
>> 800kb (compressed)
>>
>> How are you going to be using the data?
>> This data can be used for viewing and doing calculations. The
>> calculation is basically going to seek thru the file and its fields
>> and perform calculations.
>>
>> basically, zcat foo.csv.gz | awk -F, '{ print ($3 + $9) }'
>>
>> What languages will you be using it from?
>> C/C++/Perl and Java (yuck)
>>
>> How much data are you talking about?
>> As of this morning, 80TB
>>
>>
>>
>> On Thu, Mar 26, 2009 at 9:43 PM, Michael Leone <turgon@mike-leone.com> wrote:
>>> Edmond Rodriguez wrote:
>>>> Isn't size a big factor?
>>>
>>> My, what a loaded line ... LOL
>>>
>>> ___________________________________________________________________________
>>> Philadelphia Linux Users Group -- http://www.phillylinux.org
>>> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
>>> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>>>
>> ___________________________________________________________________________
>> Philadelphia Linux Users Group -- http://www.phillylinux.org
>> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
>> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>>
> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|