Huge Collections Management

[claudiocarcaci]

Hi, I've a HUGE problem with HUGE set of data
The problem is that I want to manage 3-4GB of data in Lists, Sets or Maps but obviously with canonical collections native types I can't.

I've tried with Vanilla lib but that lib is not complete, Maps are not usable
Also I've tried with -Xms command line option for manual setting the heap but this are not enough.

Someone knows some lib like Vanilla that do direct swapping between memory and disk giving the same interfaces of List, Set and Map?

[rfq]

Look at https://github.com/jankotek/JDBM3

[doWhile]

Are your required to have it all in memory at once? A database like Derby can store the data such that you can access it at runtime without needing it all in memory at once.

[claudiocarcaci]

Look at https://github.com/jankotek/JDBM3

Nice!
Thanks!

Are your required to have it all in memory at once? A database like Derby can store the data such that you can access it at runtime without needing it all in memory at once.

Unfortunately yes, because is a scientific software that do computations on the whole set, reading/writing databases is a nightmare for performances :(

[farrukh]

I have come across similar situations a few times where the data held in memory (collections or Arrays) grew more then initially anticipated. I ended up optimizing code and using some combination of in memory storage and database. One of the important thing to consider is your application reliability requirements. If you can not afford your application to fail and still want to use a library that is still in beta mode (or even any recent open source library) then make sure you write a lot of test cases.

Also, in your post you mentioned that you tried setting -xms. This is just your initial heap size. did you try bumping up your -Xmx as well?

[claudiocarcaci]

I have come across similar situations a few times where the data held in memory (collections or Arrays) grew more then initially anticipated. I ended up optimizing code and using some combination of in memory storage and database. One of the important thing to consider is your application reliability requirements. If you can not afford your application to fail and still want to use a library that is still in beta mode (or even any recent open source library) then make sure you write a lot of test cases.

Also, in your post you mentioned that you tried setting -xms. This is just your initial heap size. did you try bumping up your -Xmx as well?

Also -Xmx I've tried, but nothing, information is too huge :(
As I tell using database are out of mind, that lib is in beta but yes, with test cases and some modifications in code I can reach good performance ;)

[Tolls]

If you'll pardon my French, but how the hell is that library going to be any better than a database?
It's stored on disk, which is the primary source of delay in database queries, so I really don't see what advantage you are going to get?
It's a database mapped onto the Collections framework!

[claudiocarcaci]

If you'll pardon my French, but how the hell is that library going to be any better than a database?
It's stored on disk, which is the primary source of delay in database queries, so I really don't see what advantage you are going to get?
It's a database mapped onto the Collections framework!

A database by definition must contain structured and organized data.
Data that I manage are huge sets of strings, then it use unproperly the database theory.
Then a database offer a multi-tier structure: disk<->dbms page manager<->dbms logical manager<->resultsets
A library of that type are a structure like: disk<->library manager<->results
With every implications in terms of performances, it uses JVM native methods instead of dbms algebra and engine.
In my opinion a lib like that will be some order of magnitude better than dbms. Although I can't do an exhaustive comparison research... :(

[Tolls]

So guesswork then.

[claudiocarcaci]

So guesswork then.

Different schools of thought by now.
There aren't empirical and analytical evidences that ensures that one method still best than other.
I'd prefer remain in JVM world, without use others technologies that will make the app heavy ;)

[Tolls]

That's a reasonable position.
:)

[claudiocarcaci]

That's a reasonable position.
:)

One scientist-man must be always over reasonable positions.

[farrukh]

claudiocarcaci , Please let us know what your findings are about the performance of JDBM3, specially if you get a chance to compare it with any DBMS for search , insert and delete time etc. I just quickly glanced over their README file and it seems like the library shares alot of same concepts and utilizes bunch of same strategies as common DBMS. Honestly, I will be surprised if this library gives better performance then any well known DBMS for large number of big records(> 1GB data)

[claudiocarcaci]

claudiocarcaci , Please let us know what your findings are about the performance of JDBM3, specially if you get a chance to compare it with any DBMS for search , insert and delete time etc. I just quickly glanced over their README file and it seems like the library shares alot of same concepts and utilizes bunch of same strategies as common DBMS. Honestly, I will be surprised if this library gives better performance then any well known DBMS for large number of big records(> 1GB data)

When I obtain detailed results I will report it there ;)