Results 1 to 14 of 14
  1. #1
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Huge Collections Management

    Hi, I've a HUGE problem with HUGE set of data
    The problem is that I want to manage 3-4GB of data in Lists, Sets or Maps but obviously with canonical collections native types I can't.

    I've tried with Vanilla lib but that lib is not complete, Maps are not usable
    Also I've tried with -Xms command line option for manual setting the heap but this are not enough.

    Someone knows some lib like Vanilla that do direct swapping between memory and disk giving the same interfaces of List, Set and Map?

  2. #2
    rfq
    rfq is offline Member
    Join Date
    Mar 2012
    Location
    Novosibirsk, Russia
    Posts
    15
    Rep Power
    0

    Default Re: Huge Collections Management


  3. #3
    doWhile is offline Moderator
    Join Date
    Jul 2010
    Location
    California
    Posts
    1,642
    Rep Power
    7

    Default Re: Huge Collections Management

    Are your required to have it all in memory at once? A database like Derby can store the data such that you can access it at runtime without needing it all in memory at once.

  4. #4
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Re: Huge Collections Management

    Quote Originally Posted by rfq View Post
    Nice!
    Thanks!

    Quote Originally Posted by doWhile View Post
    Are your required to have it all in memory at once? A database like Derby can store the data such that you can access it at runtime without needing it all in memory at once.
    Unfortunately yes, because is a scientific software that do computations on the whole set, reading/writing databases is a nightmare for performances :(

  5. #5
    farrukh is offline Member
    Join Date
    Aug 2012
    Posts
    13
    Rep Power
    0

    Default Re: Huge Collections Management

    I have come across similar situations a few times where the data held in memory (collections or Arrays) grew more then initially anticipated. I ended up optimizing code and using some combination of in memory storage and database. One of the important thing to consider is your application reliability requirements. If you can not afford your application to fail and still want to use a library that is still in beta mode (or even any recent open source library) then make sure you write a lot of test cases.

    Also, in your post you mentioned that you tried setting -xms. This is just your initial heap size. did you try bumping up your -Xmx as well?

  6. #6
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Re: Huge Collections Management

    Quote Originally Posted by farrukh View Post
    I have come across similar situations a few times where the data held in memory (collections or Arrays) grew more then initially anticipated. I ended up optimizing code and using some combination of in memory storage and database. One of the important thing to consider is your application reliability requirements. If you can not afford your application to fail and still want to use a library that is still in beta mode (or even any recent open source library) then make sure you write a lot of test cases.

    Also, in your post you mentioned that you tried setting -xms. This is just your initial heap size. did you try bumping up your -Xmx as well?
    Also -Xmx I've tried, but nothing, information is too huge :(
    As I tell using database are out of mind, that lib is in beta but yes, with test cases and some modifications in code I can reach good performance ;)

  7. #7
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    11,826
    Rep Power
    19

    Default Re: Huge Collections Management

    If you'll pardon my French, but how the hell is that library going to be any better than a database?
    It's stored on disk, which is the primary source of delay in database queries, so I really don't see what advantage you are going to get?
    It's a database mapped onto the Collections framework!
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  8. #8
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Re: Huge Collections Management

    Quote Originally Posted by Tolls View Post
    If you'll pardon my French, but how the hell is that library going to be any better than a database?
    It's stored on disk, which is the primary source of delay in database queries, so I really don't see what advantage you are going to get?
    It's a database mapped onto the Collections framework!
    A database by definition must contain structured and organized data.
    Data that I manage are huge sets of strings, then it use unproperly the database theory.
    Then a database offer a multi-tier structure: disk<->dbms page manager<->dbms logical manager<->resultsets
    A library of that type are a structure like: disk<->library manager<->results
    With every implications in terms of performances, it uses JVM native methods instead of dbms algebra and engine.
    In my opinion a lib like that will be some order of magnitude better than dbms. Although I can't do an exhaustive comparison research... :(

  9. #9
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    11,826
    Rep Power
    19

    Default Re: Huge Collections Management

    So guesswork then.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  10. #10
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Re: Huge Collections Management

    Quote Originally Posted by Tolls View Post
    So guesswork then.
    Different schools of thought by now.
    There aren't empirical and analytical evidences that ensures that one method still best than other.
    I'd prefer remain in JVM world, without use others technologies that will make the app heavy ;)

  11. #11
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    11,826
    Rep Power
    19

    Default Re: Huge Collections Management

    That's a reasonable position.
    :)
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  12. #12
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Re: Huge Collections Management

    Quote Originally Posted by Tolls View Post
    That's a reasonable position.
    :)
    One scientist-man must be always over reasonable positions.


  13. #13
    farrukh is offline Member
    Join Date
    Aug 2012
    Posts
    13
    Rep Power
    0

    Default Re: Huge Collections Management

    claudiocarcaci , Please let us know what your findings are about the performance of JDBM3, specially if you get a chance to compare it with any DBMS for search , insert and delete time etc. I just quickly glanced over their README file and it seems like the library shares alot of same concepts and utilizes bunch of same strategies as common DBMS. Honestly, I will be surprised if this library gives better performance then any well known DBMS for large number of big records(> 1GB data)
    Last edited by farrukh; 08-28-2012 at 05:52 PM.

  14. #14
    Join Date
    Dec 2011
    Posts
    19
    Rep Power
    0

    Default Re: Huge Collections Management

    Quote Originally Posted by farrukh View Post
    claudiocarcaci , Please let us know what your findings are about the performance of JDBM3, specially if you get a chance to compare it with any DBMS for search , insert and delete time etc. I just quickly glanced over their README file and it seems like the library shares alot of same concepts and utilizes bunch of same strategies as common DBMS. Honestly, I will be surprised if this library gives better performance then any well known DBMS for large number of big records(> 1GB data)
    When I obtain detailed results I will report it there ;)

Similar Threads

  1. Huge memory allocation
    By mctommek in forum Advanced Java
    Replies: 2
    Last Post: 12-08-2011, 01:06 PM
  2. Huge Newbie Question: save so others can use
    By Zuty in forum New To Java
    Replies: 11
    Last Post: 06-16-2011, 03:29 AM
  3. Huge req on java in my company
    By g.vinod.r in forum Jobs Offered
    Replies: 2
    Last Post: 09-13-2010, 11:03 AM
  4. Storing and managing huge arrays
    By trust in forum Java Software
    Replies: 2
    Last Post: 08-22-2009, 02:47 PM
  5. Huge VB.Net conversion
    By bullstreetboy in forum New To Java
    Replies: 3
    Last Post: 04-09-2009, 01:30 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •