Large Scale Search

[sadronmeldir]

Hello,

I'm currently working on a program in an effort to expand my skill set, but I'm having a bit of a trouble getting pointed in the right direction.

I have about ten million sets of information, each representing a single line of conversation and storing such information as the user, the message text, a timestamp and so on. I am trying to find an efficient way to search through this data efficiently without exceeding the available memory.

The process has to be portable, so that I can easily wrap the whole program up in an installer and send it to others with relative ease.

I'm not looking for people to provide the code, but some clues would be appreciates. How best do I store millions of objects in a single collection or database? Which approaches or libraries should I use for applying efficient, complex searches to the entire 10 million sets?

I've posted this question on a few other sites because I'm new to the Java forums and I don't know which ones are more helpful, but if my question is answered elsewhere, I'll be sure to link it here.

Any advice at all would be appreciated. Thanks!

[pbrockway2]

Also posted at Sun forums.

[sadronmeldir]

Apologies! I'm new and didn't know that was taboo.

As you can plainly see, I'm someone with little forum experience. I posted on multiple sites because I did not know it was frowned upon, and since I genuinely need I wanted to reach a helpful, understanding audience who'd be patient with new forum members as they learn the ropes.

I really do need help with this problem, and I hope the community can offer advice with the same devotion they have to forum etiquette.

To simplify things, please post responses on the Sun Forums