parsing/storing large text data

[hkansal]

Hi All,

I have seen a lot of pre-cooked java forum packages on the net but I would like to code my own and hence need some support. I have expertise in databases and have designed the required schema. Now to start I would like to have some help on this issue:

In forums, eg this one, users may reply with long text inputs. What would be the best way to store them? One option is to use a Text field in the database which stores 65K characters. Another is to convert the large text to binary and store.

Pls tell if there is any other option, else how shall I about accomplishing the second one?

Thank You.

[Eranga]

I don't know there is any other way is available. Over the years I'm using Text in SQL. It's much better so far so good. Even though data converted into binary values, I think no need to convert them into binaries within our code.

What you are trying to do with this. Performance wise I don't see any issues working with Text.

[Nicholas Jordan]

Hi All, I have seen a lot of pre-cooked java forum packages on the net but I would like to code my own and hence need some support. I have expertise in databases and have designed the required schema. Now to start I would like to have some help on this issue:

Can you provide some detail as to whether these databases are commercial in nature or student work, and is the database to be loaded onto a server remotely or can you code the whole thing directly in the application. The reason I ask this is expertise in databases and would like to have some help on this issue taken together suggests to me you will be taking a pure database approch. That raises in my mind an obvious ambiguity along the lines of one can write a file very simply with code, and storing data as a string or as binary is not a paticularly difficult design issue. So it is, politely intended, what gives here. (?)

In forums, eg this one, users may reply with long text inputs. What would be the best way to store them? One option is to use a Text field in the database which stores 65K characters. Another is to convert the large text to binary and store.

Compression, through available Java classes is available but given the massive storage available at commodity pricing I would focus on "what makes sense to you" before I would look to established design for preliminary concepting. I see a lot of work in Marty Hall's Core Servlets book that may provide a better starting place than considering the matter directly, there are some issues related to scaling that may be of profound consequence later. If this will ever run on a distributed Tier-1 arena, that would be well worth the effort later.

Pls tell if there is any other option, else how shall I about accomplishing the second one?

There are plenty of options, as long as you stay with Java, the competitors are not True Beans.

[hkansal]

The database is designed to hold data in a recursive manner, self-referencing keys leading to space optimization. The doubt I had was raised because if text is saved as text itself,
1. it occupies more space.
2. there is a limit to the number of characters. Posts in a forum may exceed that.
also there is a feature in all databases to store large objects, and objects in binary form. this method i can use to also save images in the db itself(though i am not planning to do that).

Kindly advise me the good techniques to go about solving my problem.

[hkansal]

And yes, this is a student work and but the whole db can be coded in the application itself or uploaded remotely, both.