data file structure

[Nicholas Jordan]

I just burnt out, been working on this ten days and have 1,800 lines of code. Am beginning to spin on how to store the data. I have one file I am generating test runs from, doing decent at it. I have placed markers in the data to find the five or ten fields with regexes, slow for a few hours but arrived at a design pattern that I just replicated for each of the fields. Somewhat clumsy but it works.

All the data that matters is vewable by client, and may be any of ascii / UTF-8 \u0020 through \u007e but regexes seem to be feed-forward only and make indeterminate conditions trying to arrive at patterns that regex functions well with that are not likely to be entered by user interface. I have a specialized class I rebuilt earlier today that makes field / value access cleanly all in one place in the code base. I am considering some sort of binary character and a dedicated data file buried in the folder tree. Moments ago I wrote the following test code:

public class test
{
    private static transient final java.lang.Character ETB= new java.lang.Character('\u001c');// end of transmission block
    private static transient final java.lang.Character GS = new java.lang.Character('\u001d');// group separator
    private static transient final java.lang.Character RS = new java.lang.Character('\u001e');// record separator
    private static transient final java.lang.Character US = new java.lang.Character('\u001f');// unit seprator
    String etb = new String("// end of transmission block ");
    String gs  = new String("// group separator ");
    String rs  = new String("// record separator ");
    String us  = new String("// unit seprator ");
    public static void main(String[] args)
    {
        // C:\\Documents and Settings\\All Users\\Documents\\Reports
        try
        {
            test tes= new test();
            java.io.FileWriter fw = new java.io.FileWriter("__blanked for posting__");//
            fw.write(US);
            fw.write(tes.etb,0,tes.etb.length());
            fw.write(GS);
            fw.write(tes.gs,0,tes.gs.length());
            fw.write(RS);
            fw.write(tes.rs,0,tes.rs.length());
            fw.write(US);
            fw.write(tes.us,0,tes.us.length());
            fw.flush();fw.close();
        }
        catch(java.lang.Throwable t){System.out.println(t.toString());}
    }
}

and opened the file with a text editor. Bad Bean if people who use this open the file with a Word Processor, just the sort of thing that could happen. Ideally, I would cleanly separate the data storage from display, but if I do not make the data readable with standard text editors, crash recover becomes bloated with all the places in the code that have to allow for that moment, I would rather have a jump-gap measure that gets over the moment and leads to open ended improvement.

Not to be demeaning, user is a twit. I have to allow for this.

I do not mind writing a great deal of code if it will deal with this, weak encipherment is appealing but makes data recovery a "get it right the first time." issue.

[Steve11235]

I may not fully understand what you need to do, so if I whiff, just ignore me...

I suggest separating your fields with tab characters '\t' and data rows with new lines. I assume your user is on Windows, so that is "\r\n".

Make sure you always write out the full number of tabs, even if there is no data at the end of the line.

To recover the data, read a certain number of lines back into a String. You can read them all at once if the output file is not too large. Use String.split("\\r\\n") (escape the backslashes) to get an array of String, one for each record. On each record, use String.split("\\t") to get an array of String for each field. This is a very easy way to recover the data.

Your user can open the file using Excel. You can output a row of column headings in the first record if you want.

If you have multiple types of records, you can use the same approach, just make the first field a record type indicator, so you know what to expect when you read it back in.

I hope that helps...

[Nicholas Jordan]

I just went through an interview with client on the whole challenge, they are currently using an approach which is in the arena of your solution, I could code Java to work within the context ( I'm formulating preliminary concepting - whiffs are to be expected ) of excel, access and sql ... the ... uh ... several issues.

My thinking at the moment and when I wrote the above is to somehow disentangle data storage / recovery so far from conventional approaches that I have already commented out cr/lf from the static fields ( in my Java source code for several base classes )

To be sure, you understand what it is I am trying to do. And as well, my current "for my own use" implementation that I am already using relyed on a solution done within the framework your solution exists in.

There are performance problems with that approach, we are going to bust out of the loop entirely - I am going for some get-going code tonight that does it like java.io.DataOutput does it, eg a String has two bytes as the first field, indicating the length of the String.

What I am going for is some sort of "all binary" control fields that give data-typing, field sizes and so on,... and as a consequence of interview today with client, will just skip *all* the excel stuff and so on, including all the database based solutions, writing tabular data using JTables.

There are numerous issuse that are easy for me to overlook, gimme some whifs.