Results 1 to 17 of 17
  1. #1
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default Reading Binary File using java

    Hi,

    I am trying to read a binary file using java and I am getting weird results. I will explain the background a little better.

    I am working on a legacy system called Advantage Plus and it's database is called Metropolis. The Metropolis database is very similar to mainframe VSAM file (if you know about VSAM KSDS). Each database contains a 512 byte database header with some info and the actual data starts from 513th byte. Each database has a corresponding record layout in a file called copybook. The binary data file is in AIX 5.3 and I downloaded into my local PC (Intel P4) to read it.

    I am trying to read a file called lecfil.dbd and its copybook is lecfil.cpy.

    For string fields, I am getting the expected results. However, for the binary (byte) fields, I am not getting the correct results. Can some one help?

    I have attached my source code which uses the RandomAccessFile class, also attached my data file and a screenshot of actual data in the data file.

    Thanks,
    Balaji.
    Attached Files Attached Files

  2. #2
    travishein's Avatar
    travishein is offline Senior Member
    Join Date
    Sep 2009
    Location
    Canada
    Posts
    684
    Rep Power
    5

    Default

    I wonder, would it be an endian encoding issue, if the source machine is bigendian, and you are working on a little endian machine?

  3. #3
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    With the binaries there can be encoding issues a lot. With the database storing as well. Did you check those things?

  4. #4
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    Hi,

    Thanks for the replies.

    The data stored in AIX is Big-Endian format.

    However, I was able to figure out something. I took our production database lecfil.dbd and the values are below in the database for the fields that I am trying to read.

    CNO = 1 (Company Number)
    NXLOGO = 40178 (Next Logo #)

    I did convert the NXLOGO (40,178) into Binary and "stretched" it as a 3 byte field and the value is given below.

    Upper Byte Mid Byte Lower Byte
    00000000 10011100 11110010
    0 156 242

    The corresponding Decimal values are given above (below the binary values). I have coded a java program which uses the RandomAccessFile class and read the 3 byte value from the database. The sample code is given below.

    ==========================================
    byte[] logoNumber = new byte[3];
    inFile.readFully(logoNumber);
    int logoNum = 0;
    for (int i = 0; i < logoNumber.length; i++)
    {
    System.out.println("logoNumber[" + i + "] = " + logoNumber[i]);
    }
    ==========================================
    inFile is an instance of RandomAccessFile.

    The output I am getting is given below.
    ==========================================
    logoNumber[0] = -14 (242 -Lower byte) = (242 - 256)
    logoNumber[1] = -100 (100 - Mid byte) = (156 - 256)
    logoNumber[2] = 0 (0 - Upper Byte)
    ==========================================

    I could see that since a java byte can hold values only from -128 thru 127, the values are converted to negative values in circular fashion.

    So, java is able to read the binary values from DB correctly. A byte in our MET DB is an unsigned one and it can have values from 0 thru 256. Since java is not able to handle the unsigned bytes, it is interpreting the positive byte as a negative one.

    Now, I need to figure out a logic that takes these negative values, convert them to unsigned positive (may have to use short, since byte cannot handle) and then get the result value 40,178.

    Can some one tell me how to get derive this value from what java has outputted? I guess I may need to use bitwise operators, but not sure what exactly needs to be done.

    Thanks,
    Balaji.

  5. #5
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    java is not able to handle the unsigned bytes
    The problem is it extends bytes to ints spreading the sign bit. If you "AND" the int with 0XFF that will strip off the extended sign bits.

  6. #6
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    Great. Thanks for your help.

    Here is my revised code.

    ========================================== byte[] logoNumber = new byte[3];
    inFile.readFully(logoNumber);
    int logoNum = 0;
    for (int i = 0; i < logoNumber.length; i++)
    {
    logoNum = (int)logoNumber[i] & 0xFF;
    System.out.println("logoNumber[" + i + "] = " + logoNum);
    }
    ==========================================

    And the output is below.
    ==========================================
    logoNumber[0] = 242
    logoNumber[1] = 156
    logoNumber[2] = 0

    Now how do I interpret this and get the value 40,178. Do I need to convert the individidual bytes into binary, append them and convert back to decimal? or is there a better way to do it?
    ==========================================

    Thanks,
    Balaji.

  7. #7
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    Are you asking how to build an int from bytes?
    An int in Java has 4 bytes. If you OR in the bytes to their correct positions in the int, you can build the int. Do this by shifting each byte by 8 or 16 for 24 bits as required.

  8. #8
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    Hi,

    Thanks for your help. I was able to read this database lecfil.dbd successfully and able to parse all data from this DB. Please note that this database has only one record in it as it is a control record for other databases.

    I took the next example fipfil.dbd which contains multiple records and the format is below.

    DB header = 512 bytes (0 thru 511)

    Field Declaration Position
    Company Number = 1 byte 512
    FIP Number = String of 5 chars 513
    FIP Name = String of 30 chars 518
    Filler = String of 20 chars 548

    I am able to read the first record using RandomAccessFile class with out any problems. However, I am not able to read the subsequent records (Each record has a length of 56 chars).

    I tried the code inFile.seek(512+56) to position the pointer at the second record, but it doesn't seem to work.

    Any ideas?

    Thanks,
    Balaji.

  9. #9
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    What didn't work? The seek to the beginning of the next record or the reading of the next record after doing a seek?

  10. #10
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    Reading the next record after doing a seek is not working. My output is below.

    ==========================================
    1st Record:
    ========
    Read DB with file.seek(512). 0-511 bytes are DB header.
    Data Record Length is 56.
    Company Number = 1
    FIPS Number = 01001
    FIPS County Name = AUTAUGA
    FIPS Filler =

    2nd Record:
    ========
    Read DB with file.seek(512+56). Output is below. Actual DB contains company number as 1, FIP number as 01003, FIP Name as BALDWIN and Filler is blank.

    Company Number = 0
    FIPS Number =
    FIPS County Name =
    FIPS Filler =

    ==========================================

    Let me know if I am missing something.

    Thanks,
    Balaji.

  11. #11
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    You'll have to show the contents of the bytes read. Your edited text is no use.
    Show what was read for the first record and then the second.
    You haven't shown the format for the 56 byte records.
    Can you post the code that is reading and displaying the bytes from the file

  12. #12
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    Uploaded a zip file.

    The zip file contains the following components.

    1. LECRead.java ==> Class to read the DB.
    2. METRead.java ==> Supporting class for methods.
    3. fipfil.dbd ==> Actual database file (binary)
    4. fip_contents.jpg ==> Screenshots of the data of first two records.

    The record layout is below.

    MAP1 FIP'REC
    MAP2 FIP'CNO,B,1 ! Company #
    MAP2 FIP'NO,S,5 ! FIP Number
    MAP2 FIP'COUNTY'NAME,S,30 ! County Name
    MAP2 FIP'FILLER,S,20 ! Filler

    FIP'CNO ==> B,1 is 1 Byte.
    FIP'NO ==> S,5 is a string of 5 characters
    FIP'COUNTY'NAME ==>S,30 is a string of 30 characters
    FIP'FILLER ==> S,20 is a string of 20 characters

    Hope this helps.

    Thanks,
    Balaji.
    Attached Files Attached Files

  13. #13
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    Show what was read for the first record and then the second.
    Can you post the code(a few statements) that is reading and displaying the bytes from the file

  14. #14
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    This is the code that displays the first record.

    =====================================
    inFile.seek(512);
    System.out.println("\nCompany Number = " + inFile.readByte());
    System.out.println("\nFIPS Number = " + METRead.getString(inFile,513,5));
    System.out.println("\nFIPS County Name = " + METRead.getString(inFile, 518, 30));
    System.out.println("\nFIPS Filler = " + METRead.getString(inFile, 548, 20));
    =====================================

    And the output for the 1st record is below.
    =====================================
    Company Number = 1

    FIPS Number = 01001

    FIPS County Name = AUTAUGA

    FIPS Filler =

    =====================================

    The code that reads the 2nd record is below.

    =====================================
    inFile.seek(512+56);
    System.out.println("\nCompany Number = " + inFile.readByte());
    System.out.println("\nFIPS Number = " + METRead.getString(inFile,513+56,5));
    System.out.println("\nFIPS County Name = " + METRead.getString(inFile, 518+56, 30));
    System.out.println("\nFIPS Filler = " + METRead.getString(inFile, 548+56, 20));
    =====================================

    And the output for the 2nd record is below.
    =====================================
    Company Number = 0

    FIPS Number =

    FIPS County Name =

    FIPS Filler =
    =====================================

    It is supposed to display the following values.

    =====================================
    Company Number = 1

    FIPS Number = 01003

    FIPS County Name = BALDWIN

    FIPS Filler =
    =====================================

    Hope this helps.

    Thanks,
    Balaji.

  15. #15
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    Can you use some other code to read and display the data byte by byte.
    Read the data in as bytes and use a loop with the Integer.toHexString(aByte) to display the bytes.

  16. #16
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,403
    Rep Power
    25

    Default

    Look at your data file in a hex editor. The company names are at approx. x200, x400, x600

  17. #17
    pnbalaji is offline Member
    Join Date
    Jun 2010
    Location
    Dodgeville, WI
    Posts
    9
    Rep Power
    0

    Default

    That's a good catch.

    I will try to read the exact location. So, our Metropolis database is always storing the records in a 512 byte record irrespective of its actual record length.

    Thanks,
    Balaji.

Similar Threads

  1. Replies: 2
    Last Post: 11-17-2009, 04:12 PM
  2. Replies: 3
    Last Post: 05-09-2009, 11:31 PM
  3. Reading a Field in a Binary File
    By janakiram.attuluri in forum Advanced Java
    Replies: 2
    Last Post: 01-09-2008, 10:47 AM
  4. Reading Binary File and Changing data
    By janakiram.attuluri in forum Advanced Java
    Replies: 1
    Last Post: 12-21-2007, 10:10 AM
  5. reading a binary file with a RAF
    By jkurth in forum Advanced Java
    Replies: 2
    Last Post: 12-20-2007, 07:30 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •