Page 1 of 2 12 LastLast
Results 1 to 20 of 38
Like Tree4Likes

Thread: Converting bits to integer from a big array of bytes

  1. #1
    igorland is offline Member
    Join Date
    Jan 2013
    Posts
    89
    Rep Power
    0

    Default Converting bits to integer from a big array of bytes

    Hello. I have a large array of bytes (byte[] buffer). Inside this array, I have, say, 1000 integers encoded using 9 bits each starting from byte 6 of the array. I am somewhat confused how to do that. I tried the following:

    Java Code:
    private ArrayList<Integer> getValuesFromDataBits(int initialByte, int fieldWidth, int totalFields)
        {
            ArrayList<Integer> finalResult = new ArrayList<>();
            
            if(buffer.length >= (initialByte + (fieldWidth*totalFields)))
            {
                // Current octet serving the beginning of the range
                int currentBit = initialByte;
    		   
                for(int i=currentBit; i<(fieldWidth*totalFields+currentBit);)
                {
                    byte[] bytesTemp = Arrays.copyOfRange(buffer, i, (i+fieldWidth));
    			   
                    finalResult.add(bytesToInt(bytesTemp));
    
                    i += fieldWidth;
                }
            }
    	   
            return finalResult;
        }
    
    
    public static int bytesToInt(byte[] b) 
        {
            int result = 0;
            
            for (int i = 0; i < b.length; i++) 
            {
                result <<= 8;
                result |= (b[i] & 0xFF);
            }
            
            return result;
        }

    However, this is not working as expected. It gives me large values (like -1967037776), although, I know, these should be pretty small positive integers. Also, when I have integers encoded using 4 or 5 bits, the array returns completely empty.

    Thank you!

  2. #2
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    Can you reduce the size of the input data so it is easier to debug?
    Then execute the program, copy the output here and add some comments saying what is wrong with the output and show what the output should be.
    If you don't understand my response, don't ignore it, ask a question.

  3. #3
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    Quote Originally Posted by igorland View Post
    Hello. I have a large array of bytes (byte[] buffer). Inside this array, I have, say, 1000 integers encoded using 9 bits each starting from byte 6 of the array. I am somewhat confused how to do that.
    What do you mean when you say encoded using 9 bits? Are you just saying you have 9 bit
    integer values? And when you say you are confused how to do that, what to you mean by "that?"

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  4. #4
    igorland is offline Member
    Join Date
    Jan 2013
    Posts
    89
    Rep Power
    0

    Default Re: Converting bits to integer from a big array of bytes

    Quote Originally Posted by jim829 View Post
    What do you mean when you say encoded using 9 bits? Are you just saying you have 9 bit integer values?
    Jim. If this is the binary file I have (where numbers represent bits):
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 31 32

    My integers should be encoded as follows.

    Case when there are 3 integers encoded using 9 bits each:
    Integer 1: bits from 1 to 9
    Integer 2: bits from 10 to 18
    Integer 3: bits from 19 to 27

    Case when there are 6 integers encoded using 5 bits each:
    Integer 1: bits from 1 to 5
    Integer 2: bits from 6 to 10
    Integer 3: bits from 11 to 15
    Integer 4: bits from 16 to 20
    Integer 5: bits from 21 to 25
    Integer 6: bits from 26 to 30


    Quote Originally Posted by jim829 View Post
    And when you say you are confused how to do that, what to you mean by "that?"
    Getting integers from the bits using the above schemes.

    Thank you.

  5. #5
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    The way to extract a group of bits from an int is to
    AND the int with a mask of bits that selects the desired group of bits:
    for example to get the right most 3 bits AND with 0B111
    then shift the int's contents to the right as needed.
    No shift needed for rightmost 3 bits
    igorland likes this.
    If you don't understand my response, don't ignore it, ask a question.

  6. #6
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    Okay. I am assuming the low order three bits are not used. So first, right shift them off. Use >>> so you don't extend
    the sign bit. Then start shifting right 9 bits and ANDing them with 0x1FF (similar to what Norm said).

    Note. If you do it this way I suggested, the integers will be reversed (first 3, then 2, and then 1). You can also
    shift left, but then you need a different mask and also have to adjust the result so the bits are in their proper, rightmost
    position. But the values will be in order (1, 2, and 3).

    Regards,
    Jim
    igorland likes this.
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  7. #7
    igorland is offline Member
    Join Date
    Jan 2013
    Posts
    89
    Rep Power
    0

    Default Re: Converting bits to integer from a big array of bytes

    Thank you, guys. This is helpful.

    I have found this formula to calculate the bit mask. I am wondering if this is the best way to get it (param = number of bits).

    Java Code:
    int param = 9;
    int nMask = (1 << param) - 1;
    Output:
    param = 9;
    NMASK HEX: 1ff
    NMASK BIN: 111111111

    param = 9;
    NMASK HEX: 7
    NMASK BIN: 111

    Cheers.

  8. #8
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    It's one way. But assuming param doesn't need to change during the execution of the program, you should
    make it final. That way, the mask will be computed at compile time. Minor detail actually, but it
    is a tad more efficient.

    Regards,
    Jim
    Last edited by jim829; 11-07-2016 at 02:53 PM. Reason: clarification
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  9. #9
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    is the best way
    Depends on the requirements. Another way is to have an array of masks.
    If you don't understand my response, don't ignore it, ask a question.

  10. #10
    igorland is offline Member
    Join Date
    Jan 2013
    Posts
    89
    Rep Power
    0

    Default Re: Converting bits to integer from a big array of bytes

    Quote Originally Posted by jim829 View Post
    But assuming param doesn't need to change during the execution of the program, you should make it final.
    Thanks for the suggestion, Jim, but the binary data includes chunks with integers represented by 4,5,9, etc. bits.

    Many thanks to both of you, gentlemen. I have one more thing to clarify before I can test the code. As I said, the binary data is an array of bytes with several thousands of components. Some components, say 0 to 31, represent integers encoded using full bytes. Then, the bytes in the array from 32 to N represent integers encoded only with 4 or 5 or 9 bits.

    Something like this:

    Java Code:
    {...1 2 3 4 5 6 7 8,   9 10 11 12 13 14 15 16,   17 18 19 20 21 22 23 24,   25 26 27 28 29 30 31 32 ...}
        --------- ------------  -------------- ----------------- ----------------  --------------
          Int 1        Int 2           Int 3         Int 4           Int 5            Int 6

    In this case, I cannot simply iterate through my array of bytes as some bits will remain unprocessed. What is the best solution in this situation? Convert the array of bytes to a binary string?

    Thank you so much for bearing with me.

    Igor
    Last edited by igorland; 11-08-2016 at 03:44 AM.

  11. #11
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    I'm not sure I understand your diagram notation.
    Does this list describe how to unpack the integer values: (1-32 based vs 0-31 based)
    int1 = i1(1-5)
    int2 = i1(6-10)
    int3= i1(11-15)
    int4=i1(16-20)
    etc
    to get 6 int values from a 32 bit int

    some bits will remain unprocessed.
    Can you explain what bits you are talking about?
    If you don't understand my response, don't ignore it, ask a question.

  12. #12
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    I would also be interested in understanding how you determine which mask to use in 32 thru N? In other words,
    as you progress thru the list, how to you determine the proper encoding of each group of bits.

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  13. #13
    igorland is offline Member
    Join Date
    Jan 2013
    Posts
    89
    Rep Power
    0

    Default Re: Converting bits to integer from a big array of bytes

    Quote Originally Posted by Norm View Post
    I'm not sure I understand your diagram notation.
    Does this list describe how to unpack the integer values: (1-32 based vs 0-31 based)
    int1 = i1(1-5)
    int2 = i1(6-10)
    int3= i1(11-15)
    int4=i1(16-20)
    etc
    to get 6 int values from a 32 bit int
    Norm, you are correct.

    Quote Originally Posted by jim829 View Post
    I would also be interested in understanding how you determine which mask to use in 32 thru N? In other words, as you progress thru the list, how to you determine the proper encoding of each group of bits.
    Jim, this information is included in the data. For example, byte 1 shows number of bits that are used to encode the first group of integers, byte 2 shows the number of bits that are used to encode the second group of integers, byte 3 -- third group of integers; byte 4 -- how many bits are used for the first group; byte 5 -- how many bits are used for the second group; byte 6 -- how many bits are used for the third group. This is followed by the bits that contain the actual values. This is very schematic. In case you are interested, I am trying to decode GRIB: data format commonly used in meteorology to store historical and forecast weather data.

    I am thinking that the following approach could be used (has not tried this code myself yet):

    Java Code:
    // Number of bits used to encode one integer in the current chunk
    int bitWidth = 5;
    
    // Total number of bits used to encode integer values in the current section of the data (array of bytes)
    // byte[] is the array of bytes that contains the current chunk of data. In my previous example, byte.length = 4
    int numberOfBits = byte.length * 8; // = 32
    
    // Number of integers encoded in the current chuck of data
    int numberOfValues = 6
    
    // Bits at the end that must be shifted to the right since they are added only to fill up the byte border
    int extraBits = numberOfBits % (bitWidth*numberOfValues); // = 2
    
    // Shift right extra bits
    byte = byte >>> extraBits // Shifting 2 bits right
    
    // Get mask
    int nMask = (1 << bitWidth) - 1;
    
    // Get values
    for (int i = (numberOfBits-extraBits); i=0;)
    {
        // Get integer using the mask of bits
        int currentInt = byte & nMask;
     
        // Shift the array of bytes to the right by the number of bits
        byte >>> bitWidth;
     
        System.out.println("Current integer extracted: " + currentInt);
     
        // Iterate
        i -= numberOfBits;
     
    }
    
    // Code to reverse the above values
    Any comments, guys? Thanks a lot for your help!

  14. #14
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    I'm still confused about your byte numbering and groups. Sorry for being so dense. I looked up GRIB and it looks like there are a variety of possible
    formats.

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  15. #15
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    I'm still confused about your byte numbering and groups
    Me too.
    Can you post a simple list of how the bits in the input get unpacked into int values? Something like I started in post#11
    If you don't understand my response, don't ignore it, ask a question.

  16. #16
    igorland is offline Member
    Join Date
    Jan 2013
    Posts
    89
    Rep Power
    0

    Default Re: Converting bits to integer from a big array of bytes

    Sorry for the confusion. The actual data packing is pretty complicated, but I will try to simplify it for your. Imagine I need to extract 10 integers, each of which is encoded using 9 bits; and 10 integers, each of which is encoded using 5 bits.

    The data will be packed as follows:

    • Octets (bytes) 1 to 6: Original unreferenced values. Each value is 2 octets long. I have extracted these, no need for help.
    • Bit 49: Shows me how many bits are used to encode integers in the first group (=9)
    • Bit 50: Shows me how many bits are used to encode integers in the second group (=5)
    • Bits 51-56: Set to 0 to end on the octet border.
    • Bit 57: Shows me how many integers are there in the first group (=10)
    • Bit 58: Shows me how many integers are there in the second group (=10)
    • Bits 59-64: Set to 0 to end on the octet border.
    • Bits 65-155: 10 integers of the first group, each of which are encoded with 9 bits
    • Bits 156-206: 10 integers of the second group, each of which are encoded with 5 bits.
    • Bits 207-208: Set to 0 to end on the octet border.


    This is schematically how values are encoded. The above code is not working since I cannot perform bitwise operations on the array of bytes. I need to convert them to integers. But how do I do it if an integer encoded with 9 bits occupies two bytes. For example, the first integer in group one occupies bits 65 to 73: 8 bits from one byte and one bit from the second byte; whereas the second integer occupies 7 remaining bits in the second byte and 2 bits in the third byte, etc. Should I convert the array of bytes to a string of binary sequence first?

    Hope, it is a bit clearer now.
    Last edited by igorland; 11-10-2016 at 03:26 AM.

  17. #17
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    I need to extract 10 integers, each of which is encoded using 9 bits; and 10 integers, each of which is encoded using 5 bits.
    Where are the 9 bits that are to go into each integer?
    Are the contents of the input bytes considered as a continuous string of bits?
    B0(0-7)+B1(0) << 9 bits using bits from 2 bytes: B0 and B1
    B1(1-7)+B2(0-1) << 9 bits using bits from 2 bytes: B1 and B2
    etc

    Some of the statements are confusing:
    Bit 57: Shows me how many integers are there in the first group (=10)
    1 bit has 2 values? How can it say the number of integers? What does a value of 0 mean? A value of 1?
    If you don't understand my response, don't ignore it, ask a question.

  18. #18
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    Unfortunately, no. Here is what is confusing.


    • Bit 57: Shows me how many integers are there in the first group (=10)
    • Bit 58: Shows me how many integers are there in the second group (=10)
    • Bits 59-64: Set to 0 to end on the octet border.


    You are using a single bit to describe an encoding length and a single bit to describe a group length.

    A bit is either on or off. So how does a single bit show a group length?

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  19. #19
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    6,226
    Rep Power
    13

    Default Re: Converting bits to integer from a big array of bytes

    I tell you what. I am trying to see the encoding for my own curiosity but not really helping you solve your problem.

    I suggest you read the data into a BitSet class. It is a class of bits of arbitrary length. It has a rich set of bit
    manipulating methods. It may facilitate solving your problem. One of the problems I gather is how to handle
    encodings which overlap normal primitive boundaries (e.g. byte, short, int, long). BitSet should help with that.

    BitSet (Java Platform SE 8 )

    I also found this online. I don't know how useful it is but it looks like the source code is there.
    http://jgrib.sourceforge.net/

    Regards,
    Jim
    Last edited by jim829; 11-10-2016 at 04:11 AM.
    igorland likes this.
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  20. #20
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    20,001
    Rep Power
    33

    Default Re: Converting bits to integer from a big array of bytes

    Here's my roll-your-own solution idea:
    assuming there is an array of bytes, there are NumberOfBytes*8 bits numbered from 0 to NumberOfBytes*8-1
    Write boolean method that returns true if the bit at index idx is on

    To copy N bits from the byte array to the loworder end of an int in an array of ints:
    set BitIdx = starting bit to test
    loop through the int array using intIdx as index
    set ORmask for the Nth bit (eg 0b10 for the N=2)
    loop for the N bits in the int at intIdx
    if bit at BitIdx set, OR ORmask with int at intIdx
    shift OSRmask right 1
    increment BitIdx
    end loop for N bits
    end loop for int array
    igorland likes this.
    If you don't understand my response, don't ignore it, ask a question.

Page 1 of 2 12 LastLast

Similar Threads

  1. Converting strings to bits/bytes and vice versa
    By AlexGraal in forum New To Java
    Replies: 18
    Last Post: 02-23-2014, 11:28 PM
  2. Reading/Writing Integer Bytes
    By AndrewM16921 in forum New To Java
    Replies: 1
    Last Post: 04-03-2011, 11:56 PM
  3. Converting a byte to individual bits
    By Aaron_Sharp in forum New To Java
    Replies: 18
    Last Post: 01-06-2011, 09:09 AM
  4. Help with random bits array
    By AnimeKitty in forum New To Java
    Replies: 11
    Last Post: 07-30-2010, 04:37 AM
  5. Replies: 1
    Last Post: 04-10-2010, 04:37 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •