Readline from an input stream and nothing more.

[couling]

Hi All

I'm trying to achieve something quite specific and at the moment it looks like I'm going to have to code this myself from scratch, but I wanted to run it past people first to see if they know a way to do this from standard J2SE classes.

I need a method which will:

1. Read a line of text (as a string) from an input stream
2. Ideally using any supported charset (those obtainable as a Charset object) but at minimum UTF-8
3. Return null or throw an exception if the input line is longer than a specified charicter length
4. Read no more bytes from the input stream than are required for the line. This point is important for two reasons:
   1. The bytes following the bytes for the new line charicter may be raw binary and not conform to any charset.
   2. The InputStream may block indefanately if all the bytes for a line have already been read and the readline method then tries to read more

Obveously I've considdered a BufferedReader but this doesnt come close to requirement.

Thanks for your time.

[JosAH]

I'm trying to achieve something quite specific and at the moment it looks like I'm going to have to code this myself from scratch, but I wanted to run it past people first to see if they know a way to do this from standard J2SE classes.

I need a method which will:

1. Read a line of text (as a string) from an input stream
2. Ideally using any supported charset (those obtainable as a Charset object) but at minimum UTF-8
3. Return null or throw an exception if the input line is longer than a specified charicter length
4. Read no more bytes from the input stream than are required for the line. This point is important for two reasons:
   1. The bytes following the bytes for the new line charicter may be raw binary and not conform to any charset.
   2. The InputStream may block indefanately if all the bytes for a line have already been read and the readline method then tries to read more

Obveously I've considdered a BufferedReader but this doesnt come close to requirement.

It takes a bit of programming: read single bytes from an InputStream and store them in a byte array. Stop reading when the end of line character(s) have been read or the array is full. Convert your byte array to a String using a certain decoding (e.g. UTF-8). An InputStreamReader comes to mind, wrapped around a ByteArrayInputStream for that purpose.

kind regards,

Jos

[couling]

Yer I thought I'd have to code it myself. :rolleyes:

I'm not famular with the ByteArrayInputStream. What additional benafit does wrapping my input stream in one offer?

One of the problems I'm facing is that decoding (bytes -> chars) using a Charset object looks to be something which is done on a buffer, not charicter by charicter. As mentioned in my OP this has the danger of throwing an exception because of an attempt to decode data which is not in any charicter set (let alone the one I'm trying to decode). The marker between charicter data and raw binary data lies in decoded charicter data, not directly in the binary data.

Is there anything in the specification of the InputStreamReader to suggest it will not fall foul of this?

Thanks

[JosAH]

Yer I thought I'd have to code it myself. :rolleyes:

I'm not famular with the ByteArrayInputStream. What additional benafit does wrapping my input stream in one offer?

One of the problems I'm facing is that decoding (bytes -> chars) using a Charset object looks to be something which is done on a buffer, not charicter by charicter. As mentioned in my OP this has the danger of throwing an exception because of an attempt to decode data which is not in any charicter set (let alone the one I'm trying to decode).

Is there anything in the specification of the InputStreamReader to suggest it will not fall foul of this?

If you have read the bytes in a byte array, create a ByteArrayInputStream (using that array) and wrap it in an InputStreamReader with a certain en/decoding. If the decoding fails, what are you to do but let the InputStreamReader throw its Exception?

edit: another approach would be to read your partial text file with a BufferedReader with a buffer size of one.

kind regards,

Jos

[ra4king]

You don't have to code anything by yourself. Just wrap your InputStream in a BufferedReader or a Scanner and if "line.length() > requiredLength" throw an Exception.

[couling]

No, that wont work... The over-reads are a real problem....

This isnt the protocall I'm working with but, take HTTP for an example. A response might be something along the lines of:

HTTP 1/1 200 Ok<cr><lf>
SomeHeader: SomeValue<cr><lf>
<cr><lf>
<raw data ...>

In this context you need to interpret the header (everything up to and including the last <lf>) as plain text. The following data can be a raw file (eg: jpeg data).

In HTTP this isnt a problem because everything in the header is in ASCII (according to the standard). So charicters and bytes map 1 to 1 and there is no translation between them. That is... you can read bytes as chars. The protocall I'm trying to work with doesnt have such a limitation.

Another way to view the problem is to say I need to be able to reverse map char count to byte count. When reading a line I need to read up to and including the new line charicter(s) and know precisely how many bytes that represents. (All I have to do then is buffer.. mark, rewind and skip).

[couling]

Love the idea of a bufferedreader with a size of 1. Sadly the test failed... see code and output:

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;


public class Test {
	public static void main(String args[]) throws IOException {
		BufferedReader r = new BufferedReader(new InputStreamReader(new TestInputStream(new FileInputStream("c:\\pound.txt"))),1);
		System.out.println(r.readLine());
		
	}
}

class TestInputStream extends InputStream {
	private final InputStream input;
	
	public void close() throws IOException {
		input.close();
	}
	
	public TestInputStream(InputStream input) {
		this.input = input;
	}
	
	public int read() throws IOException {
		System.out.println("read()");
		return input.read();
	}
	
	public int read(byte [] b) throws IOException {
		System.out.print("reader(");
		System.out.print(b.length);
		System.out.print(") returned ");
		int x = input.read(b);
		System.out.println(x);
		return x;
	}
	
	public int read(byte [] b, int off, int len) throws IOException {
		System.out.print("reader(");
		System.out.print(b.length);
		System.out.print(",");
		System.out.print(off);
		System.out.print(",");
		System.out.print(len);
		System.out.print(") returned ");
		int x = input.read(b, off, len);
		System.out.println(x);
		return x;
	}
	
	public void reset() throws IOException {
		System.out.println("reset()");
		input.reset();
	}
	
	public void mark(int x) {
		System.out.print("mark(");
		System.out.print(x);
		System.out.println(")");
		input.mark(x);
	}
	
	public void skip(int x) throws IOException {
		System.out.print("skip(");
		System.out.print(x);
		System.out.println(")");
		input.skip(x);
	}

}

reader(8192,0,8192) returned 13
£
[code]
It read 8192 bytes into an array of 8192 bytes doh!

[JosAH]

reader(8192,0,8192) returned 13
£
[code]
It read 8192 bytes into an array of 8192 bytes doh!

No it didn't, your array is 8192 bytes long and the read method is allowed to read 8192 bytes (starting at offset 0) but it only read 13 bytes in total.

kind regards,

Jos

[couling]

That's because the file is 13 bytes long. :p

The reader tried to read 8192 bytes, and didn't try to rewind the overread.

The file contained a single byte '£' in some form of ASCII I can't remember, followed by a new line followed by some other bytes. Those bytes are thus lost because they were not supposed to be decoded into unicode but were lost somewhere in the reader or buffer.

The more I look at this the more I convince myself I need to code manually. For utf8 I can cheat a little and assume that new line characters can be mapped directly to and from bytes safely, so I can search the incoming bytes for the end of line before I invoke the charset decoding function.

Thanks for the discussion.

[ra4king]

Reading bytes from a file get lost? Either this is something I have never heard before or you're doing something wrong. Unless something is wrong with your hardware, you can't lose bytes reading from a file. ;)

[JosAH]

Reading bytes from a file get lost? Either this is something I have never heard before or you're doing something wrong. Unless something is wrong with your hardware, you can't lose bytes reading from a file. ;)

No, those bytes aren't lost but they are in a buffer somewhere; given the scenario created by the OP:

BufferedReader r = new BufferedReader(new InputStreamReader(new TestInputStream(new FileInputStream("c:\\pound.txt"))),1);

The TestInputStream reports that 13 bytes have been read so either the InputStream reader or the Buffered reader must've requested for (at least) that many bytes; an InputStreamReader doesn't really buffer so it must've been the BufferedReader that bufffers those bytes (converted to chars). I don't understand that because it has a buffer of one single char (see its ctor). Of course the FileInputStream can buffer all it wants, those bytes remain logically in the input stream. I have to think about this; of course it can be solved by doing everything yourself (as I suggested in my first reply), but I find that clumsy ...

kind regards,

Jos

[Tolls]

No, those bytes aren't lost but they are in a buffer somewhere; given the scenario created by the OP:

The InputStreamReader holds a buffer (heapbytebuffer or something like that).
That's where the stuff has ended up.

[Tolls]

Just to clarify, the buffer is in the StreamDecoder which is in the InputStreamReader.

[JosAH]

Just to clarify, the buffer is in the StreamDecoder which is in the InputStreamReader.

Yes, but the API documentation says:

Each invocation of one of an InputStreamReader's read() methods may cause one or more bytes to be read from the underlying byte-input stream. To enable the efficient conversion of bytes to characters, more bytes may be read ahead from the underlying stream than are necessary to satisfy the current read operation.

and a simple pound sign doesn't make it buffer 13 chars ...

kind regards,

Jos

[Tolls]

Using the above code this:

System.out.println(r.readLine());

Read in the entire 2 line file I used into the sd buffer.
Debug it and watch.

[JosAH]

Using the above code this:

System.out.println(r.readLine());

Read in the entire 2 line file I used into the sd buffer.
Debug it and watch.

Yep, I debugged it and saw the InputStreamReader slorping in as much as it could; that ruins my 1 char buffer idea for the BufferedReader ... Oh well, programming it yourself takes just a few lines of code (although I find it a bit clumsy).

kind regards,

Jos

[Tolls]

It does seem odd that you can't control the underlying StreamDecoder, but there is no handle on that at all.

[JosAH]

It does seem odd that you can't control the underlying StreamDecoder, but there is no handle on that at all.

Yep, I also tried it with a Scanner object (it can read lines also) but it uses a decoder internally just as the InputStreamReader does, so the results are the same ...

kind regards,

Jos

[ra4king]

What are you trying to do that can't be satisfied by the current classes?

[JosAH]

What are you trying to do that can't be satisfied by the current classes?

Read the OP's first post; it can't be done with Readers (or more exact: anything that uses a CharsetDecoder) because the darn things always reads ahead. I'm afraid it has to be done 'manually' by using a simple InputStream.

kind regards,

Jos