to log the very fast data source

[jerry_nf]

HI, all,

I am writing a client logging program to log the incoming UDP data frames.

It is very fast, 1000 frames/second coming over.


I am using the following mechanism to read the UDP data:

But I find it seems this mechanism cannot catch up the speed of the data source sometime, there are always data frames missing. And, the more data frames I logged, the more possible frames getting lost. I can see the gap of the sequence numbers between 2 frames from the output error messages.

When I use the same software to log the 1 frame/second data source, it is fine.

Any suggesting to improve this ?

Thanks,
Jerry

//$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $$$$$$$$
public class Data_Listener extends UDP_Listener {

private static long myseq = 0L;

public ProcessData_Listener(UDP_Client client)throws IOException {

super(client);
}

@Override
public void run() {
// TODO Auto-generated method stub
while( true){


Read_Data();

}//end of while

}


private void Read_Data() {


data = new byte[DATA_LEN];
for (int i =0; i<data.length; i++)
data[i] = 0;

try {
// read data from message
data = udpClient.readMsg(data, DATA_LEN);

// read sequence number from data frame
long seq = readSeq(data);

// check any data frames missing
if (seq != myseq + 1 && myseq != 0) {
System.out.println("ERROR: -- data received, seq: " + seq);
System.out.println("ERROR: -- previous seq: " + myseq);
}

myseq = seq;



// logging process
.....


} catch (IOException e) {

e.printStackTrace();
}

}

}

[mrmatt1111]

"Lacking reliability, UDP applications must generally be willing to accept some loss, errors or duplication."

User Datagram Protocol - Wikipedia, the free encyclopedia

[jerry_nf]

yes, you are right. UDP is connectionless protocol.

But, in this case, I used WireShark to catch the incoming data frames. No one missing.

So, I think the problem is in my java client.

[toadaly]

Anything you insert in your processing loop that slows it down, including logging, is going to cause you to fall behind. There's a sort of uncertainty principle at play here as a result.

If you want to minimize lost packets, take everything out of the read loop except the read itself:

- pre-allocate a cache of byte arrays big enough to read the packets and use those rather than creating them new

- once you've read the byte array, put it into a queue to be processed in one or more other threads to readSeq and log, rather than doing those steps in the same thread