I have an interesting problem and I'm not sure whether this is the right section to post it in. Anyway here goes.
I'm developing a real time packet capturing tool in Java. I'm using a third party library for capturing packets off the network interfaces. I have a thread (let's call it the listener thread) which gets packets from the 3rd party library and dumps them in an array list (buffer). I have another thread (say output thread) which reads packets from the array list and updates my swing GUI. I don't remove packets from the array list because I want to later access them to perform some analysis on user command.
The above technique works fine but there's a catch. When the application keeps running for a long time obviously it runs out of memory and crashes. So then I considered the option of writing the packets to a file once updating the UI so that later I can bring them in from the file. The problem with that is it slows down the application drastically. It becomes so slow that it cannot update the UI in real time.
Does anybody have any idea as to how to get around this problem? I don't want my application to run out of memory and I want it to be real time.
This is a classic Doctor Doctor problem. To which the only answer is:
Originally Posted by hiranya
Do not do that!
First, you can't do "real time" in Java. Its impossible. So if you really need real time, use another language. Classic example: the JVM will do garbage collection whenever it wants to. You can't control it. If the GC happens between two real time events, your Java code will fail.
its a feature.
Second, pull the data out of the buffer. Otherwise, no matter how much memory you have, you will exceed it. Again, another Doctor Doctor problem.
In general, its better to have a ring of buffers that your writer code fills up and switches to the next empty one, and then you have a second thread that reads full buffers and writes them to disk.
The biggest RAM in any computer in my house is 3GB. That is not enough to hold a movie, even if the operating system, programs, etc took zero space.
Rethink your approach.
Some pointers, depending on exact details of what you're doing:
- From your description with ArrayList, sounds like you've got one object per packet; if you have a large number of "small" packets, can you store them more compactly than this?
- agree with previous poster about "real time": ideally, you probably want something "closer to the operating system" (DLL, native library) actually capturing the packets, then a native method that interfaces between your app and the packet capturing code to pull the "latest captured packets" out of a buffer periodically; cater for the case where your Java app can't poll for packets in time and just drop them from the native buffer (with a suitable message to the app that this has happened next time it polls) -- as the previous poster says, you can't rely on things being "real time" in Java, so you need a contingency so as not to compromise system performance;
- as far as a UI is concerned, "real time" can still be every few seconds;
- don't create a Swing component per item displayed (if you're doing that) -- look at using (say) a custom list model that pulls the displayable version of your packets out "on the fly";
- buffer things so that you persist to disk in large-ish batches.