Killing an Ill-behaved Thread

John6715 · #1 (**permalink**) 10-02-2008, 07:32 PM

I am working on a problem wherein my code is calling to a hardware device driver (written in Java and native code) and the thread is freezing. This in turn causes my application to freeze solid, requiring us to kill the JVM and restart it. To make a long story short, this is a support nightmare and one we would rather work around if possible.

The flow looks something like this:

1. Application controller thread receives a request to perform an action: this involves hardware interaction.

2. Our hardware framework spawns a thread to service the request.

3. The thread calls into a proprietary, closed-source driver for which we have no source code and cannot make any changes.

4. Said driver can, under specific circumstances, reach a point where it freezes. This causes our thread to freeze, which in turn will cascade up and cause the application to freeze. This is not technically deadlock but has similar characteristics. Removing this deadlock is what I am trying to do, but changing an application framework is not something to do lightly.

So far we have not been able to fix this. Part of the problem is design on our part, the system architecture is fairly robust but in some cases is very susceptible to threading problems. The second half of the problem is Java's lack of a "kill no matter what" mechanism for threads.

The driver itself traps all exceptions and all method calls that might stop it. It simply catches whatever we throw at it and continues waiting: for what we do not know.

Is there maybe a special Executor implementation that could do this? Maybe one that is written in native code, bypassing the (weak) Java kill mechanisms? I know the contract of Executor is that it provides additional thread management capabilities, is there any implementation that can do this?

While we have a support ticket open with the hardware vendor, this is not the first time this issue has come up. If I can find a way to kill such a thread then I will change our framework to handle this more gracefully. Neither myself nor my team's resident hardware guru are aware of any way to do this.

masijade · #2 (**permalink**) 10-02-2008, 07:50 PM

Would it be possible to split of this device interaction into a completely separate process and communicate with it using jms or something to that effect. That way, the "main" program can kill and restart it itself whenever it gets into this "freeze" mode, without having to stop and restart the application.

masijade · #3 (**permalink**) 10-02-2008, 07:51 PM

Also, if you have "robust" system, then hopefully, you have some software (probably a cluster software) that monitors the state of the application. That software could be the one that monitors the "device" part and restarts it when necessary, thereby removing that responsibilty from the application.

John6715 · #4 (**permalink**) 10-02-2008, 10:58 PM

We cannot use multiple processes to handle thread communication. That would make this easier.

The application itself is fairly robust and we did implement a fix that prevented the lockup, but has other side effects. Ideally we could kill the thread and try again, but so far that is not possible.

That is why I was hoping for an Executor that could dip down to the native level and kill the thread even if it does not want to die.

masijade · #5 (**permalink**) 10-02-2008, 11:51 PM

Like I said, not "thread" communication. Use JMS (Java Messaging Service, I believe) or something similar. In other words the one process sends a request to the other and waits on a response.

And no, there is no safe way to "kill" a thread. And the "unsafe" way (with stop) is not likely to help you, especially if you're already having sync issues.

Edit: And, besides, I don't think killing the thread would help you in the first, as the process itself is probably what has all the locks, etc, on the device and those probably wouldn't go away simply by killing the thread that was actually performing the interaction, and so the problem would likely still exist after killing and restarting the thread.

John6715 · #6 (**permalink**) 10-03-2008, 01:01 AM

This is one application: due to constraints with how our customers use it, we cannot have multiple processes running. We have to use threads or we open ourselves up to much larger problems.

At this point if I could kill the thread it is worth a try. It may not work, but that is why we have a QA lab: to test this and document what happens under what circumstances. Unfortunately we cannot even use Thread.stop() as the inherent problem is the thread does not behave in the first place. Hopefully the hardware vendor can fix their Java driver, but that will take time, and like most vendors, they write perfect code and the problem is in the host application.

So I am back to my first question: is there some other way to kill this thread, even if I need to change how I manage its creation and use? E.g. using a different ThreadGroup or Executor? Even if I need native code to do it, there has to be a way. Maybe a third-party library that allows greater control?

masijade · #7 (**permalink**) 10-03-2008, 10:35 AM

It's not about control. It's about the fact that registers will be left open on any object it was using and will lead to other threads having incomplete objects and, therefore undefined behaviour.

But hey, if you are simply going to dismiss any suggestion anyone makes and stay doggedly by the belief that your idea is the only one that can work. More power to you.

BTW, there are plenty of applications that use more than process, and I can guarantee you have a few of them running on your home PC whenever you turn it on, so, no, I don't buy that argument. It is simply not what you want to do, or just because it is not your idea, so you dismiss it out of hand. Well, good luck then. If you want some third party thread library, Google for one.