I havent been on such cases but I have seen JBoss with cluster mode able to discover all the machines running JBoss in the LAN. They used JGroups for the multicast networking you might get something from JBoss or JGroups codes both are avialable in Jboss.org.
Multicast ping can be one thing you might want to research on. Or regular unicast ping with generated ip for local network can be other bad* option.
For traffic
SNMP4J - Free Open Source SNMP API for Java might help .. I am not sure if you can use packet sniffers for this purpose as well
WinPcap Network Tools and Links
Lists some of the java port of winpcap.
These are just suggestions, I wish other will contribute as well.