I have not seen this done in Java before, that doesn't mean that it hasn't, or that it cant be done! But i have seen this done many times in Flash, there is also a lot of support available, here is a website which gives a tutorial on magnification:
Mimicking a Magnifying Glass in Flash
This is one tutorial, there are many out there for this sort of thing.