Results 1 to 15 of 15
Like Tree1Likes
  • 1 Post By SurfMan

Thread: Fat Jar Runs Faster than Jars on Classpath?

  1. #1
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,963
    Rep Power
    8

    Question Fat Jar Runs Faster than Jars on Classpath?

    Hey all,

    My project is pretty standard: we have a jar file that includes our code, which relies on a bunch of library jars that need to be on the classpath.

    We've been deploying as a signed applet using JNLP, keeping the library jars intact and just adding them to the classpath. Our JNLP looks like this:

    Java Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <jnlp spec="6.0+" codebase="https://path.to.our.codebase" href="ThisFile.jnlp">
    <information>
    <title>Our Title</title>
    <vendor>Us</vendor>
    </information>
    <security><all-permissions/></security>
    <resources>
    <j2se version="1.6+" href="http://java.sun.com/products/autodl/j2se" />
    <jar href="https://path.to.our.jar" main="true" />
    
    <jar href="https://path.to.lib1.jar"  />
    <jar href="https://path.to.lib2.jar"  />
    <jar href="https://path.to.lib3.jar"  />
    </resources>
    
    <applet-desc 
    name="NAME"
    main-class="path.to.our.AppletClass"
    width="1200"
    height="700"
    align="center"
    >
    <param name="param1" value="value1"/>
    <param name="param2" value="value2"/>
    <param name="param3" value="value3"/>
    <param name="param4" value="value4"/>
    <param name="param5" value="value5"/>
    
    </applet-desc>
    </jnlp>
    (I've replaced the Strings with anonymous stuff to keep any corporate bean counters happy, but the point is that this works fine)

    This has worked fine on our machines. But our clients are experiencing unusable slowness, and some profiling shows things that take 2 seconds for us take over 20 seconds for them. And it's not one specific thing that's slower; *everything* is slower.

    We had an older version of the project that worked okay, and one of the differences is that the old version was deployed as a fat jar instead of as multiple jars (we switched to multiple jars to improve download speeds when we update our code as the library jars don't have to be redownloaded).

    Through an act of desperation, we tried re-exporting as a fat jar (using the "export as runnable jar" option from eclipse and then "package libraries into jar"), and through some strange dark sorcery, that actually worked. Response time was completely fine, and in fact it's even faster for us (from 2 seconds to less than a quarter second).

    The code is the exact same. Nothing in the code has changed. The JNLP is the same, except the library jars are now inside the main jar.

    The client machines are running Java 6, but the slowness is seen on multiple operating systems. We're running Java 7 and don't see as much slowness as them, but going to the runnable jar definitely increases the speed for us as well. I'm baffled, and I can't find anything on google.

    Has anybody seen anything like this in their own projects?

    Can anybody explain why code exported using the "export as runnable jar" and "package libraries into jar" runs faster than code exported using library jars on the classpath? I could see maybe the ClassLoader startup taking longer, but this is reliably slower during every step of the application, and no new classes are being loaded once it starts up.

    Can anybody explain why it runs *so* much slower on Java 6 than Java 7? We do get a slight speedup with the runnable jar, but the client machines are unusably slow.

    Color me confused.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  2. #2
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,014
    Rep Power
    20

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Not sure where the jar files are cached, but I would bet it's down to the caching.
    How have you marked the jar dependencies in the jnlp file?
    There may be something there that indicates some sort of lazy loading?
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  3. #3
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,963
    Rep Power
    8

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Thanks for the reply.

    I haven't marked the dependencies with any special tags; what you see in the above jnlp file is a direct copy of what we're using.

    I believe Java caches them in a temp directory somewhere, but we're using whatever the default is without any special settings.

    The whole program runs slowly, not just the beginning when classes are still being loaded. Even if the program is just sitting there, things like opening a file menu will take forever. And once the program is up, it should be done loading the classes from the jars.

    And we aren't doing anything on the EDT, and the same exact code works fine as long as it's packaged up in a runnable jar. It's just when the library jars are separated that we see the slowness.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  4. #4
    SurfMan's Avatar
    SurfMan is online now Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    947
    Rep Power
    2

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Maybe totally unrelated, but possibly worthwile checking out: a virusscanner. In my experience, especially McAfee is notorious for unpacking all jars and checking each and every classfile. Try to disable the AV and see if it still runs slow.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  5. #5
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,963
    Rep Power
    8

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    It seems very likely that an antivirus would be running, since the client is in a pretty secure location. I also doubt we'll be allowed to disable the scanner. However, shouldn't performance go back up once the virus scanner was done sniffing around? And why would deploying as a single jar be *so* much better?

    I know asking about the behavior of a theoretical virus scanner is a bit too broad, but I'm really puzzled here.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  6. #6
    SurfMan's Avatar
    SurfMan is online now Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    947
    Rep Power
    2

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Have you used Process Explorer or something similar to see which process is eating up CPU at the slow times? (Windows Sysinternals: Documentation, downloads and additional resources)
    gimbal2 likes this.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  7. #7
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,963
    Rep Power
    8

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    That's a great suggestion, thanks. I've never heard of that before!

    I passed your tip along to our guy in the field. I'll report back when I have more info.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  8. #8
    SurfMan's Avatar
    SurfMan is online now Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    947
    Rep Power
    2

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Quote Originally Posted by KevinWorkman View Post
    That's a great suggestion, thanks. I've never heard of that before!

    I passed your tip along to our guy in the field. I'll report back when I have more info.
    Here's the exact link to the tool I mentioned: Process Explorer
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  9. #9
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,014
    Rep Power
    20

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    You could also try Fiddler to see whether there is any networky thing going on.

    Since you see a speed difference in your environment you could use that to see whether there is a difference post-launch in how the fat- and multi-jar versions work.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  10. #10
    gimbal2 is offline Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    4,029
    Rep Power
    6

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    You know this is not the first time I have actually seen this exact problem being mentioned relating to webstart/JNLP... Now I have to dig in my memory where I saw that because there was a really nice technical discussion attached to it. I think it was on OTN, I'll see if I can dig something up with a forum search.

    edit: well plenty of similar to identical problems but not a single answer or hint so far :/
    Last edited by gimbal2; 04-09-2014 at 10:40 AM.
    "Syntactic sugar causes cancer of the semicolon." -- Alan Perlis

  11. #11
    jashburn is offline Senior Member
    Join Date
    Feb 2014
    Posts
    219
    Rep Power
    1

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    I recommend VisualVM. It comes with the JDK (and so your client will need to install it if they haven't already.) It has a nice overview monitor GUI, and you can dive in to check heap, thread and CPU usage. For performance problems such as the one you described they're usually either CPU- or IO (network)-bound. For CPU the CPU profiler will give you everything you need. For IO you can find out if there is a pattern of running threads that do not look right, e.g., threads that are supposed to run intermittently appear to run all the time, or threads that are unexpectedly blocked.

    Another tool that you can try is jvmtop - Java monitoring for the command-line - Google Project Hosting. It is like Unix/Linux top but for the JVM. It can operate at a high level (it gives you a view of all the JVMs running in the machine) or at a low level (threads running within a JVM).

    Speaking of monitoring at a high-level, a case that I've come across before at my previous employer was a performance problem with an application that was (much) later found that there was absolutely nothing wrong at all with the application. The problem was caused by a 3rd party application running in the machine interfering all other applications, and so I wouldn't discount antivirus being a possibility.

    In general, investigation into performance problems can go high-to-low or low-to-high. An example of high-to-low would be:
    1. Check on the machine level memory, CPU and IO usage, and get a view of what other processes are also running.
    2. If indications point to your application's JVM process, check on the JVM level.
    3. If needed, you can drill in deeper into the application itself using its logging mechanism.

    For #3, I'm pretty sure for Web Start applications you an view trace information on the client-side Java Console. Once the Java Console is launched, you can type 1-5 on the console to control its verbosity. E.g., trace level 2 will set the console to display all network accesses being made by the application. Also, all System.out and System.err printlns from the application will come up in the console, and so this can be used for application logging. Finally, you can set up the client-side JRE to write the console output into a file for persistent logging.

    Good luck!

  12. #12
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,963
    Rep Power
    8

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Thanks for the replies, guys.

    We haven't been able to test in the field yet, but our own tests reveal that the antivirus might be the culprit: using ProcessExplorer, we see that the antivirus with the single runnable jar takes .04 of the cpu usage, and with the multi-jar version it spikes to .3 of the cpu usage. This is still pretty small, but then again we don't see the crazy slowdown that the client does. I'm curious to see what goes on at the client site.

    I don't think anything is going on with the network. That was actually our first guess too, but the only network thing that happens is a file gets downloaded once a minute. Profiling reveals that the download takes less than a second, and other processes are taking much longer.

    The file we're downloading is a serialized file, and after we deserialize it, we pass that Object around. Hmmm, does deserialization do lazy loading by default? I want to say not, but if it does, that could be another culprit... (Edit: Nevermind, if that was the culprit, it would be slow in both cases, not just from the multiple jar setup)


    Gimbal, I'd be *very* curious to see that discussion if you can find it. My googling hasn't revealed much.

    Jashburn, I'm very familiar with the Java console, and we've done some initial profiling of our own. The problem is that there isn't a single aspect of the program that's slowing down, it's everything. Every single piece of the program is taking over 10 times longer when we run it from multiple jars instead of from a runnable jar. This leads me to believe that the problem isn't inherently code-based, but more to do with the system itself. It's just strange that the same exact code works fine as long as it's in a fat jar.
    Last edited by KevinWorkman; 04-09-2014 at 03:04 PM.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  13. #13
    gimbal2 is offline Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    4,029
    Rep Power
    6

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    The trouble is that it harkens back to the days of the good old OTN forum, now I have to deal with the new forum shit and that is a bit demotivating...

    does deserialization do lazy loading by default? I want to say not, but if it does, that could be another culprit...
    Pardon my ignorance, but lazy loading of what exactly?
    "Syntactic sugar causes cancer of the semicolon." -- Alan Perlis

  14. #14
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,963
    Rep Power
    8

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Quote Originally Posted by gimbal2 View Post
    The trouble is that it harkens back to the days of the good old OTN forum, now I have to deal with the new forum shit and that is a bit demotivating...
    I used to be a frequent visitor to the Sun forums, but when Oracle took them over, they became unusable. Sadly, many of those discussions have been lost to time. I appreciate your search efforts though!

    Quote Originally Posted by gimbal2 View Post
    Pardon my ignorance, but lazy loading of what exactly?
    That was mostly a passing thought, and I don't think it's the issue here. But to use a generic example, let's say I serialize an object that contains 1000 different variables, and let's say a List that contains 1000 other objects. When I deserialize the file, is *all* of that data loaded into the object, or is it only loaded into the object when I actually access it in the code for the first time? Are all 1000 variables loaded from the file, or does the code have to go back and load from the file every time a variable is referenced?

    I am pretty sure that no lazy loading takes place, at least by default (we're just using standard deserialization). And even if there was lazy loading, that would be a problem in both versions, not just the version that uses multiple jars.

    I was going to edit my post and delete the bit about deserialization because I don't think that plays a role here at all, but I figured I'd keep it in for posterity.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  15. #15
    gimbal2 is offline Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    4,029
    Rep Power
    6

    Default Re: Fat Jar Runs Faster than Jars on Classpath?

    Ah, no then there is no lazy loading taking place. Deserialization is quite dumb.

    The other way around - serialization - is more interesting. A few years back I was working on some tech that used Hibernate for ORM persistence and those objects were serialized to if need be. You have to realize that during serialization ALL properties will be touched, including the ones that are lazy loaded from the database. So as soon as you serialize an ORM entity it will be loaded entirely from the database. That's a good source of performance problems, but that's besides the point.

    Then the day came around that the entire application just grinded to a screeching halt. It took a long time before we figured out why: one of the ORM entities contained a very useless mapping that pulled like a 150000 records from the database. It just so happened that due to one unrelated change (a configuration check box was checked), serialization of that object would then suddenly be triggered, the serialization would touch the useless mapping property which would then start to tank the database into memory under water. Ouch ouch ouch.
    "Syntactic sugar causes cancer of the semicolon." -- Alan Perlis

Similar Threads

  1. Replies: 1
    Last Post: 05-03-2011, 02:28 AM
  2. Classpath with two jars not working
    By asdfjklö in forum New To Java
    Replies: 1
    Last Post: 02-03-2011, 10:02 AM
  3. Which is faster?
    By Dan0100 in forum New To Java
    Replies: 19
    Last Post: 09-26-2010, 09:53 PM
  4. Replies: 0
    Last Post: 08-19-2008, 04:44 PM
  5. Adding JARs to the classpath at runtime?
    By johann_p in forum Advanced Java
    Replies: 1
    Last Post: 06-26-2007, 10:42 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •