Results 1 to 6 of 6
  1. #1
    RameshIyerV is offline Member
    Join Date
    Jul 2013
    Posts
    3
    Rep Power
    0

    Unhappy Need help in analyzing some contentions we observe in the Lucene execution

    Hi All,

    I need some help in analyzing some contentions we observe in the Lucene execution.

    We are supporting the Sterling 9.0 fulfillment application and it uses Lucene 2.4 for catalog search functionality.

    ---The Issue----
    This system is Live in production since Nov 2012 and only recently (Mid June 2013) our application is forming stuck threads during lucene invocations, this causes our application to crash.

    This occurs 2 - 3 times a week, on other days we see spikes of very slow performance on the exact places that causes stuck threads.

    ---The research---
    We have validated that the data or the usage has not grown between Jan 2012 & now.

    We took snapshot of the code execution (through visual VM) and for slow running treads we validated that too much time is spent at certain spots (these very same spots appear in the stack trace of the stuck threads).

    ---Help needed---
    If you can guide me on what kind of contentions (heap, IO, Data, CPU, JVM params) can cause such a behavior it will really help.


    ---Lucene Invocation contentions observed---
    (We find stuck threads & slowness at the following spots, ordered in the order of severity [high to low])
    1. java.io.RandomAccessFile.readBytes(Native Method)
    java.io.RandomAccessFile.read(RandomAccessFile.jav a:338)
    org.apache.lucene.store.FSDirectory$FSIndexInput.r eadInternal(FSDirectory.java:596)
    org.apache.lucene.store.BufferedIndexInput.readByt es(BufferedIndexInput.java:136)

    2. org.apache.lucene.index.TermBuffer.toTerm(TermBuff er.java:122)
    org.apache.lucene.index.SegmentTermEnum.term(Segme ntTermEnum.java:167)
    org.apache.lucene.search.FieldCacheImpl$10.createV alue(FieldCacheImpl.java:373)
    org.apache.lucene.search.FieldCacheImpl$Cache.get( FieldCacheImpl.java:71)
    org.apache.lucene.search.FieldCacheImpl.getStringI ndex(FieldCacheImpl.java:351)

    3. org.apache.lucene.store.IndexInput.readVInt(IndexI nput.java:80)
    org.apache.lucene.index.TermBuffer.read(TermBuffer .java:65)
    org.apache.lucene.index.SegmentTermEnum.next(Segme ntTermEnum.java:127)
    org.apache.lucene.search.FieldCacheImpl$10.createV alue(FieldCacheImpl.java:389)
    org.apache.lucene.search.FieldCacheImpl$Cache.get( FieldCacheImpl.java:71)
    org.apache.lucene.search.FieldCacheImpl.getStringI ndex(FieldCacheImpl.java:351)

    4. java.io.RandomAccessFile.seek(Native Method)
    org.apache.lucene.store.FSDirectory$FSIndexInput.r eadInternal(FSDirectory.java:591)
    org.apache.lucene.store.BufferedIndexInput.readByt es(BufferedIndexInput.java:136)

  2. #2
    gimbal2 is offline Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    4,354
    Rep Power
    6

    Default Re: Need help in analyzing some contentions we observe in the Lucene execution

    While this is not the place to ask for tech support with Lucene, it is pretty obvious from the stack dumps that this is caused by file I/O.

    You seem to focus on Java here, I would rate it a higher chance that there is something going on in the machine itself. For example a scheduled job or something that is running rampant and causing your Java process to run out of air.

  3. #3
    DarrylBurke's Avatar
    DarrylBurke is offline Forum Police
    Join Date
    Sep 2008
    Location
    Madgaon, Goa, India
    Posts
    11,458
    Rep Power
    20

    Default Re: Need help in analyzing some contentions we observe in the Lucene execution

    If you're forever cleaning cobwebs, it's time to get rid of the spiders.

  4. #4
    RameshIyerV is offline Member
    Join Date
    Jul 2013
    Posts
    3
    Rep Power
    0

    Default Re: Need help in analyzing some contentions we observe in the Lucene execution

    Quote Originally Posted by gimbal2 View Post
    While this is not the place to ask for tech support with Lucene, it is pretty obvious from the stack dumps that this is caused by file I/O.

    You seem to focus on Java here, I would rate it a higher chance that there is something going on in the machine itself. For example a scheduled job or something that is running rampant and causing your Java process to run out of air.
    Thanks for the prompt reply bro.

    I pretty much had the same understanding and went to our storage team to get some stats on IO waits. The waits are under 5 ms for most of the day and spike occasionally to 10 ms. Our stuck thread setting is at 300 seconds.

    So I analyzed the Lucene code pieces and found that several of those operations are also memory intensive (apart from being an IO operation).

    I am posting here so that i can get some guidance from the experts here, hoping you will be able to look at the spots (code pieces) were we see these contentions and guide on what kind of hardware / system issue can cause this to happen. So could it be anything apart from IO?

  5. #5
    gimbal2 is offline Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    4,354
    Rep Power
    6

    Default Re: Need help in analyzing some contentions we observe in the Lucene execution

    Sorry, I can't perform magic; I can't make answers appear from thin air. I don't have access to your machine so I don't have anything at all to work with. You'll have to figure this out with the help of your coworkers. I'm still not convinced this has anything to do with Java or Lucene at all. But perhaps your crosspost on the Lucene mailing list will prove differently.

  6. #6
    RameshIyerV is offline Member
    Join Date
    Jul 2013
    Posts
    3
    Rep Power
    0

    Default Re: Need help in analyzing some contentions we observe in the Lucene execution

    Quote Originally Posted by gimbal2 View Post
    Sorry, I can't perform magic; I can't make answers appear from thin air. I don't have access to your machine so I don't have anything at all to work with. You'll have to figure this out with the help of your coworkers. I'm still not convinced this has anything to do with Java or Lucene at all. But perhaps your crosspost on the Lucene mailing list will prove differently.
    Thanks for giving it a shot.

Similar Threads

  1. Replies: 6
    Last Post: 11-30-2011, 09:39 PM
  2. Event.observe problem
    By yoni101kessler@ in forum New To Java
    Replies: 1
    Last Post: 09-07-2011, 10:26 PM
  3. Analyzing a line of a text file
    By Ragoune in forum New To Java
    Replies: 3
    Last Post: 06-29-2011, 04:54 PM
  4. Replies: 10
    Last Post: 11-29-2010, 03:02 PM
  5. Reading & Analyzing a Text file and
    By A.M.S in forum New To Java
    Replies: 1
    Last Post: 11-07-2009, 07:47 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •