I set up an ES cluster a couple weeks ago dedicated to a specific search
and document pattern and have been experiencing problems with it since then.
Every 18-24 hours we need to restart our cluster because we run out of
heap. Either there's a memory leak or problems with GC. Here is an image of
the sample memory usage:
Note: drops to 0 are cluster restarts
We deployed with JDK 1.7.u25 and v0.90.5. Relevant stats:
- 4 nodes (AWS 2xlarge), 1 replica
- 16G reserved heap
- 15 shards per index, 25 indexes, only 11M docs, relatively uniformly
distributed over indexes ( I know the allocation is overkill right now but
we're preparing for a huge influx of data)
- 200-500 searches/s
- mlockall = true
- Using the Java API in Scala
Things we then tried:
- Per this posthttp://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/ I
updated to a fixed thread pool with unbounded queues. However I understand
this wasn't necessary for 90.5? Nothing changed.
- Changed heap to 8G. Got worse.
- Downgraded JDK to 1.6u41 since it was working on another box. Nothing
- Finally upgraded to 90.7 and 1.7.u45 per this use casehttps://groups.google.com/forum/#!searchin/elasticsearch/jvm$20heap/elasticsearch/tAZIC_ffAiU/n3wPpMu6FzgJ.
Slightly better, now the graphs look like this (we can last 2-3 days
without a restart):
We have a QA setup that is not experiencing problems:
- Identical document structure and query patterns
- 5 shards per index, 500K total docs, ~10-50 searches/s
- 4 nodes, medium instances, 1 replica
- JDK 1.6.u41
I know it's hard to diagnose with just this information, but I was
wondering if anyone has seen something similar and/or if there's something
obvious setting I'm overlooking that I should be checking on. Do I simply
have not enough nodes? Is there any other information I can provide that
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eea5d1ab-e70a-447e-a5a8-4f2e6de210f4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.