I'm having a tough time to keep ElasticSearch running healthily for even
20-30 mins in my setup. At an indexing rate of 28-36K per second, the CPU
utilization soon drops to 100% and never recovers. All client requests fail
with UnavailbleShardException or "No Nodes" exception. The logs show
warnings from "monitor.jvm" saying that GC did not free up much of memory.
The ultimate requirement is to import data into the ES cluster at around
60K per second on a setup explained below. The only operation being
performed is bulk import of documents. Soon the ES nodes become
unresponsive and the CPU utilization drops to 100% (from 400-500%). They
don't seem to recover even after the bulk import operations are ceased.
Any suggestions on how to tune the GC based on my requirements? What
other information would be needed to look into this?
- Cluster: a 4 node cluster of c3.2xlarge instances on aws-ec2.
- Load: The only operation during this test is bulk import of data. The
documents are small around the size of ~200-500 bytes and are being bulk
imported into the cluster using storm.
- Bulk Import: A total of 7-9 storm workers using a single BulkProcessor
each to import data into the ES cluster. As seen from the logs, each of the
worker processes are importing around 4K docs per second from each worker
i.e. around 28-36K docs per second getting imported into ES.
- JVM Args: Around 8G of heap, tried with CMS collector as well as G1
- ES configuration:
- "mlockall": true
- "threadpool.bulk.size": 20
- "threadpool.bulk.queue_size": 500
- "indices.memory.index_buffer_size": "50%"
- "index.refresh_interval": "30s"
- "index.merge.policy.segments_per_tier": 100
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHhx-GLy1mftUtFT6eyrnuzcpNTu%3DDt3maj3YnuEdYKP4NaYWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.