We are using Elasticsearch to storage messages from our internal messanger. Currently we have around 7 billions messages to import to Elasticsearch.
During import we collect additional data from our internal API and prepare bulk request to ES.
What we have:
1 cluster with 8 nodes where one is master and other are master-eligible nodes:
virtualization (we do not use bare metal)
JVM 1.8.0_121 (OpenJDK)
64 GB RAM where 31 GB RAM is set for JVM,
1.1 TB SSD storage
1GHZ 16 core proc
8 shards + 1 replica
We are working with month indicies and currently we need around 140 indicies for messages. We also keep additional data like user informations in 3 additional indicies.
Where is the problem:
We have problem with heap during importing lots of data. Currently we put around 8000 messages per seconds to Elasticseach using multiple workers. Single worker add 1000 messages in one bulk request (aroung 500kB in size).
At the beginning everything looks good, after few hours when we have more than 10 milion messages (around 9GB on storage including replicas), heap on every node increase to 90% - 96% and keep this high level.
We test many settings
- change garbage collector to g1gc - heap works nice, but CPU usage increase dramatically, JVM stop responding, one random node vanish from cluster and everything hang
- change threadpools - same results like with garbage collector
- swapiness set to 5 - small improvement but now enough
- we achive better result when we leave _id generation for ES instead of using own int or uuid4
We still see logs like during search and insert:
Caused by: java.lang.OutOfMemoryError: Java heap space [2017-02-09 00:49:56,690][DEBUG][action.search ] [archive-es0] [archive_2015-06], node[DDLpT-CvQMukscvKIORzUw], [P], v, s[STARTED], a[id=lI_Pm7z5RAiOfkj-hUIIiQ]: Failed to execute [org.elasticsearch.action.search.SearchRequest@2d30088b] lastShard [true] RemoteTransportException[[archive-es0][10.114.1.48:9300][indices:data/read/search[phase/query]]]; nested: ElasticsearchException[java.lang.OutOfMemoryError: Java heap space]; nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space]; nested: OutOfMemoryError[Java heap space]; Caused by: ElasticsearchException[java.lang.OutOfMemoryError: Java heap space]; nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space]; nested: OutOfMemoryError[Java heap space];
Doeas anyone had simillar problems?