Hello everyone!
I have an elasticsearch cluster: 1 master node with logstash on board and 3 data nodes.
And everytime in kopf i see next picture: http://prntscr.com/7xv67x
What is the reason of this ?
Please give me some advice. Thanks!
Looks like you may have heap pressure, how much data in the cluster, how much heap?
Look here:
Any advices, guys?
Are you experiencing high GC?
I'd suggest seeing if you can increase the heap, or reduce the amount of data in the cluster and see if that helps.
I will try it,
i also see that my swap is overloaded on the data nodes. Maybe i need to do something with that?
free -h
total used free shared buffers cached
Mem: 15G 15G 584M 0B 38M 4,0G
-/+ buffers/cache: 11G 4,6G
Swap: 1,9G 1,9G 0B
Disable swap! https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_swapping_is_the_death_of_performance
I disable it!
Now there is no initializing shards, cluster is green, but heap is still big:
Am i still need to increase it ?
If it's still high, yes.
i have similar problem with initializing. Using elasticsearch 2.0, heap size 10gb
Log:
`[2015-11-30 20:27:31,111][WARN ][cluster.action.shard ] [orion] [indexer.queue][0] received shard failed for [indexer.queue][0], node[EHA2PbjXTsmipmdGppfHUQ], [P], v[7], s[INITIALIZING], a[id=OwlEK_iWTnGAAa_zKMrhkA], unassigned_info[[reason=ALLOCATION_FAILED], at[2015-11-30T18:27:26.913Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]]; ]], indexUUID [Qn7a3si5TlmWT5jQZRUQPQ], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]]; ]
[indexer.queue][[indexer.queue][0]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:258)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:60)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [indexer.queue][[indexer.queue][0]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:157)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1349)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1344)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:889)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:866)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:249)
... 5 more
Caused by: [indexer.queue][[indexer.queue][0]] EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:233)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:154)
... 11 more
Caused by: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.translog.TranslogReader.readSize(TranslogReader.java:102)
at org.elasticsearch.index.translog.TranslogReader.access$000(TranslogReader.java:46)
at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.readOperation(TranslogReader.java:297)
at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.next(TranslogReader.java:290)
at org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:70)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:219)
... 12 more