Initializing shards everytime!


(IT2) #1

Hello everyone!
I have an elasticsearch cluster: 1 master node with logstash on board and 3 data nodes.
And everytime in kopf i see next picture: http://prntscr.com/7xv67x
What is the reason of this ?
Please give me some advice. Thanks!


(Mark Walkom) #2

Looks like you may have heap pressure, how much data in the cluster, how much heap?


(IT2) #3

Look here:


(IT2) #4

Any advices, guys?


(Mark Walkom) #5

Are you experiencing high GC?
I'd suggest seeing if you can increase the heap, or reduce the amount of data in the cluster and see if that helps.


(IT2) #6

I will try it,
i also see that my swap is overloaded on the data nodes. Maybe i need to do something with that?

free -h
total used free shared buffers cached
Mem: 15G 15G 584M 0B 38M 4,0G
-/+ buffers/cache: 11G 4,6G
Swap: 1,9G 1,9G 0B


(Mark Walkom) #7

Disable swap! https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_swapping_is_the_death_of_performance


(IT2) #8

I disable it!
Now there is no initializing shards, cluster is green, but heap is still big:



Am i still need to increase it ?


(Mark Walkom) #9

If it's still high, yes.


(Dmitriy) #10

i have similar problem with initializing. Using elasticsearch 2.0, heap size 10gb
Log:
`[2015-11-30 20:27:31,111][WARN ][cluster.action.shard ] [orion] [indexer.queue][0] received shard failed for [indexer.queue][0], node[EHA2PbjXTsmipmdGppfHUQ], [P], v[7], s[INITIALIZING], a[id=OwlEK_iWTnGAAa_zKMrhkA], unassigned_info[[reason=ALLOCATION_FAILED], at[2015-11-30T18:27:26.913Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]]; ]], indexUUID [Qn7a3si5TlmWT5jQZRUQPQ], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]]; ]
[indexer.queue][[indexer.queue][0]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:258)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:60)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [indexer.queue][[indexer.queue][0]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:157)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1349)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1344)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:889)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:866)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:249)
... 5 more
Caused by: [indexer.queue][[indexer.queue][0]] EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:233)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:154)
... 11 more
Caused by: ElasticsearchException[unexpected exception reading from translog snapshot of /usr/local/elasticsearch/data/elasticsearch/nodes/0/indices/indexer.queue/0/translog/translog-666.tlog]; nested: EOFException[read past EOF. pos [217667] length: [4] end: [217667]];
at org.elasticsearch.index.translog.TranslogReader.readSize(TranslogReader.java:102)
at org.elasticsearch.index.translog.TranslogReader.access$000(TranslogReader.java:46)
at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.readOperation(TranslogReader.java:297)
at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.next(TranslogReader.java:290)
at org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:70)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:219)
... 12 more


(Christian Dahlqvist) #11

@dmugyr Please create your own thread for this.


(system) #12