Elastic node down unable to find reason?

sandeep_verma1 · March 2, 2017, 6:12am

Elasticsearch node down but unable to find exact reason.
Elasticserach Infra:
1 master
5 data node each has 28 gb ram ,550 gb storage, 8 core
and attached screenshot of marvel of that time (9-10 am).

I major part node down when indexing rate increase over 700 /sec.

Christian_Dahlqvist · March 2, 2017, 8:38am

What is in the Elasticsearch logs?

sandeep_verma1 · March 2, 2017, 9:57am

[2017-02-24 12:00:02,614][DEBUG][index.fielddata.plain [2017-02-24 12:00:02,769][DEBUG][index.fielddata.plain [2017-02-24 12:00:02,906][DEBUG][index.fielddata.plain [2017-02-24 12:00:02,926][DEBUG][index.fielddata.plain [2017-02-24 12:00:02,927][DEBUG][index.fielddata.plain [2017-02-24 12:00:02,972][DEBUG][index.fielddata.plain [2017-02-24 12:00:02,993][DEBUG][index.fielddata.plain [2017-02-24 12:00:03,033][DEBUG][index.fielddata.plain [2017-02-24 12:00:03,048][DEBUG][index.fielddata.plain [2017-02-24 12:00:03,315][DEBUG][index.fielddata.plain [2017-02-24 12:00:05,041][DEBUG][index.engine [2017-02-24 12:00:06,101][DEBUG][index.engine [2017-02-24 12:00:09,762][DEBUG][index.engine [2017-02-24 12:00:11,881][DEBUG][index.engine [2017-02-24 12:00:13,669][DEBUG][index.engine [2017-02-24 12:00:18,397][DEBUG][index.engine [2017-02-24 12:00:23,849][DEBUG][index.engine [2017-02-24 12:00:24,814][DEBUG][index.engine [2017-02-24 12:00:29,081][DEBUG][indices.memory [2017-02-24 12:00:29,081][DEBUG][index.shard ] [es2--node1] [jal] Global-ordinals[_parent] took 1.4s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.6s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.7s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.7s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.7s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.8s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.8s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.8s
] [es2--node1] [jal] Global-ordinals[_parent] took 1.8s
] [es2--node1] [jal] Global-ordinals[_parent] took 2.1s
] [es2--node1] [jal][11] merge segment [_43qt] done: took [36s], [27.4 MB], [69,931 docs], [0s stopped], [0s throttled], [51.3 MB written], [Infinity MB/sec throttle]
] [es2--node1] [jal][19] merge segment [_44ph] done: took [28.5s], [21.7 MB], [54,866 docs], [0s stopped], [0s throttled], [39.3 MB written], [Infinity MB/sec throttle]
] [es2--node1] [jal][18] merge segment [_492c] done: took [26.5s], [19.1 MB], [48,648 docs], [0s stopped], [0s throttled], [36.2 MB written], [Infinity MB/sec throttle]
] [es2--node1] [jal][12] merge segment [_42gv] done: took [30.4s], [20.9 MB], [54,315 docs], [0s stopped], [0s throttled], [39.3 MB written], [Infinity MB/sec throttle]
] [es2--node1] [jal][12] merge segment [_42fi] done: took [2.4m], [90.3 MB], [224,246 docs], [0s stopped], [16.2s throttled], [180.3 MB written], [7.1 MB/sec throttle]
] [es2--node1] [jal][16] merge segment [_43m8] done: took [26.8s], [20.7 MB], [55,371 docs], [0s stopped], [0s throttled], [39.3 MB written], [Infinity MB/sec throttle]
] [es2--node1] [jal][4] merge segment [_484o] done: took [1h], [4,977.0 MB], [11,553,054 docs], [0s stopped], [4.8m throttled], [3,240.3 MB written], [6.5 MB/sec throttle]
] [es2--node1] [jal][4] merge segment [_49it] done: took [3m], [113.1 MB], [279,751 docs], [0s stopped], [20.3s throttled], [208.0 MB written], [6.5 MB/sec throttle]
] [es2--node1] recalculating shard indexing buffer, total is [4.1gb] with [12] active shards, each shard set to indexing=[356.7mb], translog=[64kb]
] [es2--node1] [jal][13] updateBufferSize: engine is closed; skipping

sandeep_verma1 · March 2, 2017, 10:19am

118000 indexing/second

Christian_Dahlqvist · March 2, 2017, 11:03am

These logs do not seem to be from around the time when the node went down. Is there anything notable around that timeframe?

system · March 30, 2017, 11:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.