Indexing stopped working


#1

Hi guys,
I have one node with 4 different indices on it. Each index has 2 shard and one replica (which is just defined without actual replicating since only one node is currently applicable). Sizes of indexing is:
ads1
size: 18.5G (18.5G)
docs: 4,414,787 (4,414,787)

ads2
size: 19.1G (19.1G)
docs: 4,259,828 (4,259,828)

ads3
size: 18.9G (18.9G)
docs: 4,221,637 (4,221,637)

ads4
size: 20.3G (20.3G)
docs: 4,263,329 (4,263,329).

From some point the ES stopped responding to rest request, even as simple as count
http://somemachine:9200/ads1/pagedocuments/_count
The Head plugin, however, seems to be working - it show the info about indices, etc..
Latest logs from ES console are:
[2014-06-22 17:13:58,922][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 17:13:58,922][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 19:16:13,384][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 19:16:13,535][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 20:54:46,650][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 20:54:46,655][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 23:19:08,747][INFO ][monitor.jvm ] [adds_node_one] [gc][young][36246][12421] duration [886ms], collections [1]/[1.6s], total [886ms]/[5.7m], memory [3.2gb]->[2.9gb]/[4.9gb], all_po
ols {[young] [449.6mb]->[107.5mb]/[532.5mb]}{[survivor] [40.9mb]->[63.2mb]/[66.5mb]}{[old] [2.8gb]->[2.8gb]/[4.3gb]}
[2014-06-22 23:19:50,901][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 23:19:50,901][INFO ][index.engine.internal ] [adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 23:24:24,878][WARN ][monitor.jvm ] [adds_node_one] [gc][young][36494][12533] duration [1.5s], collections [1]/[3.1s], total [1.5s]/[5.8m], memory [3.4gb]->[3.3gb]/[4.9gb], all_pool
s {[young] [234.6mb]->[62.1mb]/[532.5mb]}{[survivor] [65.4mb]->[50.5mb]/[66.5mb]}{[old] [3.1gb]->[3.2gb]/[4.3gb]}
[2014-06-22 23:47:07,611][WARN ][monitor.jvm ] [adds_node_one] [gc][young][37569][13012] duration [1.7s], collections [1]/[2.6s], total [1.7s]/[6.1m], memory [3.6gb]->[3.3gb]/[4.9gb], all_pool
s {[young] [341.7mb]->[13.8mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [3.2gb]->[3.2gb]/[4.3gb]}
[2014-06-22 23:57:15,573][WARN ][monitor.jvm ] [adds_node_one] [gc][young][38039][13231] duration [1s], collections [1]/[2.6s], total [1s]/[6.3m], memory [3.2gb]->[3.1gb]/[4.9gb], all_pools {[
young] [301.7mb]->[215.7mb]/[532.5mb]}{[survivor] [48.3mb]->[37.9mb]/[66.5mb]}{[old] [2.8gb]->[2.8gb]/[4.3gb]}
[2014-06-22 23:57:25,552][WARN ][monitor.jvm ] [adds_node_one] [gc][young][38046][13235] duration [1s], collections [1]/[2.5s], total [1s]/[6.3m], memory [3.2gb]->[3gb]/[4.9gb], all_pools {[yo
ung] [293.9mb]->[78.1mb]/[532.5mb]}{[survivor] [62.9mb]->[49.4mb]/[66.5mb]}{[old] [2.9gb]->[2.9gb]/[4.3gb]}

Does anybody have any ideas? Perhaps we came close to some limit of ES on this machine?

Any advice will be appreciated..


#2

After Flush and Optimize on all 4 indices it seems it's back alive. I probably don't understand something in ES/Lucene policy concerning merging, throttling, etc.. that kept some large chunk in memory without flushing it to disk for a long time/specific command...

the indexing happen in a very high rhythm, perhaps that is the reason.. Weird.


(system) #3