Indexing stopped working

Hi guys,
I have one node with 4 different indices on it. Each index has 2 shard and
one replica (which is just defined without actual replicating since only
one node is currently applicable). Sizes of indexing is:
ads1
size: 18.5G (18.5G)
docs: 4,414,787 (4,414,787)

ads2
size: 19.1G (19.1G)
docs: 4,259,828 (4,259,828)

ads3
size: 18.9G (18.9G)
docs: 4,221,637 (4,221,637)

ads4
size: 20.3G (20.3G)
docs: 4,263,329 (4,263,329).

From some point the ES stopped responding to rest request, even as simple
as count
http://somemachine:9200/ads1/pagedocuments/_count
The Head plugin, however, seems to be working - it show the info about
indices, etc..
Latest logs from ES console are:
[2014-06-22 17:13:58,922][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 17:13:58,922][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 19:16:13,384][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 19:16:13,535][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 20:54:46,650][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 20:54:46,655][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 23:19:08,747][INFO ][monitor.jvm ] [adds_node_one]
[gc][young][36246][12421] duration [886ms], collections [1]/[1.6s], total
[886ms]/[5.7m], memory [3.2gb]->[2.9gb]/[4.9gb], all_po
ols {[young] [449.6mb]->[107.5mb]/[532.5mb]}{[survivor]
[40.9mb]->[63.2mb]/[66.5mb]}{[old] [2.8gb]->[2.8gb]/[4.3gb]}
[2014-06-22 23:19:50,901][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
[2014-06-22 23:19:50,901][INFO ][index.engine.internal ] [adds_node_one]
[ads2][0] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2014-06-22 23:24:24,878][WARN ][monitor.jvm ] [adds_node_one]
[gc][young][36494][12533] duration [1.5s], collections [1]/[3.1s], total
[1.5s]/[5.8m], memory [3.4gb]->[3.3gb]/[4.9gb], all_pool
s {[young] [234.6mb]->[62.1mb]/[532.5mb]}{[survivor]
[65.4mb]->[50.5mb]/[66.5mb]}{[old] [3.1gb]->[3.2gb]/[4.3gb]}
[2014-06-22 23:47:07,611][WARN ][monitor.jvm ] [adds_node_one]
[gc][young][37569][13012] duration [1.7s], collections [1]/[2.6s], total
[1.7s]/[6.1m], memory [3.6gb]->[3.3gb]/[4.9gb], all_pool
s {[young] [341.7mb]->[13.8mb]/[532.5mb]}{[survivor]
[66.5mb]->[66.5mb]/[66.5mb]}{[old] [3.2gb]->[3.2gb]/[4.3gb]}
[2014-06-22 23:57:15,573][WARN ][monitor.jvm ] [adds_node_one]
[gc][young][38039][13231] duration [1s], collections [1]/[2.6s], total
[1s]/[6.3m], memory [3.2gb]->[3.1gb]/[4.9gb], all_pools {[
young] [301.7mb]->[215.7mb]/[532.5mb]}{[survivor]
[48.3mb]->[37.9mb]/[66.5mb]}{[old] [2.8gb]->[2.8gb]/[4.3gb]}
[2014-06-22 23:57:25,552][WARN ][monitor.jvm ] [adds_node_one]
[gc][young][38046][13235] duration [1s], collections [1]/[2.5s], total
[1s]/[6.3m], memory [3.2gb]->[3gb]/[4.9gb], all_pools {[yo
ung] [293.9mb]->[78.1mb]/[532.5mb]}{[survivor]
[62.9mb]->[49.4mb]/[66.5mb]}{[old] [2.9gb]->[2.9gb]/[4.3gb]}

Does anybody have any ideas? Perhaps we came close to some limit of ES on
this machine?

Any advice will be appreciated..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b78fe25-23e0-4b2a-beb3-54691cd3f468%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Check out ElasticHQ instead of head, it gives you more insight into cluster
and node performance.

How much is your heap? What ES and java versions are you on? It's probable
you are running into the limits of your node, but a bit more info would
clarify this.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 June 2014 18:20, Maxim Terletsky sxamt33@gmail.com wrote:

Hi guys,
I have one node with 4 different indices on it. Each index has 2 shard and
one replica (which is just defined without actual replicating since only
one node is currently applicable). Sizes of indexing is:
ads1
size: 18.5G (18.5G)
docs: 4,414,787 (4,414,787)

ads2
size: 19.1G (19.1G)
docs: 4,259,828 (4,259,828)

ads3
size: 18.9G (18.9G)
docs: 4,221,637 (4,221,637)

ads4
size: 20.3G (20.3G)
docs: 4,263,329 (4,263,329).

From some point the ES stopped responding to rest request, even as simple
as count
http://somemachine:9200/ads1/pagedocuments/_count
The Head plugin, however, seems to be working - it show the info about
indices, etc..
Latest logs from ES console are:
[2014-06-22 17:13:58,922][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4,
maxNumMerges=5
[2014-06-22 17:13:58,922][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6,
maxNumMerges=5
[2014-06-22 19:16:13,384][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4,
maxNumMerges=5
[2014-06-22 19:16:13,535][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6,
maxNumMerges=5
[2014-06-22 20:54:46,650][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4,
maxNumMerges=5
[2014-06-22 20:54:46,655][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6,
maxNumMerges=5
[2014-06-22 23:19:08,747][INFO ][monitor.jvm ]
[adds_node_one] [gc][young][36246][12421] duration [886ms], collections
[1]/[1.6s], total [886ms]/[5.7m], memory [3.2gb]->[2.9gb]/[4.9gb], all_po
ols {[young] [449.6mb]->[107.5mb]/[532.5mb]}{[survivor]
[40.9mb]->[63.2mb]/[66.5mb]}{[old] [2.8gb]->[2.8gb]/[4.3gb]}
[2014-06-22 23:19:50,901][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] stop throttling indexing: numMergesInFlight=4,
maxNumMerges=5
[2014-06-22 23:19:50,901][INFO ][index.engine.internal ]
[adds_node_one] [ads2][0] now throttling indexing: numMergesInFlight=6,
maxNumMerges=5
[2014-06-22 23:24:24,878][WARN ][monitor.jvm ]
[adds_node_one] [gc][young][36494][12533] duration [1.5s], collections
[1]/[3.1s], total [1.5s]/[5.8m], memory [3.4gb]->[3.3gb]/[4.9gb], all_pool
s {[young] [234.6mb]->[62.1mb]/[532.5mb]}{[survivor]
[65.4mb]->[50.5mb]/[66.5mb]}{[old] [3.1gb]->[3.2gb]/[4.3gb]}
[2014-06-22 23:47:07,611][WARN ][monitor.jvm ]
[adds_node_one] [gc][young][37569][13012] duration [1.7s], collections
[1]/[2.6s], total [1.7s]/[6.1m], memory [3.6gb]->[3.3gb]/[4.9gb], all_pool
s {[young] [341.7mb]->[13.8mb]/[532.5mb]}{[survivor]
[66.5mb]->[66.5mb]/[66.5mb]}{[old] [3.2gb]->[3.2gb]/[4.3gb]}
[2014-06-22 23:57:15,573][WARN ][monitor.jvm ]
[adds_node_one] [gc][young][38039][13231] duration [1s], collections
[1]/[2.6s], total [1s]/[6.3m], memory [3.2gb]->[3.1gb]/[4.9gb], all_pools
{[
young] [301.7mb]->[215.7mb]/[532.5mb]}{[survivor]
[48.3mb]->[37.9mb]/[66.5mb]}{[old] [2.8gb]->[2.8gb]/[4.3gb]}
[2014-06-22 23:57:25,552][WARN ][monitor.jvm ]
[adds_node_one] [gc][young][38046][13235] duration [1s], collections
[1]/[2.5s], total [1s]/[6.3m], memory [3.2gb]->[3gb]/[4.9gb], all_pools
{[yo
ung] [293.9mb]->[78.1mb]/[532.5mb]}{[survivor]
[62.9mb]->[49.4mb]/[66.5mb]}{[old] [2.9gb]->[2.9gb]/[4.3gb]}

Does anybody have any ideas? Perhaps we came close to some limit of ES on
this machine?

Any advice will be appreciated..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1b78fe25-23e0-4b2a-beb3-54691cd3f468%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1b78fe25-23e0-4b2a-beb3-54691cd3f468%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZXiFRDma0MYkrAnb72x3sEJaVRtBkGFfZLaCujFvt7%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi,
Our computer RAM is 16GB.
In Elasticsearch file located in D:\elasticsearch-1.2.0\bin we put
ES_HEAP_SIZE = 9GB
For Java we put
_JAVA_OPTIONS
-Xmx5g -Xms5g
in Environment Variables.
Version of ES:
number": "1.2.0",
"build_hash": "c82387f290c21505f781c695f365d0ef4098b272",
"build_timestamp": "2014-05-22T12:49:13Z",
"build_snapshot": false,
"lucene_version": "4.8"
Version of Java
In JAVA_HOME in environment variables I see this path
C:\Java\server-jre-8u5-windows-x64\jdk1.8.0_05\jre
so I guess it's
C:\Java\server-jre-8u5-windows-x64\jdk1.8.0_05\bin>java -version
Picked up _JAVA_OPTIONS: -Xmx5g -Xms5g
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)