# Very sluggish Elasticsearch node; not sure why **URL:** \ **Category:** Elasticsearch\ **Created:** [April 19, 2015, 3:03am UTC](https://discuss.elastic.co/t/very-sluggish-elasticsearch-node-not-sure-why/23308 "2015-04-19T03:03:34Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Dave\_Galbraith](https://avatars.discourse-cdn.com/v4/letter/d/c5a1d2/32.png) [@Dave\_Galbraith](https://discuss.elastic.co/u/Dave_Galbraith)\ **Post date:** [April 19, 2015, 3:03am UTC](https://discuss.elastic.co/t/very-sluggish-elasticsearch-node-not-sure-why/23308/1 "2015-04-19T03:03:34Z")
So I've got this Elasticsearch single-node instance running out on an EC2 m3.2xlarge in the cloud. I'm sending about 2,000 documents per second to it for writing. But it's dropping almost all of them on the floor, and any search queries I try to make against it are taking an extraordinarily long time to complete, if they ever do at all. So I'm a novice at Elasticsearch system administration, so I'm hoping the group here can help me figure out why my Elasticsearch is so unhealthy. Here's what I get from curl -XGET '[http://localhost:9200/\_nodes/stats](http://localhost:9200/_nodes/stats)', which takes a couple minutes to compute: { "cluster\_name": "campfire.production.local", "nodes": { "ZkP5567KSNmab6x0oq6HaQ": { "timestamp": 1429411379818, "name": "node-0ba1d794-5a6a-41fd-8c23-6ccc45fb35fb", "transport\_address": [redacted], "host": [redacted], "ip": [ "inet[redacted]", "NONE" ], "indices": { "docs": { "count": 2257831457, "deleted": 20 }, "store": { "size\_in\_bytes": 677010690614, "throttle\_time\_in\_millis": 340863 }, "indexing": { "index\_total": 11743407, "index\_time\_in\_millis": 95117790, "index\_current": 69923, "delete\_total": 6, "delete\_time\_in\_millis": 40, "delete\_current": 0 }, "get": { "total": 6, "time\_in\_millis": 75, "exists\_total": 6, "exists\_time\_in\_millis": 75, "missing\_total": 0, "missing\_time\_in\_millis": 0, "current": 0 }, "search": { "open\_contexts": 0, "query\_total": 99643, "query\_time\_in\_millis": 208069, "query\_current": 0, "fetch\_total": 32537, "fetch\_time\_in\_millis": 46771, "fetch\_current": 0 }, "merges": { "current": 0, "current\_docs": 0, "current\_size\_in\_bytes": 0, "total": 9259, "total\_time\_in\_millis": 132148793, "total\_docs": 81942369, "total\_size\_in\_bytes": 26105798379 }, "refresh": { "total": 83142, "total\_time\_in\_millis": 17075927 }, "flush": { "total": 404, "total\_time\_in\_millis": 182115 }, "warmer": { "current": 0, "total": 166077, "total\_time\_in\_millis": 113904 }, "filter\_cache": { "memory\_size\_in\_bytes": 13076, "evictions": 0 }, "id\_cache": { "memory\_size\_in\_bytes": 0 }, "fielddata": { "memory\_size\_in\_bytes": 0, "evictions": 0 }, "percolate": { "total": 0, "time\_in\_millis": 0, "current": 0, "memory\_size\_in\_bytes": -1, "memory\_size": "-1b", "queries": 0 }, "completion": { "size\_in\_bytes": 0 }, "segments": { "count": 9225, "memory\_in\_bytes": 22644267832, "index\_writer\_memory\_in\_bytes": 1885176, "version\_map\_memory\_in\_bytes": 95472 }, "translog": { "operations": 39008, "size\_in\_bytes": 0 }, "suggest": { "total": 0, "time\_in\_millis": 0, "current": 0 } }, "os": { "timestamp": 1429411459205, "uptime\_in\_millis": 2863626, "load\_average": [ 3.16, 3.01, 3.19 ], "cpu": { "sys": 1, "user": 35, "idle": 62, "usage": 36, "stolen": 0 }, "mem": { "free\_in\_bytes": 183566336, "used\_in\_bytes": 31380459520, "free\_percent": 30, "used\_percent": 69, "actual\_free\_in\_bytes": 9597153280, "actual\_used\_in\_bytes": 21966872576 }, "swap": { "used\_in\_bytes": 0, "free\_in\_bytes": 0 } }, "process": { "timestamp": 1429411459206, "open\_file\_descriptors": 25719, "cpu": { "percent": 142, "sys\_in\_millis": 6301800, "user\_in\_millis": 191977900, "total\_in\_millis": 198279700 }, "mem": { "resident\_in\_bytes": 8216096768, "share\_in\_bytes": 14045184, "total\_virtual\_in\_bytes": 269005692928 } }, "jvm": { "timestamp": 1429411459236, "uptime\_in\_millis": 131281770, "mem": { "heap\_used\_in\_bytes": 7815295968, "heap\_used\_percent": 99, "heap\_committed\_in\_bytes": 7821852672, "heap\_max\_in\_bytes": 7821852672, "non\_heap\_used\_in\_bytes": 54967760, "non\_heap\_committed\_in\_bytes": 84406272, "pools": { "young": { "used\_in\_bytes": 558432256, "max\_in\_bytes": 558432256, "peak\_used\_in\_bytes": 558432256, "peak\_max\_in\_bytes": 558432256 }, "survivor": { "used\_in\_bytes": 65672464, "max\_in\_bytes": 69730304, "peak\_used\_in\_bytes": 69730304, "peak\_max\_in\_bytes": 69730304 }, "old": { "used\_in\_bytes": 7191197144, "max\_in\_bytes": 7193690112, "peak\_used\_in\_bytes": 7193690112, "peak\_max\_in\_bytes": 7193690112 } } }, "threads": { "count": 133, "peak\_count": 140 }, "gc": { "collectors": { "young": { "collection\_count": 1056, "collection\_time\_in\_millis": 6987 }, "old": { "collection\_count": 50480, "collection\_time\_in\_millis": 82300221 } } }, "buffer\_pools": { "direct": { "count": 147, "used\_in\_bytes": 25199773, "total\_capacity\_in\_bytes": 25199773 }, "mapped": { "count": 9265, "used\_in\_bytes": 256402636679, "total\_capacity\_in\_bytes": 256402636679 } } }, "thread\_pool": { "generic": { "threads": 1, "queue": 0, "active": 0, "rejected": 0, "largest": 5, "completed": 14687 }, "index": { "threads": 8, "queue": 0, "active": 0, "rejected": 0, "largest": 8, "completed": 14 }, "snapshot\_data": { "threads": 0, "queue": 0, "active": 0, "rejected": 0, "largest": 0, "completed": 0 }, "bench": { "threads": 0, "queue": 0, "active": 0, "rejected": 0, "largest": 0, "completed": 0 }, "get": { "threads": 6, "queue": 0, "active": 0, "rejected": 0, "largest": 6, "completed": 6 }, "snapshot": { "threads": 0, "queue": 0, "active": 0, "rejected": 0, "largest": 0, "completed": 0 }, "merge": { "threads": 4, "queue": 0, "active": 0, "rejected": 0, "largest": 4, "completed": 79309 }, "suggest": { "threads": 0, "queue": 0, "active": 0, "rejected": 0, "largest": 0, "completed": 0 }, "bulk": { "threads": 8, "queue": 0, "active": 0, "rejected": 0, "largest": 8, "completed": 64924 }, "optimize": { "threads": 0, "queue": 0, "active": 0, "rejected": 0, "largest": 0, "completed": 0 }, "warmer": { "threads": 3, "queue": 0, "active": 0, "rejected": 0, "largest": 4, "completed": 83012 }, "flush": { "threads": 4, "queue": 0, "active": 0, "rejected": 0, "largest": 4, "completed": 50856 }, "search": { "threads": 24, "queue": 0, "active": 0, "rejected": 0, "largest": 24, "completed": 132495 }, "percolate": { "threads": 0, "queue": 0, "active": 0, "rejected": 0, "largest": 0, "completed": 0 }, "management": { "threads": 5, "queue": 0, "active": 1, "rejected": 0, "largest": 5, "completed": 2467169 }, "refresh": { "threads": 4, "queue": 0, "active": 0, "rejected": 0, "largest": 4, "completed": 81978 } }, "network": { "tcp": { "active\_opens": 20988754, "passive\_opens": 23904325, "curr\_estab": 61, "in\_segs": 3495928683, "out\_segs": 4792521542, "retrans\_segs": 158505, "estab\_resets": 52113, "attempt\_fails": 284654, "in\_errs": 260, "out\_rsts": 447516 } }, "fs": { "timestamp": 1429411459237, "total": { "total\_in\_bytes": 1082125373440, "free\_in\_bytes": 236036149248, "available\_in\_bytes": 181043793920, "disk\_reads": 67918592, "disk\_writes": 93727805, "disk\_io\_op": 161646397, "disk\_read\_size\_in\_bytes": 2662825100288, "disk\_write\_size\_in\_bytes": 9759008243712, "disk\_io\_size\_in\_bytes": 12421833344000 }, "data": [ { "path": [redacted], "mount": [redacted], "dev": "/dev/xvdf", "total\_in\_bytes": 1082125373440, "free\_in\_bytes": 236036149248, "available\_in\_bytes": 181043793920, "disk\_reads": 67918592, "disk\_writes": 93727805, "disk\_io\_op": 161646397, "disk\_read\_size\_in\_bytes": 2662825100288, "disk\_write\_size\_in\_bytes": 9759008243712, "disk\_io\_size\_in\_bytes": 12421833344000 } ] }, "transport": { "server\_open": 13, "rx\_count": 10, "rx\_size\_in\_bytes": 3870, "tx\_count": 10, "tx\_size\_in\_bytes": 3870 }, "http": { "current\_open": 6, "total\_opened": 5457 }, "fielddata\_breaker": { "maximum\_size\_in\_bytes": 4693111603, "maximum\_size": "4.3gb", "estimated\_size\_in\_bytes": 0, "estimated\_size": "0b", "overhead": 1.03, "tripped": 0 } } } } And here's curl -XGET '[http://localhost:9200/\_cluster/stats?human&pretty](http://localhost:9200/_cluster/stats?human&pretty)' (also took a couple minutes to complete): { "timestamp" : 1429412322912, "cluster\_name" : "campfire.production.local", "status" : "green", "indices" : { "count" : 236, "shards" : { "total" : 1180, "primaries" : 1180, "replication" : 0.0, "index" : { "shards" : { "min" : 5, "max" : 5, "avg" : 5.0 }, "primaries" : { "min" : 5, "max" : 5, "avg" : 5.0 }, "replication" : { "min" : 0.0, "max" : 0.0, "avg" : 0.0 } } }, "docs" : { "count" : 2257932671, "deleted" : 20 }, "store" : { "size" : "630.5gb", "size\_in\_bytes" : 677043574900, "throttle\_time" : "5.7m", "throttle\_time\_in\_millis" : 342217 }, "fielddata" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "filter\_cache" : { "memory\_size" : "12.7kb", "memory\_size\_in\_bytes" : 13076, "evictions" : 0 }, "id\_cache" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0 }, "completion" : { "size" : "0b", "size\_in\_bytes" : 0 }, "segments" : { "count" : 9230, "memory" : "21gb", "memory\_in\_bytes" : 22645584136, "index\_writer\_memory" : "0b", "index\_writer\_memory\_in\_bytes" : 0, "version\_map\_memory" : "0b", "version\_map\_memory\_in\_bytes" : 0 }, "percolate" : { "total" : 0, "get\_time" : "0s", "time\_in\_millis" : 0, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 0 } }, "nodes" : { "count" : { "total" : 1, "master\_only" : 0, "data\_only" : 0, "master\_data" : 1, "client" : 0 }, "versions" : ["1.3.2"], "os" : { "available\_processors" : 8, "mem" : { "total" : "29.3gb", "total\_in\_bytes" : 31564025856 }, "cpu" : [ { "vendor" : "Intel", "model" : "Xeon", "mhz" : 2500, "total\_cores" : 8, "total\_sockets" : 8, "cores\_per\_socket" : 32, "cache\_size" : "25kb", "cache\_size\_in\_bytes" : 25600, "count" : 1 } ] }, "process" : { "cpu" : { "percent" : 168 }, "open\_file\_descriptors" : { "min" : 25733, "max" : 25733, "avg" : 25733 } }, "jvm" : { "max\_uptime" : "1.5d", "max\_uptime\_in\_millis" : 132087449, "versions" : [ { "version" : "1.7.0\_72", "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm\_version" : "24.72-b04", "vm\_vendor" : "Oracle Corporation", "count" : 1 } ], "mem" : { "heap\_used" : "7.2gb", "heap\_used\_in\_bytes" : 7751476496, "heap\_max" : "7.2gb", "heap\_max\_in\_bytes" : 7821852672 }, "threads" : 134 }, "fs" : { "total" : "1007.8gb", "total\_in\_bytes" : 1082125373440, "free" : "219.6gb", "free\_in\_bytes" : 235869388800, "available" : "168.4gb", "available\_in\_bytes" : 180877033472, "disk\_reads" : 67926240, "disk\_writes" : 93745279, "disk\_io\_op" : 161671519, "disk\_read\_size" : "2.4tb", "disk\_read\_size\_in\_bytes" : 2663667340288, "disk\_write\_size" : "8.8tb", "disk\_write\_size\_in\_bytes" : 9760728731648, "disk\_io\_size" : "11.2tb", "disk\_io\_size\_in\_bytes" : 12424396071936 }, "plugins" : [] } } I'm fairly confident that somewhere in that morass of information there are some big, glowing red flags that say what's going wrong, but the output is largely Greek to me; the documentation is pretty sparse on the matter. Any help figuring out what's wrong with my Elasticsearch would be greatly appreciated. And let me know if there's any other diagnostics I can run to help debug. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6212a969-0ec2-4fc4-9f0e-696cf83b548f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6212a969-0ec2-4fc4-9f0e-696cf83b548f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 19, 2015, 9:46am UTC](https://discuss.elastic.co/t/very-sluggish-elasticsearch-node-not-sure-why/23308/2 "2015-04-19T09:46:44Z")
I'd guess that you are hitting the capacity of the node. Try closing/deleting indices or upgrading the instance. On 19 April 2015 at 13:03, Dave Galbraith [david92galbraith@gmail.com](mailto:david92galbraith@gmail.com) wrote: > So I've got this Elasticsearch single-node instance running out on an EC2 > m3.2xlarge in the cloud. I'm sending about 2,000 documents per second to it > for writing. But it's dropping almost all of them on the floor, and any > search queries I try to make against it are taking an extraordinarily long > time to complete, if they ever do at all. So I'm a novice at Elasticsearch > system administration, so I'm hoping the group here can help me figure out > why my Elasticsearch is so unhealthy. Here's what I get from curl -XGET ' > [http://localhost:9200/\_nodes/stats](http://localhost:9200/_nodes/stats)', which takes a couple minutes to > compute: > > { > "cluster\_name": "campfire.production.local", > "nodes": { > "ZkP5567KSNmab6x0oq6HaQ": { > "timestamp": 1429411379818, > "name": "node-0ba1d794-5a6a-41fd-8c23-6ccc45fb35fb", > "transport\_address": [redacted], > "host": [redacted], > "ip": [ > "inet[redacted]", > "NONE" > ], > "indices": { > "docs": { > "count": 2257831457, > "deleted": 20 > }, > "store": { > "size\_in\_bytes": 677010690614, > "throttle\_time\_in\_millis": 340863 > }, > "indexing": { > "index\_total": 11743407, > "index\_time\_in\_millis": 95117790, > "index\_current": 69923, > "delete\_total": 6, > "delete\_time\_in\_millis": 40, > "delete\_current": 0 > }, > "get": { > "total": 6, > "time\_in\_millis": 75, > "exists\_total": 6, > "exists\_time\_in\_millis": 75, > "missing\_total": 0, > "missing\_time\_in\_millis": 0, > "current": 0 > }, > "search": { > "open\_contexts": 0, > "query\_total": 99643, > "query\_time\_in\_millis": 208069, > "query\_current": 0, > "fetch\_total": 32537, > "fetch\_time\_in\_millis": 46771, > "fetch\_current": 0 > }, > "merges": { > "current": 0, > "current\_docs": 0, > "current\_size\_in\_bytes": 0, > "total": 9259, > "total\_time\_in\_millis": 132148793, > "total\_docs": 81942369, > "total\_size\_in\_bytes": 26105798379 > }, > "refresh": { > "total": 83142, > "total\_time\_in\_millis": 17075927 > }, > "flush": { > "total": 404, > "total\_time\_in\_millis": 182115 > }, > "warmer": { > "current": 0, > "total": 166077, > "total\_time\_in\_millis": 113904 > }, > "filter\_cache": { > "memory\_size\_in\_bytes": 13076, > "evictions": 0 > }, > "id\_cache": { > "memory\_size\_in\_bytes": 0 > }, > "fielddata": { > "memory\_size\_in\_bytes": 0, > "evictions": 0 > }, > "percolate": { > "total": 0, > "time\_in\_millis": 0, > "current": 0, > "memory\_size\_in\_bytes": -1, > "memory\_size": "-1b", > "queries": 0 > }, > "completion": { > "size\_in\_bytes": 0 > }, > "segments": { > "count": 9225, > "memory\_in\_bytes": 22644267832, > "index\_writer\_memory\_in\_bytes": 1885176, > "version\_map\_memory\_in\_bytes": 95472 > }, > "translog": { > "operations": 39008, > "size\_in\_bytes": 0 > }, > "suggest": { > "total": 0, > "time\_in\_millis": 0, > "current": 0 > } > }, > "os": { > "timestamp": 1429411459205, > "uptime\_in\_millis": 2863626, > "load\_average": [ > 3.16, > 3.01, > 3.19 > ], > "cpu": { > "sys": 1, > "user": 35, > "idle": 62, > "usage": 36, > "stolen": 0 > }, > "mem": { > "free\_in\_bytes": 183566336, > "used\_in\_bytes": 31380459520, > "free\_percent": 30, > "used\_percent": 69, > "actual\_free\_in\_bytes": 9597153280, > "actual\_used\_in\_bytes": 21966872576 > }, > "swap": { > "used\_in\_bytes": 0, > "free\_in\_bytes": 0 > } > }, > "process": { > "timestamp": 1429411459206, > "open\_file\_descriptors": 25719, > "cpu": { > "percent": 142, > "sys\_in\_millis": 6301800, > "user\_in\_millis": 191977900, > "total\_in\_millis": 198279700 > }, > "mem": { > "resident\_in\_bytes": 8216096768, > "share\_in\_bytes": 14045184, > "total\_virtual\_in\_bytes": 269005692928 > } > }, > "jvm": { > "timestamp": 1429411459236, > "uptime\_in\_millis": 131281770, > "mem": { > "heap\_used\_in\_bytes": 7815295968, > "heap\_used\_percent": 99, > "heap\_committed\_in\_bytes": 7821852672, > "heap\_max\_in\_bytes": 7821852672, > "non\_heap\_used\_in\_bytes": 54967760, > "non\_heap\_committed\_in\_bytes": 84406272, > "pools": { > "young": { > "used\_in\_bytes": 558432256, > "max\_in\_bytes": 558432256, > "peak\_used\_in\_bytes": 558432256, > "peak\_max\_in\_bytes": 558432256 > }, > "survivor": { > "used\_in\_bytes": 65672464, > "max\_in\_bytes": 69730304, > "peak\_used\_in\_bytes": 69730304, > "peak\_max\_in\_bytes": 69730304 > }, > "old": { > "used\_in\_bytes": 7191197144, > "max\_in\_bytes": 7193690112, > "peak\_used\_in\_bytes": 7193690112, > "peak\_max\_in\_bytes": 7193690112 > } > } > }, > "threads": { > "count": 133, > "peak\_count": 140 > }, > "gc": { > "collectors": { > "young": { > "collection\_count": 1056, > "collection\_time\_in\_millis": 6987 > }, > "old": { > "collection\_count": 50480, > "collection\_time\_in\_millis": 82300221 > } > } > }, > "buffer\_pools": { > "direct": { > "count": 147, > "used\_in\_bytes": 25199773, > "total\_capacity\_in\_bytes": 25199773 > }, > "mapped": { > "count": 9265, > "used\_in\_bytes": 256402636679, > "total\_capacity\_in\_bytes": 256402636679 > } > } > }, > "thread\_pool": { > "generic": { > "threads": 1, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 5, > "completed": 14687 > }, > "index": { > "threads": 8, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 8, > "completed": 14 > }, > "snapshot\_data": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "bench": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "get": { > "threads": 6, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 6, > "completed": 6 > }, > "snapshot": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "merge": { > "threads": 4, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 79309 > }, > "suggest": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "bulk": { > "threads": 8, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 8, > "completed": 64924 > }, > "optimize": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "warmer": { > "threads": 3, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 83012 > }, > "flush": { > "threads": 4, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 50856 > }, > "search": { > "threads": 24, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 24, > "completed": 132495 > }, > "percolate": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "management": { > "threads": 5, > "queue": 0, > "active": 1, > "rejected": 0, > "largest": 5, > "completed": 2467169 > }, > "refresh": { > "threads": 4, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 81978 > } > }, > "network": { > "tcp": { > "active\_opens": 20988754, > "passive\_opens": 23904325, > "curr\_estab": 61, > "in\_segs": 3495928683, > "out\_segs": 4792521542, > "retrans\_segs": 158505, > "estab\_resets": 52113, > "attempt\_fails": 284654, > "in\_errs": 260, > "out\_rsts": 447516 > } > }, > "fs": { > "timestamp": 1429411459237, > "total": { > "total\_in\_bytes": 1082125373440, > "free\_in\_bytes": 236036149248, > "available\_in\_bytes": 181043793920, > "disk\_reads": 67918592, > "disk\_writes": 93727805, > "disk\_io\_op": 161646397, > "disk\_read\_size\_in\_bytes": 2662825100288, > "disk\_write\_size\_in\_bytes": 9759008243712, > "disk\_io\_size\_in\_bytes": 12421833344000 > }, > "data": [ > { > "path": [redacted], > "mount": [redacted], > "dev": "/dev/xvdf", > "total\_in\_bytes": 1082125373440, > "free\_in\_bytes": 236036149248, > "available\_in\_bytes": 181043793920, > "disk\_reads": 67918592, > "disk\_writes": 93727805, > "disk\_io\_op": 161646397, > "disk\_read\_size\_in\_bytes": 2662825100288, > "disk\_write\_size\_in\_bytes": 9759008243712, > "disk\_io\_size\_in\_bytes": 12421833344000 > } > ] > }, > "transport": { > "server\_open": 13, > "rx\_count": 10, > "rx\_size\_in\_bytes": 3870, > "tx\_count": 10, > "tx\_size\_in\_bytes": 3870 > }, > "http": { > "current\_open": 6, > "total\_opened": 5457 > }, > "fielddata\_breaker": { > "maximum\_size\_in\_bytes": 4693111603, > "maximum\_size": "4.3gb", > "estimated\_size\_in\_bytes": 0, > "estimated\_size": "0b", > "overhead": 1.03, > "tripped": 0 > } > } > } > } > > And here's curl -XGET '[http://localhost:9200/\_cluster/stats?human&pretty](http://localhost:9200/_cluster/stats?human&pretty)' > (also took a couple minutes to complete): > > { > "timestamp" : 1429412322912, > "cluster\_name" : "campfire.production.local", > "status" : "green", > "indices" : { > "count" : 236, > "shards" : { > "total" : 1180, > "primaries" : 1180, > "replication" : 0.0, > "index" : { > "shards" : { > "min" : 5, > "max" : 5, > "avg" : 5.0 > }, > "primaries" : { > "min" : 5, > "max" : 5, > "avg" : 5.0 > }, > "replication" : { > "min" : 0.0, > "max" : 0.0, > "avg" : 0.0 > } > } > }, > "docs" : { > "count" : 2257932671, > "deleted" : 20 > }, > "store" : { > "size" : "630.5gb", > "size\_in\_bytes" : 677043574900, > "throttle\_time" : "5.7m", > "throttle\_time\_in\_millis" : 342217 > }, > "fielddata" : { > "memory\_size" : "0b", > "memory\_size\_in\_bytes" : 0, > "evictions" : 0 > }, > "filter\_cache" : { > "memory\_size" : "12.7kb", > "memory\_size\_in\_bytes" : 13076, > "evictions" : 0 > }, > "id\_cache" : { > "memory\_size" : "0b", > "memory\_size\_in\_bytes" : 0 > }, > "completion" : { > "size" : "0b", > "size\_in\_bytes" : 0 > }, > "segments" : { > "count" : 9230, > "memory" : "21gb", > "memory\_in\_bytes" : 22645584136, > "index\_writer\_memory" : "0b", > "index\_writer\_memory\_in\_bytes" : 0, > "version\_map\_memory" : "0b", > "version\_map\_memory\_in\_bytes" : 0 > }, > "percolate" : { > "total" : 0, > "get\_time" : "0s", > "time\_in\_millis" : 0, > "current" : 0, > "memory\_size\_in\_bytes" : -1, > "memory\_size" : "-1b", > "queries" : 0 > } > }, > "nodes" : { > "count" : { > "total" : 1, > "master\_only" : 0, > "data\_only" : 0, > "master\_data" : 1, > "client" : 0 > }, > "versions" : ["1.3.2"], > "os" : { > "available\_processors" : 8, > "mem" : { > "total" : "29.3gb", > "total\_in\_bytes" : 31564025856 > }, > "cpu" : [ { > "vendor" : "Intel", > "model" : "Xeon", > "mhz" : 2500, > "total\_cores" : 8, > "total\_sockets" : 8, > "cores\_per\_socket" : 32, > "cache\_size" : "25kb", > "cache\_size\_in\_bytes" : 25600, > "count" : 1 > } ] > }, > "process" : { > "cpu" : { > "percent" : 168 > }, > "open\_file\_descriptors" : { > "min" : 25733, > "max" : 25733, > "avg" : 25733 > } > }, > "jvm" : { > "max\_uptime" : "1.5d", > "max\_uptime\_in\_millis" : 132087449, > "versions" : [ { > "version" : "1.7.0\_72", > "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", > "vm\_version" : "24.72-b04", > "vm\_vendor" : "Oracle Corporation", > "count" : 1 > } ], > "mem" : { > "heap\_used" : "7.2gb", > "heap\_used\_in\_bytes" : 7751476496, > "heap\_max" : "7.2gb", > "heap\_max\_in\_bytes" : 7821852672 > }, > "threads" : 134 > }, > "fs" : { > "total" : "1007.8gb", > "total\_in\_bytes" : 1082125373440, > "free" : "219.6gb", > "free\_in\_bytes" : 235869388800, > "available" : "168.4gb", > "available\_in\_bytes" : 180877033472, > "disk\_reads" : 67926240, > "disk\_writes" : 93745279, > "disk\_io\_op" : 161671519, > "disk\_read\_size" : "2.4tb", > "disk\_read\_size\_in\_bytes" : 2663667340288, > "disk\_write\_size" : "8.8tb", > "disk\_write\_size\_in\_bytes" : 9760728731648, > "disk\_io\_size" : "11.2tb", > "disk\_io\_size\_in\_bytes" : 12424396071936 > }, > "plugins" : > } > } > > I'm fairly confident that somewhere in that morass of information there > are some big, glowing red flags that say what's going wrong, but the output > is largely Greek to me; the documentation is pretty sparse on the matter. > Any help figuring out what's wrong with my Elasticsearch would be greatly > appreciated. And let me know if there's any other diagnostics I can run to > help debug. Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/6212a969-0ec2-4fc4-9f0e-696cf83b548f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6212a969-0ec2-4fc4-9f0e-696cf83b548f%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/6212a969-0ec2-4fc4-9f0e-696cf83b548f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6212a969-0ec2-4fc4-9f0e-696cf83b548f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_TeUaueuKa3vgen--m3W%2BfLD6yTjmhJW\_5Z-V%3D5NrCEw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_TeUaueuKa3vgen--m3W%2BfLD6yTjmhJW_5Z-V%3D5NrCEw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![cdahlqvist](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@cdahlqvist](https://discuss.elastic.co/u/cdahlqvist)\ **Post date:** [April 19, 2015, 12:55pm UTC](https://discuss.elastic.co/t/very-sluggish-elasticsearch-node-not-sure-why/23308/3 "2015-04-19T12:55:15Z")
Hi, You seem to have quite a large number of shards (1180) for a single node with only 7GB heap. As the total data volume is a bit over 600GB, the average shard size is only a bit over 500MB, which is not very large. As each shard is a separate Lucene index and carries some overhead, you would probably benefit from having fewer, larger shards. With 236 indices it looks like you still have the default 5 shards per index. I would recommend reducing the number of shards per index to 1 going forward in order to increase the average shard size. It is also generally recommended to allocate 50% of the available RAM to heap, so you may want to increase it from 7GB to 15GB as you have 30GB available on a EC2 m3.2xlarge instance. As there is some throttling going on I would also recommend taking a look at iostat to check if you are limited by the performance of your storage. If this does not help it is possible that you, as Mark pointed out, are hitting the capacity of the node. Best regards, Christian On Sunday, April 19, 2015 at 4:03:35 AM UTC+1, Dave Galbraith wrote: > So I've got this Elasticsearch single-node instance running out on an EC2 > m3.2xlarge in the cloud. I'm sending about 2,000 documents per second to it > for writing. But it's dropping almost all of them on the floor, and any > search queries I try to make against it are taking an extraordinarily long > time to complete, if they ever do at all. So I'm a novice at Elasticsearch > system administration, so I'm hoping the group here can help me figure out > why my Elasticsearch is so unhealthy. Here's what I get from curl -XGET ' > [http://localhost:9200/\_nodes/stats](http://localhost:9200/_nodes/stats)', which takes a couple minutes to > compute: > > { > "cluster\_name": "campfire.production.local", > "nodes": { > "ZkP5567KSNmab6x0oq6HaQ": { > "timestamp": 1429411379818, > "name": "node-0ba1d794-5a6a-41fd-8c23-6ccc45fb35fb", > "transport\_address": [redacted], > "host": [redacted], > "ip": [ > "inet[redacted]", > "NONE" > ], > "indices": { > "docs": { > "count": 2257831457, > "deleted": 20 > }, > "store": { > "size\_in\_bytes": 677010690614, > "throttle\_time\_in\_millis": 340863 > }, > "indexing": { > "index\_total": 11743407, > "index\_time\_in\_millis": 95117790, > "index\_current": 69923, > "delete\_total": 6, > "delete\_time\_in\_millis": 40, > "delete\_current": 0 > }, > "get": { > "total": 6, > "time\_in\_millis": 75, > "exists\_total": 6, > "exists\_time\_in\_millis": 75, > "missing\_total": 0, > "missing\_time\_in\_millis": 0, > "current": 0 > }, > "search": { > "open\_contexts": 0, > "query\_total": 99643, > "query\_time\_in\_millis": 208069, > "query\_current": 0, > "fetch\_total": 32537, > "fetch\_time\_in\_millis": 46771, > "fetch\_current": 0 > }, > "merges": { > "current": 0, > "current\_docs": 0, > "current\_size\_in\_bytes": 0, > "total": 9259, > "total\_time\_in\_millis": 132148793, > "total\_docs": 81942369, > "total\_size\_in\_bytes": 26105798379 > }, > "refresh": { > "total": 83142, > "total\_time\_in\_millis": 17075927 > }, > "flush": { > "total": 404, > "total\_time\_in\_millis": 182115 > }, > "warmer": { > "current": 0, > "total": 166077, > "total\_time\_in\_millis": 113904 > }, > "filter\_cache": { > "memory\_size\_in\_bytes": 13076, > "evictions": 0 > }, > "id\_cache": { > "memory\_size\_in\_bytes": 0 > }, > "fielddata": { > "memory\_size\_in\_bytes": 0, > "evictions": 0 > }, > "percolate": { > "total": 0, > "time\_in\_millis": 0, > "current": 0, > "memory\_size\_in\_bytes": -1, > "memory\_size": "-1b", > "queries": 0 > }, > "completion": { > "size\_in\_bytes": 0 > }, > "segments": { > "count": 9225, > "memory\_in\_bytes": 22644267832, > "index\_writer\_memory\_in\_bytes": 1885176, > "version\_map\_memory\_in\_bytes": 95472 > }, > "translog": { > "operations": 39008, > "size\_in\_bytes": 0 > }, > "suggest": { > "total": 0, > "time\_in\_millis": 0, > "current": 0 > } > }, > "os": { > "timestamp": 1429411459205, > "uptime\_in\_millis": 2863626, > "load\_average": [ > 3.16, > 3.01, > 3.19 > ], > "cpu": { > "sys": 1, > "user": 35, > "idle": 62, > "usage": 36, > "stolen": 0 > }, > "mem": { > "free\_in\_bytes": 183566336, > "used\_in\_bytes": 31380459520, > "free\_percent": 30, > "used\_percent": 69, > "actual\_free\_in\_bytes": 9597153280, > "actual\_used\_in\_bytes": 21966872576 > }, > "swap": { > "used\_in\_bytes": 0, > "free\_in\_bytes": 0 > } > }, > "process": { > "timestamp": 1429411459206, > "open\_file\_descriptors": 25719, > "cpu": { > "percent": 142, > "sys\_in\_millis": 6301800, > "user\_in\_millis": 191977900, > "total\_in\_millis": 198279700 > }, > "mem": { > "resident\_in\_bytes": 8216096768, > "share\_in\_bytes": 14045184, > "total\_virtual\_in\_bytes": 269005692928 > } > }, > "jvm": { > "timestamp": 1429411459236, > "uptime\_in\_millis": 131281770, > "mem": { > "heap\_used\_in\_bytes": 7815295968, > "heap\_used\_percent": 99, > "heap\_committed\_in\_bytes": 7821852672, > "heap\_max\_in\_bytes": 7821852672, > "non\_heap\_used\_in\_bytes": 54967760, > "non\_heap\_committed\_in\_bytes": 84406272, > "pools": { > "young": { > "used\_in\_bytes": 558432256, > "max\_in\_bytes": 558432256, > "peak\_used\_in\_bytes": 558432256, > "peak\_max\_in\_bytes": 558432256 > }, > "survivor": { > "used\_in\_bytes": 65672464, > "max\_in\_bytes": 69730304, > "peak\_used\_in\_bytes": 69730304, > "peak\_max\_in\_bytes": 69730304 > }, > "old": { > "used\_in\_bytes": 7191197144, > "max\_in\_bytes": 7193690112, > "peak\_used\_in\_bytes": 7193690112, > "peak\_max\_in\_bytes": 7193690112 > } > } > }, > "threads": { > "count": 133, > "peak\_count": 140 > }, > "gc": { > "collectors": { > "young": { > "collection\_count": 1056, > "collection\_time\_in\_millis": 6987 > }, > "old": { > "collection\_count": 50480, > "collection\_time\_in\_millis": 82300221 > } > } > }, > "buffer\_pools": { > "direct": { > "count": 147, > "used\_in\_bytes": 25199773, > "total\_capacity\_in\_bytes": 25199773 > }, > "mapped": { > "count": 9265, > "used\_in\_bytes": 256402636679, > "total\_capacity\_in\_bytes": 256402636679 > } > } > }, > "thread\_pool": { > "generic": { > "threads": 1, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 5, > "completed": 14687 > }, > "index": { > "threads": 8, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 8, > "completed": 14 > }, > "snapshot\_data": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "bench": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "get": { > "threads": 6, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 6, > "completed": 6 > }, > "snapshot": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "merge": { > "threads": 4, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 79309 > }, > "suggest": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "bulk": { > "threads": 8, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 8, > "completed": 64924 > }, > "optimize": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "warmer": { > "threads": 3, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 83012 > }, > "flush": { > "threads": 4, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 50856 > }, > "search": { > "threads": 24, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 24, > "completed": 132495 > }, > "percolate": { > "threads": 0, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 0, > "completed": 0 > }, > "management": { > "threads": 5, > "queue": 0, > "active": 1, > "rejected": 0, > "largest": 5, > "completed": 2467169 > }, > "refresh": { > "threads": 4, > "queue": 0, > "active": 0, > "rejected": 0, > "largest": 4, > "completed": 81978 > } > }, > "network": { > "tcp": { > "active\_opens": 20988754, > "passive\_opens": 23904325, > "curr\_estab": 61, > "in\_segs": 3495928683, > "out\_segs": 4792521542, > "retrans\_segs": 158505, > "estab\_resets": 52113, > "attempt\_fails": 284654, > "in\_errs": 260, > "out\_rsts": 447516 > } > }, > "fs": { > "timestamp": 1429411459237, > "total": { > "total\_in\_bytes": 1082125373440, > "free\_in\_bytes": 236036149248, > "available\_in\_bytes": 181043793920, > "disk\_reads": 67918592, > "disk\_writes": 93727805, > "disk\_io\_op": 161646397, > "disk\_read\_size\_in\_bytes": 2662825100288, > "disk\_write\_size\_in\_bytes": 9759008243712, > "disk\_io\_size\_in\_bytes": 12421833344000 > }, > "data": [ > { > "path": [redacted], > "mount": [redacted], > "dev": "/dev/xvdf", > "total\_in\_bytes": 1082125373440, > "free\_in\_bytes": 236036149248, > "available\_in\_bytes": 181043793920, > "disk\_reads": 67918592, > "disk\_writes": 93727805, > "disk\_io\_op": 161646397, > "disk\_read\_size\_in\_bytes": 2662825100288, > "disk\_write\_size\_in\_bytes": 9759008243712, > "disk\_io\_size\_in\_bytes": 12421833344000 > } > ] > }, > "transport": { > "server\_open": 13, > "rx\_count": 10, > "rx\_size\_in\_bytes": 3870, > "tx\_count": 10, > "tx\_size\_in\_bytes": 3870 > }, > "http": { > "current\_open": 6, > "total\_opened": 5457 > }, > "fielddata\_breaker": { > "maximum\_size\_in\_bytes": 4693111603, > "maximum\_size": "4.3gb", > "estimated\_size\_in\_bytes": 0, > "estimated\_size": "0b", > "overhead": 1.03, > "tripped": 0 > } > } > } > } > > And here's curl -XGET '[http://localhost:9200/\_cluster/stats?human&pretty](http://localhost:9200/_cluster/stats?human&pretty)' > (also took a couple minutes to complete): > > { > "timestamp" : 1429412322912, > "cluster\_name" : "campfire.production.local", > "status" : "green", > "indices" : { > "count" : 236, > "shards" : { > "total" : 1180, > "primaries" : 1180, > "replication" : 0.0, > "index" : { > "shards" : { > "min" : 5, > "max" : 5, > "avg" : 5.0 > }, > "primaries" : { > "min" : 5, > "max" : 5, > "avg" : 5.0 > }, > "replication" : { > "min" : 0.0, > "max" : 0.0, > "avg" : 0.0 > } > } > }, > "docs" : { > "count" : 2257932671, > "deleted" : 20 > }, > "store" : { > "size" : "630.5gb", > "size\_in\_bytes" : 677043574900, > "throttle\_time" : "5.7m", > "throttle\_time\_in\_millis" : 342217 > }, > "fielddata" : { > "memory\_size" : "0b", > "memory\_size\_in\_bytes" : 0, > "evictions" : 0 > }, > "filter\_cache" : { > "memory\_size" : "12.7kb", > "memory\_size\_in\_bytes" : 13076, > "evictions" : 0 > }, > "id\_cache" : { > "memory\_size" : "0b", > "memory\_size\_in\_bytes" : 0 > }, > "completion" : { > "size" : "0b", > "size\_in\_bytes" : 0 > }, > "segments" : { > "count" : 9230, > "memory" : "21gb", > "memory\_in\_bytes" : 22645584136, > "index\_writer\_memory" : "0b", > "index\_writer\_memory\_in\_bytes" : 0, > "version\_map\_memory" : "0b", > "version\_map\_memory\_in\_bytes" : 0 > }, > "percolate" : { > "total" : 0, > "get\_time" : "0s", > "time\_in\_millis" : 0, > "current" : 0, > "memory\_size\_in\_bytes" : -1, > "memory\_size" : "-1b", > "queries" : 0 > } > }, > "nodes" : { > "count" : { > "total" : 1, > "master\_only" : 0, > "data\_only" : 0, > "master\_data" : 1, > "client" : 0 > }, > "versions" : ["1.3.2"], > "os" : { > "available\_processors" : 8, > "mem" : { > "total" : "29.3gb", > "total\_in\_bytes" : 31564025856 > }, > "cpu" : [ { > "vendor" : "Intel", > "model" : "Xeon", > "mhz" : 2500, > "total\_cores" : 8, > "total\_sockets" : 8, > "cores\_per\_socket" : 32, > "cache\_size" : "25kb", > "cache\_size\_in\_bytes" : 25600, > "count" : 1 > } ] > }, > "process" : { > "cpu" : { > "percent" : 168 > }, > "open\_file\_descriptors" : { > "min" : 25733, > "max" : 25733, > "avg" : 25733 > } > }, > "jvm" : { > "max\_uptime" : "1.5d", > "max\_uptime\_in\_millis" : 132087449, > "versions" : [ { > "version" : "1.7.0\_72", > "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", > "vm\_version" : "24.72-b04", > "vm\_vendor" : "Oracle Corporation", > "count" : 1 > } ], > "mem" : { > "heap\_used" : "7.2gb", > "heap\_used\_in\_bytes" : 7751476496, > "heap\_max" : "7.2gb", > "heap\_max\_in\_bytes" : 7821852672 > }, > "threads" : 134 > }, > "fs" : { > "total" : "1007.8gb", > "total\_in\_bytes" : 1082125373440, > "free" : "219.6gb", > "free\_in\_bytes" : 235869388800, > "available" : "168.4gb", > "available\_in\_bytes" : 180877033472, > "disk\_reads" : 67926240, > "disk\_writes" : 93745279, > "disk\_io\_op" : 161671519, > "disk\_read\_size" : "2.4tb", > "disk\_read\_size\_in\_bytes" : 2663667340288, > "disk\_write\_size" : "8.8tb", > "disk\_write\_size\_in\_bytes" : 9760728731648, > "disk\_io\_size" : "11.2tb", > "disk\_io\_size\_in\_bytes" : 12424396071936 > }, > "plugins" : > } > } > > I'm fairly confident that somewhere in that morass of information there > are some big, glowing red flags that say what's going wrong, but the output > is largely Greek to me; the documentation is pretty sparse on the matter. > Any help figuring out what's wrong with my Elasticsearch would be greatly > appreciated. And let me know if there's any other diagnostics I can run to > help debug. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c28f6a00-2e0f-4ca9-a0c7-d5a28059273f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c28f6a00-2e0f-4ca9-a0c7-d5a28059273f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:18am UTC](https://discuss.elastic.co/t/very-sluggish-elasticsearch-node-not-sure-why/23308/4 "2017-07-06T00:18:51Z")