# ES issues and debugging help **URL:** \ **Category:** Elasticsearch\ **Created:** [December 29, 2011, 9:44pm UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255 "2011-12-29T21:44:31Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![John\_Cwikla](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@John\_Cwikla](https://discuss.elastic.co/u/John_Cwikla)\ **Post date:** [December 29, 2011, 9:44pm UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255/1 "2011-12-29T21:44:31Z")
We've been using ES for the last two months, slowly, getting ready to start using it into production. We have roughly 24m documents, on 3 machines, 24G, 24 core, 4 disk SSD RAID 0 in each. Index size is roughly 36G in total. JVM is given up to 18G per machine. we have it set up into two indexes, one that is using 6 shards and 1 replica with 16m documents, and one that is using 12 shards and 1 replica with about 350,000 documents. (We are basically rebuilding the first index into the second). Benchmarking we can get about 100 q/s per machine when it's doing nothing else, so we like that. I have many shards as the number of items I expect to go into here is about 150-500m. We are having the following issues and I'm looking for some tips on debugging: First, the documents aren't evenly shared, I have 24M, 16M, and 8M on different machines. If I bring it down to two machines, it's even 16/16 - is that odd, or expected? I thought it would always be equal across the 3. Every day or so, one of the machines spikes quickly (in a matter of minutes), runs out of memory, and basically becomes unresponsive. We then see the other two basically spike, and sometimes they recover, and often, they don't. This is while I'm running queries against it - if they aren't being queried against, and I take a node down, they rebalance in a matter of minutes and all is happy. We access everything through PyES. I'm really looking for ways to debug what's going on - the logs aren't that useful, I see out of memory on one, and then the loss of connection to it from the others - I can post some if it would be useful. We did see one stack trace (once) that may showed the stack in a delete, but that may have just been random. And while I'm at it, anyone (Shay? ๐Ÿ™‚ do ES consulting to potentially work with us to figure out what's going on? Here's the two shards: { "ok" : true, "\_shards" : { "total" : 6, "successful" : 6, "failed" : 0 }, "indices" : { "legacy\_places" : { "index" : { "primary\_size" : "19gb", "primary\_size\_in\_bytes" : 20476411666, "size" : "38.1gb", "size\_in\_bytes" : 40952823332 }, "translog" : { "operations" : 0 }, "docs" : { "num\_docs" : 24370334, "max\_doc" : 24385103, "deleted\_docs" : 14769 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 187, "total\_time" : "138ms", "total\_time\_in\_millis" : 138 }, "shards" : { "0" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "pNWhe9zmQjCxw2b7SIJTdw", "relocating\_node" : null, "shard" : 0, "index" : "legacy\_places" }, "state" : "STARTED", "index" : { "size" : "6.1gb", "size\_in\_bytes" : 6656257213 }, "translog" : { "id" : 1323890905623, "operations" : 0 }, "docs" : { "num\_docs" : 7917088, "max\_doc" : 7921473, "deleted\_docs" : 4385 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 32, "total\_time" : "25ms", "total\_time\_in\_millis" : 25 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "VMc1uC1uRUCnxCZTzE4S9A", "relocating\_node" : null, "shard" : 0, "index" : "legacy\_places" }, "state" : "STARTED", "index" : { "size" : "6.1gb", "size\_in\_bytes" : 6656257213 }, "translog" : { "id" : 1323890905623, "operations" : 0 }, "docs" : { "num\_docs" : 7917088, "max\_doc" : 7921473, "deleted\_docs" : 4385 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "24ms", "total\_time\_in\_millis" : 24 } } ], "1" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "pNWhe9zmQjCxw2b7SIJTdw", "relocating\_node" : null, "shard" : 1, "index" : "legacy\_places" }, "state" : "STARTED", "index" : { "size" : "6.4gb", "size\_in\_bytes" : 6954949268 }, "translog" : { "id" : 1323890905917, "operations" : 0 }, "docs" : { "num\_docs" : 8257213, "max\_doc" : 8262908, "deleted\_docs" : 5695 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "24ms", "total\_time\_in\_millis" : 24 } }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "VMc1uC1uRUCnxCZTzE4S9A", "relocating\_node" : null, "shard" : 1, "index" : "legacy\_places" }, "state" : "STARTED", "index" : { "size" : "6.4gb", "size\_in\_bytes" : 6954949268 }, "translog" : { "id" : 1323890905917, "operations" : 0 }, "docs" : { "num\_docs" : 8257213, "max\_doc" : 8262908, "deleted\_docs" : 5695 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "24ms", "total\_time\_in\_millis" : 24 } } ], "2" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "NHx8JgLkSD-IMuff8c349A", "relocating\_node" : null, "shard" : 2, "index" : "legacy\_places" }, "state" : "STARTED", "index" : { "size" : "6.3gb", "size\_in\_bytes" : 6865205185 }, "translog" : { "id" : 1323890905928, "operations" : 0 }, "docs" : { "num\_docs" : 8196033, "max\_doc" : 8200722, "deleted\_docs" : 4689 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "22ms", "total\_time\_in\_millis" : 22 } }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "VMc1uC1uRUCnxCZTzE4S9A", "relocating\_node" : null, "shard" : 2, "index" : "legacy\_places" }, "state" : "STARTED", "index" : { "size" : "6.3gb", "size\_in\_bytes" : 6865205185 }, "translog" : { "id" : 1323890905928, "operations" : 0 }, "docs" : { "num\_docs" : 8196033, "max\_doc" : 8200722, "deleted\_docs" : 4689 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "19ms", "total\_time\_in\_millis" : 19 } } ] } } } } And the smaller one: { "ok" : true, "\_shards" : { "total" : 12, "successful" : 12, "failed" : 0 }, "indices" : { "places" : { "index" : { "primary\_size" : "287.3mb", "primary\_size\_in\_bytes" : 301304028, "size" : "574.6mb", "size\_in\_bytes" : 602608056 }, "translog" : { "operations" : 0 }, "docs" : { "num\_docs" : 328917, "max\_doc" : 370550, "deleted\_docs" : 41633 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 372, "total\_time" : "140ms", "total\_time\_in\_millis" : 140 }, "shards" : { "0" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "pNWhe9zmQjCxw2b7SIJTdw", "relocating\_node" : null, "shard" : 0, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "44.9mb", "size\_in\_bytes" : 47116017 }, "translog" : { "id" : 1324591339645, "operations" : 0 }, "docs" : { "num\_docs" : 52001, "max\_doc" : 58573, "deleted\_docs" : 6572 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "13ms", "total\_time\_in\_millis" : 13 } }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "VMc1uC1uRUCnxCZTzE4S9A", "relocating\_node" : null, "shard" : 0, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "44.9mb", "size\_in\_bytes" : 47116017 }, "translog" : { "id" : 1324591339645, "operations" : 0 }, "docs" : { "num\_docs" : 52001, "max\_doc" : 58573, "deleted\_docs" : 6572 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "12ms", "total\_time\_in\_millis" : 12 } } ], "1" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "VMc1uC1uRUCnxCZTzE4S9A", "relocating\_node" : null, "shard" : 1, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "43.5mb", "size\_in\_bytes" : 45616358 }, "translog" : { "id" : 1324591339818, "operations" : 0 }, "docs" : { "num\_docs" : 50991, "max\_doc" : 55917, "deleted\_docs" : 4926 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "10ms", "total\_time\_in\_millis" : 10 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "NHx8JgLkSD-IMuff8c349A", "relocating\_node" : null, "shard" : 1, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "43.5mb", "size\_in\_bytes" : 45616358 }, "translog" : { "id" : 1324591339818, "operations" : 0 }, "docs" : { "num\_docs" : 50991, "max\_doc" : 55917, "deleted\_docs" : 4926 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "13ms", "total\_time\_in\_millis" : 13 } } ], "2" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "NHx8JgLkSD-IMuff8c349A", "relocating\_node" : null, "shard" : 2, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "54.7mb", "size\_in\_bytes" : 57375684 }, "translog" : { "id" : 1324591339466, "operations" : 0 }, "docs" : { "num\_docs" : 60920, "max\_doc" : 70373, "deleted\_docs" : 9453 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "13ms", "total\_time\_in\_millis" : 13 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "VMc1uC1uRUCnxCZTzE4S9A", "relocating\_node" : null, "shard" : 2, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "54.7mb", "size\_in\_bytes" : 57375684 }, "translog" : { "id" : 1324591339466, "operations" : 0 }, "docs" : { "num\_docs" : 60920, "max\_doc" : 70373, "deleted\_docs" : 9453 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "15ms", "total\_time\_in\_millis" : 15 } } ], "3" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "pNWhe9zmQjCxw2b7SIJTdw", "relocating\_node" : null, "shard" : 3, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "45.4mb", "size\_in\_bytes" : 47665161 }, "translog" : { "id" : 1324591339663, "operations" : 0 }, "docs" : { "num\_docs" : 52301, "max\_doc" : 58879, "deleted\_docs" : 6578 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "13ms", "total\_time\_in\_millis" : 13 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "NHx8JgLkSD-IMuff8c349A", "relocating\_node" : null, "shard" : 3, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "45.4mb", "size\_in\_bytes" : 47665161 }, "translog" : { "id" : 1324591339663, "operations" : 0 }, "docs" : { "num\_docs" : 52301, "max\_doc" : 58879, "deleted\_docs" : 6578 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "14ms", "total\_time\_in\_millis" : 14 } } ], "4" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "pNWhe9zmQjCxw2b7SIJTdw", "relocating\_node" : null, "shard" : 4, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "44.1mb", "size\_in\_bytes" : 46278154 }, "translog" : { "id" : 1324591339839, "operations" : 0 }, "docs" : { "num\_docs" : 51625, "max\_doc" : 56555, "deleted\_docs" : 4930 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "11ms", "total\_time\_in\_millis" : 11 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "NHx8JgLkSD-IMuff8c349A", "relocating\_node" : null, "shard" : 4, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "44.1mb", "size\_in\_bytes" : 46278154 }, "translog" : { "id" : 1324591339839, "operations" : 0 }, "docs" : { "num\_docs" : 51625, "max\_doc" : 56555, "deleted\_docs" : 4930 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "13ms", "total\_time\_in\_millis" : 13 } } ], "5" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "pNWhe9zmQjCxw2b7SIJTdw", "relocating\_node" : null, "shard" : 5, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "54.6mb", "size\_in\_bytes" : 57252654 }, "translog" : { "id" : 1324591339578, "operations" : 0 }, "docs" : { "num\_docs" : 61079, "max\_doc" : 70253, "deleted\_docs" : 9174 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "6ms", "total\_time\_in\_millis" : 6 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "NHx8JgLkSD-IMuff8c349A", "relocating\_node" : null, "shard" : 5, "index" : "places" }, "state" : "STARTED", "index" : { "size" : "54.6mb", "size\_in\_bytes" : 57252654 }, "translog" : { "id" : 1324591339578, "operations" : 0 }, "docs" : { "num\_docs" : 61079, "max\_doc" : 70253, "deleted\_docs" : 9174 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size" : "0b", "current\_size\_in\_bytes" : 0, "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0, "total\_docs" : 0, "total\_size" : "0b", "total\_size\_in\_bytes" : 0 }, "refresh" : { "total" : 0, "total\_time" : "0s", "total\_time\_in\_millis" : 0 }, "flush" : { "total" : 31, "total\_time" : "7ms", "total\_time\_in\_millis" : 7 } } ] } } } } John Cwikla VPE fwix ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 30, 2011, 12:40pm UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255/2 "2011-12-30T12:40:48Z")
Hey: Regarding evenly distributing "docs", elasticsearch will aim to have an even number of shards between nodes, regardless of their "weight", so you might get into this situation. We should support weighted allocation to make this better, its not there yet. Regarding the failure, do you have the logged OutOfMemory failure? Can you gist it? It usually happens when a search that suddenly requires much more memory to be loaded happens, for example, when suddenly searching with a new field to sort on, that field is loaded to memory, which can cause this. -shay.banon On Thu, Dec 29, 2011 at 11:44 PM, John Cwikla [cwikla@fwix.com](mailto:cwikla@fwix.com) wrote: > We've been using ES for the last two months, slowly, getting ready to > start using it into production. > > We have roughly 24m documents, on 3 machines, 24G, 24 core, 4 disk SSD > RAID 0 in each. Index size is roughly 36G in total. JVM is given up to > 18G per machine. > > we have it set up into two indexes, one that is using 6 shards and 1 > replica with 16m documents, and one that is using 12 shards and 1 > replica with about 350,000 documents. (We are basically rebuilding the > first index into the second). Benchmarking we can get about 100 q/s > per machine when it's doing nothing else, so we like that. I have many > shards as the number of items I expect to go into here is about > 150-500m. > > We are having the following issues and I'm looking for some tips on > debugging: > > First, the documents aren't evenly shared, I have 24M, 16M, and 8M on > different machines. If I bring it down to two machines, it's even > 16/16 - is that odd, or expected? I thought it would always be equal > across the 3. > > Every day or so, one of the machines spikes quickly (in a matter of > minutes), runs out of memory, and basically becomes unresponsive. We > then see the other two basically spike, and sometimes they recover, > and often, they don't. This is while I'm running queries against it - > if they aren't being queried against, and I take a node down, they > rebalance in a matter of minutes and all is happy. We access > everything through PyES. > > I'm really looking for ways to debug what's going on - the logs aren't > that useful, I see out of memory on one, and then the loss of > connection to it from the others - I can post some if it would be > useful. We did see one stack trace (once) that may showed the stack > in a delete, but that may have just been random. > > And while I'm at it, anyone (Shay? ๐Ÿ™‚ do ES consulting to potentially > work with us to figure out what's going on? > > Here's the two shards: > > { > "ok" : true, > "\_shards" : { > "total" : 6, > "successful" : 6, > "failed" : 0 > }, > "indices" : { > "legacy\_places" : { > "index" : { > "primary\_size" : "19gb", > "primary\_size\_in\_bytes" : 20476411666, > "size" : "38.1gb", > "size\_in\_bytes" : 40952823332 > }, > "translog" : { > "operations" : 0 > }, > "docs" : { > "num\_docs" : 24370334, > "max\_doc" : 24385103, > "deleted\_docs" : 14769 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 187, > "total\_time" : "138ms", > "total\_time\_in\_millis" : 138 > }, > "shards" : { > "0" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > "relocating\_node" : null, > "shard" : 0, > "index" : "legacy\_places" > }, > "state" : "STARTED", > "index" : { > "size" : "6.1gb", > "size\_in\_bytes" : 6656257213 > }, > "translog" : { > "id" : 1323890905623, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 7917088, > "max\_doc" : 7921473, > "deleted\_docs" : 4385 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 32, > "total\_time" : "25ms", > "total\_time\_in\_millis" : 25 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > "relocating\_node" : null, > "shard" : 0, > "index" : "legacy\_places" > }, > "state" : "STARTED", > "index" : { > "size" : "6.1gb", > "size\_in\_bytes" : 6656257213 > }, > "translog" : { > "id" : 1323890905623, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 7917088, > "max\_doc" : 7921473, > "deleted\_docs" : 4385 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "24ms", > "total\_time\_in\_millis" : 24 > } > } ], > "1" : [ { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > "relocating\_node" : null, > "shard" : 1, > "index" : "legacy\_places" > }, > "state" : "STARTED", > "index" : { > "size" : "6.4gb", > "size\_in\_bytes" : 6954949268 > }, > "translog" : { > "id" : 1323890905917, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 8257213, > "max\_doc" : 8262908, > "deleted\_docs" : 5695 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "24ms", > "total\_time\_in\_millis" : 24 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > "relocating\_node" : null, > "shard" : 1, > "index" : "legacy\_places" > }, > "state" : "STARTED", > "index" : { > "size" : "6.4gb", > "size\_in\_bytes" : 6954949268 > }, > "translog" : { > "id" : 1323890905917, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 8257213, > "max\_doc" : 8262908, > "deleted\_docs" : 5695 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "24ms", > "total\_time\_in\_millis" : 24 > } > } ], > "2" : [ { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "NHx8JgLkSD-IMuff8c349A", > "relocating\_node" : null, > "shard" : 2, > "index" : "legacy\_places" > }, > "state" : "STARTED", > "index" : { > "size" : "6.3gb", > "size\_in\_bytes" : 6865205185 > }, > "translog" : { > "id" : 1323890905928, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 8196033, > "max\_doc" : 8200722, > "deleted\_docs" : 4689 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "22ms", > "total\_time\_in\_millis" : 22 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > "relocating\_node" : null, > "shard" : 2, > "index" : "legacy\_places" > }, > "state" : "STARTED", > "index" : { > "size" : "6.3gb", > "size\_in\_bytes" : 6865205185 > }, > "translog" : { > "id" : 1323890905928, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 8196033, > "max\_doc" : 8200722, > "deleted\_docs" : 4689 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "19ms", > "total\_time\_in\_millis" : 19 > } > } ] > } > } > } > } > > And the smaller one: > > { > "ok" : true, > "\_shards" : { > "total" : 12, > "successful" : 12, > "failed" : 0 > }, > "indices" : { > "places" : { > "index" : { > "primary\_size" : "287.3mb", > "primary\_size\_in\_bytes" : 301304028, > "size" : "574.6mb", > "size\_in\_bytes" : 602608056 > }, > "translog" : { > "operations" : 0 > }, > "docs" : { > "num\_docs" : 328917, > "max\_doc" : 370550, > "deleted\_docs" : 41633 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 372, > "total\_time" : "140ms", > "total\_time\_in\_millis" : 140 > }, > "shards" : { > "0" : [ { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > "relocating\_node" : null, > "shard" : 0, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "44.9mb", > "size\_in\_bytes" : 47116017 > }, > "translog" : { > "id" : 1324591339645, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 52001, > "max\_doc" : 58573, > "deleted\_docs" : 6572 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "13ms", > "total\_time\_in\_millis" : 13 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > "relocating\_node" : null, > "shard" : 0, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "44.9mb", > "size\_in\_bytes" : 47116017 > }, > "translog" : { > "id" : 1324591339645, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 52001, > "max\_doc" : 58573, > "deleted\_docs" : 6572 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "12ms", > "total\_time\_in\_millis" : 12 > } > } ], > "1" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > "relocating\_node" : null, > "shard" : 1, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "43.5mb", > "size\_in\_bytes" : 45616358 > }, > "translog" : { > "id" : 1324591339818, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 50991, > "max\_doc" : 55917, > "deleted\_docs" : 4926 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "10ms", > "total\_time\_in\_millis" : 10 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "NHx8JgLkSD-IMuff8c349A", > "relocating\_node" : null, > "shard" : 1, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "43.5mb", > "size\_in\_bytes" : 45616358 > }, > "translog" : { > "id" : 1324591339818, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 50991, > "max\_doc" : 55917, > "deleted\_docs" : 4926 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "13ms", > "total\_time\_in\_millis" : 13 > } > } ], > "2" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "NHx8JgLkSD-IMuff8c349A", > "relocating\_node" : null, > "shard" : 2, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "54.7mb", > "size\_in\_bytes" : 57375684 > }, > "translog" : { > "id" : 1324591339466, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 60920, > "max\_doc" : 70373, > "deleted\_docs" : 9453 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "13ms", > "total\_time\_in\_millis" : 13 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > "relocating\_node" : null, > "shard" : 2, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "54.7mb", > "size\_in\_bytes" : 57375684 > }, > "translog" : { > "id" : 1324591339466, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 60920, > "max\_doc" : 70373, > "deleted\_docs" : 9453 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "15ms", > "total\_time\_in\_millis" : 15 > } > } ], > "3" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > "relocating\_node" : null, > "shard" : 3, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "45.4mb", > "size\_in\_bytes" : 47665161 > }, > "translog" : { > "id" : 1324591339663, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 52301, > "max\_doc" : 58879, > "deleted\_docs" : 6578 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "13ms", > "total\_time\_in\_millis" : 13 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "NHx8JgLkSD-IMuff8c349A", > "relocating\_node" : null, > "shard" : 3, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "45.4mb", > "size\_in\_bytes" : 47665161 > }, > "translog" : { > "id" : 1324591339663, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 52301, > "max\_doc" : 58879, > "deleted\_docs" : 6578 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "14ms", > "total\_time\_in\_millis" : 14 > } > } ], > "4" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > "relocating\_node" : null, > "shard" : 4, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "44.1mb", > "size\_in\_bytes" : 46278154 > }, > "translog" : { > "id" : 1324591339839, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 51625, > "max\_doc" : 56555, > "deleted\_docs" : 4930 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "11ms", > "total\_time\_in\_millis" : 11 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "NHx8JgLkSD-IMuff8c349A", > "relocating\_node" : null, > "shard" : 4, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "44.1mb", > "size\_in\_bytes" : 46278154 > }, > "translog" : { > "id" : 1324591339839, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 51625, > "max\_doc" : 56555, > "deleted\_docs" : 4930 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "13ms", > "total\_time\_in\_millis" : 13 > } > } ], > "5" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > "relocating\_node" : null, > "shard" : 5, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "54.6mb", > "size\_in\_bytes" : 57252654 > }, > "translog" : { > "id" : 1324591339578, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 61079, > "max\_doc" : 70253, > "deleted\_docs" : 9174 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "6ms", > "total\_time\_in\_millis" : 6 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "NHx8JgLkSD-IMuff8c349A", > "relocating\_node" : null, > "shard" : 5, > "index" : "places" > }, > "state" : "STARTED", > "index" : { > "size" : "54.6mb", > "size\_in\_bytes" : 57252654 > }, > "translog" : { > "id" : 1324591339578, > "operations" : 0 > }, > "docs" : { > "num\_docs" : 61079, > "max\_doc" : 70253, > "deleted\_docs" : 9174 > }, > "merges" : { > "current" : 0, > "current\_docs" : 0, > "current\_size" : "0b", > "current\_size\_in\_bytes" : 0, > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0, > "total\_docs" : 0, > "total\_size" : "0b", > "total\_size\_in\_bytes" : 0 > }, > "refresh" : { > "total" : 0, > "total\_time" : "0s", > "total\_time\_in\_millis" : 0 > }, > "flush" : { > "total" : 31, > "total\_time" : "7ms", > "total\_time\_in\_millis" : 7 > } > } ] > } > } > } > } > > John Cwikla > VPE > fwix ---
**Author:** ![John\_Cwikla](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@John\_Cwikla](https://discuss.elastic.co/u/John_Cwikla)\ **Post date:** [December 30, 2011, 7:35pm UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255/3 "2011-12-30T19:35:28Z")
The odd part about the balancing is that I can watch the nodes catch up, they'll each have 16m, and then a few minutes later it will end up at the 8/16/24. We are looking into getting you a gist, I want to reproduce it first - just trying to figure out the best way to reproduce our case at this point. We currently only have about 5-10 processes hitting the servers (balanced across the 3), and the memory looks like it just keeps creeping up and up. Once the nodes get to about 90% of allocated memory, if one of the nodes spikes and runs out of memory, all hell breaks loose, nodes stop talking to each other, and I need to restart them. The other interesting part is that once I've restarted the nodes, the scripts haven't stopped running (they are just logging errors and moving on), the memory starts at the minimum and starts crawling up again. It's like nothing is ever being free'd. On Dec 30, 4:40 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Hey: > > Regarding evenly distributing "docs", elasticsearch will aim to have an > even number of shards between nodes, regardless of their "weight", so you > might get into this situation. We should support weighted allocation to > make this better, its not there yet. > > Regarding the failure, do you have the logged OutOfMemory failure? Can you > gist it? It usually happens when a search that suddenly requires much more > memory to be loaded happens, for example, when suddenly searching with a > new field to sort on, that field is loaded to memory, which can cause this. > > -shay.banon > > On Thu, Dec 29, 2011 at 11:44 PM, John Cwikla [cwi...@fwix.com](mailto:cwi...@fwix.com) wrote: > > > We've been using ES for the last two months, slowly, getting ready to > > start using it into production. > > > We have roughly 24m documents, on 3 machines, 24G, 24 core, 4 disk SSD > > RAID 0 in each. Index size is roughly 36G in total. JVM is given up to > > 18G per machine. > > > we have it set up into two indexes, one that is using 6 shards and 1 > > replica with 16m documents, and one that is using 12 shards and 1 > > replica with about 350,000 documents. (We are basically rebuilding the > > first index into the second). Benchmarking we can get about 100 q/s > > per machine when it's doing nothing else, so we like that. I have many > > shards as the number of items I expect to go into here is about > > 150-500m. > > > We are having the following issues and I'm looking for some tips on > > debugging: > > > First, the documents aren't evenly shared, I have 24M, 16M, and 8M on > > different machines. If I bring it down to two machines, it's even > > 16/16 - is that odd, or expected? I thought it would always be equal > > across the 3. > > > Every day or so, one of the machines spikes quickly (in a matter of > > minutes), runs out of memory, and basically becomes unresponsive. We > > then see the other two basically spike, and sometimes they recover, > > and often, they don't. This is while I'm running queries against it - > > if they aren't being queried against, and I take a node down, they > > rebalance in a matter of minutes and all is happy. We access > > everything through PyES. > > > I'm really looking for ways to debug what's going on - the logs aren't > > that useful, I see out of memory on one, and then the loss of > > connection to it from the others - I can post some if it would be > > useful. We did see one stack trace (once) that may showed the stack > > in a delete, but that may have just been random. > > > And while I'm at it, anyone (Shay? ๐Ÿ™‚ do ES consulting to potentially > > work with us to figure out what's going on? > > > Here's the two shards: > > > { > > "ok" : true, > > "\_shards" : { > > "total" : 6, > > "successful" : 6, > > "failed" : 0 > > }, > > "indices" : { > > "legacy\_places" : { > > "index" : { > > "primary\_size" : "19gb", > > "primary\_size\_in\_bytes" : 20476411666, > > "size" : "38.1gb", > > "size\_in\_bytes" : 40952823332 > > }, > > "translog" : { > > "operations" : 0 > > }, > > "docs" : { > > "num\_docs" : 24370334, > > "max\_doc" : 24385103, > > "deleted\_docs" : 14769 > > }, > > "merges" : { > > "current" : 0, > > "current\_docs" : 0, > > "current\_size" : "0b", > > "current\_size\_in\_bytes" : 0, > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0, > > "total\_docs" : 0, > > "total\_size" : "0b", > > "total\_size\_in\_bytes" : 0 > > }, > > "refresh" : { > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0 > > }, > > "flush" : { > > "total" : 187, > > "total\_time" : "138ms", > > "total\_time\_in\_millis" : 138 > > }, > > "shards" : { > > "0" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : true, > > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > > "relocating\_node" : null, > > "shard" : 0, > > "index" : "legacy\_places" > > }, > > "state" : "STARTED", > > "index" : { > > "size" : "6.1gb", > > "size\_in\_bytes" : 6656257213 > > }, > > "translog" : { > > "id" : 1323890905623, > > "operations" : 0 > > }, > > "docs" : { > > "num\_docs" : 7917088, > > "max\_doc" : 7921473, > > "deleted\_docs" : 4385 > > }, > > "merges" : { > > "current" : 0, > > "current\_docs" : 0, > > "current\_size" : "0b", > > "current\_size\_in\_bytes" : 0, > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0, > > "total\_docs" : 0, > > "total\_size" : "0b", > > "total\_size\_in\_bytes" : 0 > > }, > > "refresh" : { > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0 > > }, > > "flush" : { > > "total" : 32, > > "total\_time" : "25ms", > > "total\_time\_in\_millis" : 25 > > } > > }, { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > > "relocating\_node" : null, > > "shard" : 0, > > "index" : "legacy\_places" > > }, > > "state" : "STARTED", > > "index" : { > > "size" : "6.1gb", > > "size\_in\_bytes" : 6656257213 > > }, > > "translog" : { > > "id" : 1323890905623, > > "operations" : 0 > > }, > > "docs" : { > > "num\_docs" : 7917088, > > "max\_doc" : 7921473, > > "deleted\_docs" : 4385 > > }, > > "merges" : { > > "current" : 0, > > "current\_docs" : 0, > > "current\_size" : "0b", > > "current\_size\_in\_bytes" : 0, > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0, > > "total\_docs" : 0, > > "total\_size" : "0b", > > "total\_size\_in\_bytes" : 0 > > }, > > "refresh" : { > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0 > > }, > > "flush" : { > > "total" : 31, > > "total\_time" : "24ms", > > "total\_time\_in\_millis" : 24 > > } > > } ], > > "1" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > > "relocating\_node" : null, > > "shard" : 1, > > "index" : "legacy\_places" > > }, > > "state" : "STARTED", > > "index" : { > > "size" : "6.4gb", > > "size\_in\_bytes" : 6954949268 > > }, > > "translog" : { > > "id" : 1323890905917, > > "operations" : 0 > > }, > > "docs" : { > > "num\_docs" : 8257213, > > "max\_doc" : 8262908, > > "deleted\_docs" : 5695 > > }, > > "merges" : { > > "current" : 0, > > "current\_docs" : 0, > > "current\_size" : "0b", > > "current\_size\_in\_bytes" : 0, > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0, > > "total\_docs" : 0, > > "total\_size" : "0b", > > "total\_size\_in\_bytes" : 0 > > }, > > "refresh" : { > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0 > > }, > > "flush" : { > > "total" : 31, > > "total\_time" : "24ms", > > "total\_time\_in\_millis" : 24 > > } > > }, { > > "routing" : { > > "state" : "STARTED", > > "primary" : true, > > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > > "relocating\_node" : null, > > "shard" : 1, > > "index" : "legacy\_places" > > }, > > "state" : "STARTED", > > "index" : { > > "size" : "6.4gb", > > "size\_in\_bytes" : 6954949268 > > }, > > "translog" : { > > "id" : 1323890905917, > > "operations" : 0 > > }, > > "docs" : { > > "num\_docs" : 8257213, > > "max\_doc" : 8262908, > > "deleted\_docs" : 5695 > > }, > > "merges" : { > > "current" : 0, > > "current\_docs" : 0, > > "current\_size" : "0b", > > "current\_size\_in\_bytes" : 0, > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0, > > "total\_docs" : 0, > > "total\_size" : "0b", > > "total\_size\_in\_bytes" : 0 > > }, > > "refresh" : { > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0 > > }, > > "flush" : { > > "total" : 31, > > "total\_time" : "24ms", > > "total\_time\_in\_millis" : 24 > > } > > } ], > > "2" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "NHx8JgLkSD-IMuff8c349A", > > "relocating\_node" : null, > > "shard" : 2, > > "index" : "legacy\_places" > > }, > > "state" : "STARTED", > > "index" : { > > "size" : "6.3gb", > > "size\_in\_bytes" : 6865205185 > > }, > > "translog" : { > > "id" : 1323890905928, > > "operations" : 0 > > }, > > "docs" : { > > "num\_docs" : 8196033, > > "max\_doc" : 8200722, > > "deleted\_docs" : 4689 > > }, > > "merges" : { > > "current" : 0, > > "current\_docs" : 0, > > "current\_size" : "0b", > > "current\_size\_in\_bytes" : 0, > > "total" : 0, > > "total\_time" : "0s", > > "total\_time\_in\_millis" : 0, > > "total\_docs" : 0, > > "total\_size" : "0b", > > ... > > read more ยป ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 30, 2011, 8:45pm UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255/4 "2011-12-30T20:45:45Z")
When the servers start fresh and you start to run your "scripts", how much memory do they consume? Basically, once Java gets OOM, it dumps the heap to a file, and elasticsearch is configured to do it: [https://github.com/elasticsearch/elasticsearch/blob/master/bin/elasticsearch.in.sh#L44](https://github.com/elasticsearch/elasticsearch/blob/master/bin/elasticsearch.in.sh#L44) . The problem is that the size of the file is as the heap you allocated, which is 18gb. If the servers start (post running your search requests) with, lets say 4gb, then maybe you can allocate 8gb to the ES (java) process, let it OOM, and gzip-\>dropbox the file so I can download it and see where the memory is being used? Another option is to get into a little bit more details as to what you run against elasticsearch, what type of search requests, for example. On Fri, Dec 30, 2011 at 9:35 PM, John Cwikla [cwikla@fwix.com](mailto:cwikla@fwix.com) wrote: > The odd part about the balancing is that I can watch the nodes catch > up, they'll > each have 16m, and then a few minutes later it will end up at the > 8/16/24. > > We are looking into getting you a gist, I want to reproduce it first - > just trying to > figure out the best way to reproduce our case at this point. We > currently only have > about 5-10 processes hitting the servers (balanced across the 3), and > the memory looks > like it just keeps creeping up and up. Once the nodes get to about > 90% of allocated > memory, if one of the nodes spikes and runs out of memory, all hell > breaks loose, nodes > stop talking to each other, and I need to restart them. > > The other interesting part is that once I've restarted the nodes, the > scripts > haven't stopped running (they are just logging errors and moving on), > the memory starts at the minimum > and starts crawling up again. It's like nothing is ever being free'd. > > On Dec 30, 4:40 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Hey: > > > > Regarding evenly distributing "docs", elasticsearch will aim to have an > > even number of shards between nodes, regardless of their "weight", so you > > might get into this situation. We should support weighted allocation to > > make this better, its not there yet. > > > > Regarding the failure, do you have the logged OutOfMemory failure? Can > > you > > gist it? It usually happens when a search that suddenly requires much > > more > > memory to be loaded happens, for example, when suddenly searching with a > > new field to sort on, that field is loaded to memory, which can cause > > this. > > > > -shay.banon > > > > On Thu, Dec 29, 2011 at 11:44 PM, John Cwikla [cwi...@fwix.com](mailto:cwi...@fwix.com) wrote: > > > > > We've been using ES for the last two months, slowly, getting ready to > > > start using it into production. > > > > > We have roughly 24m documents, on 3 machines, 24G, 24 core, 4 disk SSD > > > RAID 0 in each. Index size is roughly 36G in total. JVM is given up to > > > 18G per machine. > > > > > we have it set up into two indexes, one that is using 6 shards and 1 > > > replica with 16m documents, and one that is using 12 shards and 1 > > > replica with about 350,000 documents. (We are basically rebuilding the > > > first index into the second). Benchmarking we can get about 100 q/s > > > per machine when it's doing nothing else, so we like that. I have many > > > shards as the number of items I expect to go into here is about > > > 150-500m. > > > > > We are having the following issues and I'm looking for some tips on > > > debugging: > > > > > First, the documents aren't evenly shared, I have 24M, 16M, and 8M on > > > different machines. If I bring it down to two machines, it's even > > > 16/16 - is that odd, or expected? I thought it would always be equal > > > across the 3. > > > > > Every day or so, one of the machines spikes quickly (in a matter of > > > minutes), runs out of memory, and basically becomes unresponsive. We > > > then see the other two basically spike, and sometimes they recover, > > > and often, they don't. This is while I'm running queries against it - > > > if they aren't being queried against, and I take a node down, they > > > rebalance in a matter of minutes and all is happy. We access > > > everything through PyES. > > > > > I'm really looking for ways to debug what's going on - the logs aren't > > > that useful, I see out of memory on one, and then the loss of > > > connection to it from the others - I can post some if it would be > > > useful. We did see one stack trace (once) that may showed the stack > > > in a delete, but that may have just been random. > > > > > And while I'm at it, anyone (Shay? ๐Ÿ™‚ do ES consulting to potentially > > > work with us to figure out what's going on? > > > > > Here's the two shards: > > > > > { > > > "ok" : true, > > > "\_shards" : { > > > "total" : 6, > > > "successful" : 6, > > > "failed" : 0 > > > }, > > > "indices" : { > > > "legacy\_places" : { > > > "index" : { > > > "primary\_size" : "19gb", > > > "primary\_size\_in\_bytes" : 20476411666, > > > "size" : "38.1gb", > > > "size\_in\_bytes" : 40952823332 > > > }, > > > "translog" : { > > > "operations" : 0 > > > }, > > > "docs" : { > > > "num\_docs" : 24370334, > > > "max\_doc" : 24385103, > > > "deleted\_docs" : 14769 > > > }, > > > "merges" : { > > > "current" : 0, > > > "current\_docs" : 0, > > > "current\_size" : "0b", > > > "current\_size\_in\_bytes" : 0, > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0, > > > "total\_docs" : 0, > > > "total\_size" : "0b", > > > "total\_size\_in\_bytes" : 0 > > > }, > > > "refresh" : { > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0 > > > }, > > > "flush" : { > > > "total" : 187, > > > "total\_time" : "138ms", > > > "total\_time\_in\_millis" : 138 > > > }, > > > "shards" : { > > > "0" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : true, > > > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > > > "relocating\_node" : null, > > > "shard" : 0, > > > "index" : "legacy\_places" > > > }, > > > "state" : "STARTED", > > > "index" : { > > > "size" : "6.1gb", > > > "size\_in\_bytes" : 6656257213 > > > }, > > > "translog" : { > > > "id" : 1323890905623, > > > "operations" : 0 > > > }, > > > "docs" : { > > > "num\_docs" : 7917088, > > > "max\_doc" : 7921473, > > > "deleted\_docs" : 4385 > > > }, > > > "merges" : { > > > "current" : 0, > > > "current\_docs" : 0, > > > "current\_size" : "0b", > > > "current\_size\_in\_bytes" : 0, > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0, > > > "total\_docs" : 0, > > > "total\_size" : "0b", > > > "total\_size\_in\_bytes" : 0 > > > }, > > > "refresh" : { > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0 > > > }, > > > "flush" : { > > > "total" : 32, > > > "total\_time" : "25ms", > > > "total\_time\_in\_millis" : 25 > > > } > > > }, { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > > > "relocating\_node" : null, > > > "shard" : 0, > > > "index" : "legacy\_places" > > > }, > > > "state" : "STARTED", > > > "index" : { > > > "size" : "6.1gb", > > > "size\_in\_bytes" : 6656257213 > > > }, > > > "translog" : { > > > "id" : 1323890905623, > > > "operations" : 0 > > > }, > > > "docs" : { > > > "num\_docs" : 7917088, > > > "max\_doc" : 7921473, > > > "deleted\_docs" : 4385 > > > }, > > > "merges" : { > > > "current" : 0, > > > "current\_docs" : 0, > > > "current\_size" : "0b", > > > "current\_size\_in\_bytes" : 0, > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0, > > > "total\_docs" : 0, > > > "total\_size" : "0b", > > > "total\_size\_in\_bytes" : 0 > > > }, > > > "refresh" : { > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0 > > > }, > > > "flush" : { > > > "total" : 31, > > > "total\_time" : "24ms", > > > "total\_time\_in\_millis" : 24 > > > } > > > } ], > > > "1" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > > > "relocating\_node" : null, > > > "shard" : 1, > > > "index" : "legacy\_places" > > > }, > > > "state" : "STARTED", > > > "index" : { > > > "size" : "6.4gb", > > > "size\_in\_bytes" : 6954949268 > > > }, > > > "translog" : { > > > "id" : 1323890905917, > > > "operations" : 0 > > > }, > > > "docs" : { > > > "num\_docs" : 8257213, > > > "max\_doc" : 8262908, > > > "deleted\_docs" : 5695 > > > }, > > > "merges" : { > > > "current" : 0, > > > "current\_docs" : 0, > > > "current\_size" : "0b", > > > "current\_size\_in\_bytes" : 0, > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0, > > > "total\_docs" : 0, > > > "total\_size" : "0b", > > > "total\_size\_in\_bytes" : 0 > > > }, > > > "refresh" : { > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0 > > > }, > > > "flush" : { > > > "total" : 31, > > > "total\_time" : "24ms", > > > "total\_time\_in\_millis" : 24 > > > } > > > }, { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : true, > > > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > > > "relocating\_node" : null, > > > "shard" : 1, > > > "index" : "legacy\_places" > > > }, > > > "state" : "STARTED", > > > "index" : { > > > "size" : "6.4gb", > > > "size\_in\_bytes" : 6954949268 > > > }, > > > "translog" : { > > > "id" : 1323890905917, > > > "operations" : 0 > > > }, > > > "docs" : { > > > "num\_docs" : 8257213, > > > "max\_doc" : 8262908, > > > "deleted\_docs" : 5695 > > > }, > > > "merges" : { > > > "current" : 0, > > > "current\_docs" : 0, > > > "current\_size" : "0b", > > > "current\_size\_in\_bytes" : 0, > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0, > > > "total\_docs" : 0, > > > "total\_size" : "0b", > > > "total\_size\_in\_bytes" : 0 > > > }, > > > "refresh" : { > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0 > > > }, > > > "flush" : { > > > "total" : 31, > > > "total\_time" : "24ms", > > > "total\_time\_in\_millis" : 24 > > > } > > > } ], > > > "2" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "NHx8JgLkSD-IMuff8c349A", > > > "relocating\_node" : null, > > > "shard" : 2, > > > "index" : "legacy\_places" > > > }, > > > "state" : "STARTED", > > > "index" : { > > > "size" : "6.3gb", > > > "size\_in\_bytes" : 6865205185 > > > }, > > > "translog" : { > > > "id" : 1323890905928, > > > "operations" : 0 > > > }, > > > "docs" : { > > > "num\_docs" : 8196033, > > > "max\_doc" : 8200722, > > > "deleted\_docs" : 4689 > > > }, > > > "merges" : { > > > "current" : 0, > > > "current\_docs" : 0, > > > "current\_size" : "0b", > > > "current\_size\_in\_bytes" : 0, > > > "total" : 0, > > > "total\_time" : "0s", > > > "total\_time\_in\_millis" : 0, > > > "total\_docs" : 0, > > > "total\_size" : "0b", > > > > ... > > > > read more ยป ---
**Author:** ![John\_Cwikla](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@John\_Cwikla](https://discuss.elastic.co/u/John_Cwikla)\ **Post date:** [January 7, 2012, 2:30am UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255/5 "2012-01-07T02:30:52Z")
Well I was trying but they are pretty big. I've narrowed it down - we are using a custom scorer that balloons. Walking over an index with 12,000,000 docs, I'm finding that every half million or so the memory jumps up 2-4G (not progressive, a step of 2-4G), so after a million or so, my machines die an untimely death. I'll keep trying to reproduce a case, but in the meantime we've decided to modify our code to handle a second phase scoring in our code which should solve our problem, since without our scorer, nothing bad happens ๐Ÿ™‚ On Dec 30 2011, 12:45 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > When the servers start fresh and you start to run your "scripts", how much > memory do they consume? Basically, once Java gets OOM, it dumps the heap to > a file, and elasticsearch is configured to do it:[https://github.com/elasticsearch/elasticsearch/blob/master/bin/elasti](https://github.com/elasticsearch/elasticsearch/blob/master/bin/elasti)... > . > > The problem is that the size of the file is as the heap you allocated, > which is 18gb. If the servers start (post running your search requests) > with, lets say 4gb, then maybe you can allocate 8gb to the ES (java) > process, let it OOM, and gzip-\>dropbox the file so I can download it and > see where the memory is being used? > > Another option is to get into a little bit more details as to what you run > against elasticsearch, what type of search requests, for example. > > On Fri, Dec 30, 2011 at 9:35 PM, John Cwikla [cwi...@fwix.com](mailto:cwi...@fwix.com) wrote: > > > The odd part about the balancing is that I can watch the nodes catch > > up, they'll > > each have 16m, and then a few minutes later it will end up at the > > 8/16/24. > > > We are looking into getting you a gist, I want to reproduce it first - > > just trying to > > figure out the best way to reproduce our case at this point. We > > currently only have > > about 5-10 processes hitting the servers (balanced across the 3), and > > the memory looks > > like it just keeps creeping up and up. Once the nodes get to about > > 90% of allocated > > memory, if one of the nodes spikes and runs out of memory, all hell > > breaks loose, nodes > > stop talking to each other, and I need to restart them. > > > The other interesting part is that once I've restarted the nodes, the > > scripts > > haven't stopped running (they are just logging errors and moving on), > > the memory starts at the minimum > > and starts crawling up again. It's like nothing is ever being free'd. > > > On Dec 30, 4:40 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > Hey: > > > > Regarding evenly distributing "docs", elasticsearch will aim to have an > > > even number of shards between nodes, regardless of their "weight", so you > > > might get into this situation. We should support weighted allocation to > > > make this better, its not there yet. > > > > Regarding the failure, do you have the logged OutOfMemory failure? Can > > > you > > > gist it? It usually happens when a search that suddenly requires much > > > more > > > memory to be loaded happens, for example, when suddenly searching with a > > > new field to sort on, that field is loaded to memory, which can cause > > > this. > > > > -shay.banon > > > > On Thu, Dec 29, 2011 at 11:44 PM, John Cwikla [cwi...@fwix.com](mailto:cwi...@fwix.com) wrote: > > > > > We've been using ES for the last two months, slowly, getting ready to > > > > start using it into production. > > > > > We have roughly 24m documents, on 3 machines, 24G, 24 core, 4 disk SSD > > > > RAID 0 in each. Index size is roughly 36G in total. JVM is given up to > > > > 18G per machine. > > > > > we have it set up into two indexes, one that is using 6 shards and 1 > > > > replica with 16m documents, and one that is using 12 shards and 1 > > > > replica with about 350,000 documents. (We are basically rebuilding the > > > > first index into the second). Benchmarking we can get about 100 q/s > > > > per machine when it's doing nothing else, so we like that. I have many > > > > shards as the number of items I expect to go into here is about > > > > 150-500m. > > > > > We are having the following issues and I'm looking for some tips on > > > > debugging: > > > > > First, the documents aren't evenly shared, I have 24M, 16M, and 8M on > > > > different machines. If I bring it down to two machines, it's even > > > > 16/16 - is that odd, or expected? I thought it would always be equal > > > > across the 3. > > > > > Every day or so, one of the machines spikes quickly (in a matter of > > > > minutes), runs out of memory, and basically becomes unresponsive. We > > > > then see the other two basically spike, and sometimes they recover, > > > > and often, they don't. This is while I'm running queries against it - > > > > if they aren't being queried against, and I take a node down, they > > > > rebalance in a matter of minutes and all is happy. We access > > > > everything through PyES. > > > > > I'm really looking for ways to debug what's going on - the logs aren't > > > > that useful, I see out of memory on one, and then the loss of > > > > connection to it from the others - I can post some if it would be > > > > useful. We did see one stack trace (once) that may showed the stack > > > > in a delete, but that may have just been random. > > > > > And while I'm at it, anyone (Shay? ๐Ÿ™‚ do ES consulting to potentially > > > > work with us to figure out what's going on? > > > > > Here's the two shards: > > > > > { > > > > "ok" : true, > > > > "\_shards" : { > > > > "total" : 6, > > > > "successful" : 6, > > > > "failed" : 0 > > > > }, > > > > "indices" : { > > > > "legacy\_places" : { > > > > "index" : { > > > > "primary\_size" : "19gb", > > > > "primary\_size\_in\_bytes" : 20476411666, > > > > "size" : "38.1gb", > > > > "size\_in\_bytes" : 40952823332 > > > > }, > > > > "translog" : { > > > > "operations" : 0 > > > > }, > > > > "docs" : { > > > > "num\_docs" : 24370334, > > > > "max\_doc" : 24385103, > > > > "deleted\_docs" : 14769 > > > > }, > > > > "merges" : { > > > > "current" : 0, > > > > "current\_docs" : 0, > > > > "current\_size" : "0b", > > > > "current\_size\_in\_bytes" : 0, > > > > "total" : 0, > > > > "total\_time" : "0s", > > > > "total\_time\_in\_millis" : 0, > > > > "total\_docs" : 0, > > > > "total\_size" : "0b", > > > > "total\_size\_in\_bytes" : 0 > > > > }, > > > > "refresh" : { > > > > "total" : 0, > > > > "total\_time" : "0s", > > > > "total\_time\_in\_millis" : 0 > > > > }, > > > > "flush" : { > > > > "total" : 187, > > > > "total\_time" : "138ms", > > > > "total\_time\_in\_millis" : 138 > > > > }, > > > > "shards" : { > > > > "0" : [ { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : true, > > > > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > > > > "relocating\_node" : null, > > > > "shard" : 0, > > > > "index" : "legacy\_places" > > > > }, > > > > "state" : "STARTED", > > > > "index" : { > > > > "size" : "6.1gb", > > > > "size\_in\_bytes" : 6656257213 > > > > }, > > > > "translog" : { > > > > "id" : 1323890905623, > > > > "operations" : 0 > > > > }, > > > > "docs" : { > > > > "num\_docs" : 7917088, > > > > "max\_doc" : 7921473, > > > > "deleted\_docs" : 4385 > > > > }, > > > > "merges" : { > > > > "current" : 0, > > > > "current\_docs" : 0, > > > > "current\_size" : "0b", > > > > "current\_size\_in\_bytes" : 0, > > > > "total" : 0, > > > > "total\_time" : "0s", > > > > "total\_time\_in\_millis" : 0, > > > > "total\_docs" : 0, > > > > "total\_size" : "0b", > > > > "total\_size\_in\_bytes" : 0 > > > > }, > > > > "refresh" : { > > > > "total" : 0, > > > > "total\_time" : "0s", > > > > "total\_time\_in\_millis" : 0 > > > > }, > > > > "flush" : { > > > > "total" : 32, > > > > "total\_time" : "25ms", > > > > "total\_time\_in\_millis" : 25 > > > > } > > > > }, { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : false, > > > > "node" : "VMc1uC1uRUCnxCZTzE4S9A", > > > > "relocating\_node" : null, > > > > "shard" : 0, > > > > "index" : "legacy\_places" > > > > }, > > > > "state" : "STARTED", > > > > "index" : { > > > > "size" : "6.1gb", > > > > "size\_in\_bytes" : 6656257213 > > > > }, > > > > "translog" : { > > > > "id" : 1323890905623, > > > > "operations" : 0 > > > > }, > > > > "docs" : { > > > > "num\_docs" : 7917088, > > > > "max\_doc" : 7921473, > > > > "deleted\_docs" : 4385 > > > > }, > > > > "merges" : { > > > > "current" : 0, > > > > "current\_docs" : 0, > > > > "current\_size" : "0b", > > > > "current\_size\_in\_bytes" : 0, > > > > "total" : 0, > > > > "total\_time" : "0s", > > > > "total\_time\_in\_millis" : 0, > > > > "total\_docs" : 0, > > > > "total\_size" : "0b", > > > > "total\_size\_in\_bytes" : 0 > > > > }, > > > > "refresh" : { > > > > "total" : 0, > > > > "total\_time" : "0s", > > > > "total\_time\_in\_millis" : 0 > > > > }, > > > > "flush" : { > > > > "total" : 31, > > > > "total\_time" : "24ms", > > > > "total\_time\_in\_millis" : 24 > > > > } > > > > } ], > > > > "1" : [ { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : false, > > > > "node" : "pNWhe9zmQjCxw2b7SIJTdw", > > > > "relocating\_node" : null, > > > > "shard" : 1, > > > > "index" : "legacy\_places" > > > > }, > > > > "state" : "STARTED", > > > > "index" : { > > > > "size" : "6.4gb", > > > > "size\_in\_bytes" : 6954949268 > > > > }, > > > > "translog" : { > > > > "id" : 1323890905917, > > > > "operations" : 0 > > > > }, > > > > "docs" : { > > > > "num\_docs" : 8257213, > > > > "max\_doc" : 8262908, > > > > "deleted\_docs" : 5695 > > > > }, > > > > "merges" : { > > > > "current" : 0, > > > > "current\_docs" : 0, > > > > "current\_size" : "0b", > > > > "current\_size\_in\_bytes" : 0, > > > > "total" : 0, > > > > "total\_time" : "0s", > > ... > > read more ยป ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:43am UTC](https://discuss.elastic.co/t/es-issues-and-debugging-help/6255/6 "2017-07-06T03:43:36Z")