# What's using memory in ElasticSearch? (Details to follow...) **URL:** \ **Category:** Elasticsearch\ **Created:** [June 3, 2014, 6:59pm UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893 "2014-06-03T18:59:41Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Adam\_Georgiou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_georgiou/32/1409_2.png) [@Adam\_Georgiou](https://discuss.elastic.co/u/Adam_Georgiou)\ **Post date:** [June 3, 2014, 6:59pm UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/1 "2014-06-03T18:59:41Z")
I'm curious as to what is actually using memory in my cluster. Over the course of a day or two we'll slowly progress to about 75% heap usage, at which point we get stuck in a garbage collect loop (i.e., apparently there's nothing to free). However the slow creep has me baffled. It seems to me that the nature of our cluster should make it such that we either have enough memory to handle our load or we don't. I've been looking into the issue using the `_nodes/stats` endpoint and can't exactly get the numbers to align in order to make an education guess as to where my problem might lie. For example, at the moment, on one of three nodes, we happen to be using: - ~50% of our heap space, which comes out to \*15.9703 gb of used heap. \* - 'segments' are at _0.685781 gb_ - 'fielddata' is at _0.0926561 gb_ - 'filter\_cache' is at\* 0.176308 gb. \* - \*Total \*Data is at around _12.6 terabytes._ The other two nodes are very similar. These are the data items I have in my head as relevant, but obviously they don't add up to even close to the heap size. Does anyone see what I'm missing? Does anyone have a guess as to why memory would slowly creep up? Seems to me our memory usage should be relative to the size of our data and the frequency of our queries. However, while we are indexing in real time, our memory growth doesn't seem linearly proportional to the total amount of data (i.e., if 12 TB were too much for our hardware, i'd expect to see the cluster die way closer to start up, not 1-3 days after startup); and while we are also querying in real time, I don't see how this could effect memory so slowly, as the rate of our queries is pretty much stagnant across the course of a day. Any ideas? Running on java version 1.7.0\_55 (Open JDK) Full dump of the `_nodes/stats` output: { "cluster\_name" : "some\_cluster", "nodes" : { "eVinxIXhRmG8ZaI1pjJRRA" : { "timestamp" : 1401816162553, "name" : "newsdataa29", "transport\_address" : "inet[/164.55.96.188:9300]", "host" : "newsdataa29", "ip" : ["inet[/164.55.96.188:9300]", "NONE" ], "attributes" : { "max\_local\_storage\_nodes" : "1", "master" : "true" }, "indices" : { "docs" : { "count" : 14896392, "deleted" : 1957354 }, "store" : { "size\_in\_bytes" : 593188367560, "throttle\_time\_in\_millis" : 424927 }, "indexing" : { "index\_total" : 41194, "index\_time\_in\_millis" : 743544, "index\_current" : 0, "delete\_total" : 0, "delete\_time\_in\_millis" : 0, "delete\_current" : 0 }, "get" : { "total" : 193630, "time\_in\_millis" : 18639, "exists\_total" : 192744, "exists\_time\_in\_millis" : 18567, "missing\_total" : 886, "missing\_time\_in\_millis" : 72, "current" : 0 }, "search" : { "open\_contexts" : 0, "query\_total" : 9666, "query\_time\_in\_millis" : 2491171, "query\_current" : 0, "fetch\_total" : 1149, "fetch\_time\_in\_millis" : 63100, "fetch\_current" : 0 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size\_in\_bytes" : 0, "total" : 1588, "total\_time\_in\_millis" : 626274, "total\_docs" : 463596, "total\_size\_in\_bytes" : 6263961105 }, "refresh" : { "total" : 14459, "total\_time\_in\_millis" : 186342 }, "flush" : { "total" : 183, "total\_time\_in\_millis" : 3338 }, "warmer" : { "current" : 0, "total" : 16490, "total\_time\_in\_millis" : 7715 }, "filter\_cache" : { "memory\_size\_in\_bytes" : 189309489, "evictions" : 0 }, "id\_cache" : { "memory\_size\_in\_bytes" : 0 }, "fielddata" : { "memory\_size\_in\_bytes" : 99488720, "evictions" : 0 }, "percolate" : { "total" : 5113, "time\_in\_millis" : 735585, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 518706 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 2462, "memory\_in\_bytes" : 736351751 }, "translog" : { "operations" : 974, "size\_in\_bytes" : 0 } }, "os" : { "timestamp" : 1401816162607, "uptime\_in\_millis" : 4494603, "load\_average" : [0.12, 0.19, 0.21], "cpu" : { "sys" : 0, "user" : 0, "idle" : 99, "usage" : 0, "stolen" : 0 }, "mem" : { "free\_in\_bytes" : 1461686272, "used\_in\_bytes" : 66043281408, "free\_percent" : 47, "used\_percent" : 52, "actual\_free\_in\_bytes" : 32198017024, "actual\_used\_in\_bytes" : 35306950656 }, "swap" : { "used\_in\_bytes" : 288641024, "free\_in\_bytes" : 1808502784 } }, "process" : { "timestamp" : 1401816162608, "open\_file\_descriptors" : 687, "cpu" : { "percent" : 0, "sys\_in\_millis" : 148090, "user\_in\_millis" : 3837300, "total\_in\_millis" : 3985390 }, "mem" : { "resident\_in\_bytes" : 36110393344, "share\_in\_bytes" : 2666815488, "total\_virtual\_in\_bytes" : 643081678848 } }, "jvm" : { "timestamp" : 1401816162608, "uptime\_in\_millis" : 15236927, "mem" : { "heap\_used\_in\_bytes" : 17147996128, "heap\_used\_percent" : 53, "heap\_committed\_in\_bytes" : 32011649024, "heap\_max\_in\_bytes" : 32011649024, "non\_heap\_used\_in\_bytes" : 62292832, "non\_heap\_committed\_in\_bytes" : 92192768, "pools" : { "young" : { "used\_in\_bytes" : 933064416, "max\_in\_bytes" : 1605304320, "peak\_used\_in\_bytes" : 1605304320, "peak\_max\_in\_bytes" : 1605304320 }, "survivor" : { "used\_in\_bytes" : 110211016, "max\_in\_bytes" : 200605696, "peak\_used\_in\_bytes" : 200605696, "peak\_max\_in\_bytes" : 200605696 }, "old" : { "used\_in\_bytes" : 16104720696, "max\_in\_bytes" : 30205739008, "peak\_used\_in\_bytes" : 22759852448, "peak\_max\_in\_bytes" : 30205739008 } } }, "threads" : { "count" : 414, "peak\_count" : 418 }, "gc" : { "collectors" : { "young" : { "collection\_count" : 984, "collection\_time\_in\_millis" : 52609 }, "old" : { "collection\_count" : 2, "collection\_time\_in\_millis" : 31494 } } }, "buffer\_pools" : { "direct" : { "count" : 461, "used\_in\_bytes" : 94253910, "total\_capacity\_in\_bytes" : 94253910 }, "mapped" : { "count" : 10411, "used\_in\_bytes" : 592575795266, "total\_capacity\_in\_bytes" : 592575795266 } } }, "thread\_pool" : { "generic" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 14, "completed" : 20474 }, "index" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 46688 }, "get" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 148 }, "snapshot" : { "threads" : 5, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 5, "completed" : 14728 }, "merge" : { "threads" : 5, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 5, "completed" : 39031 }, "suggest" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "bulk" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 243 }, "optimize" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "warmer" : { "threads" : 3, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 3, "completed" : 16490 }, "flush" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 2, "completed" : 183 }, "search" : { "threads" : 96, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 96, "completed" : 10816 }, "percolate" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 5113 }, "management" : { "threads" : 5, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 5, "completed" : 32577 }, "refresh" : { "threads" : 8, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 8, "completed" : 14420 } }, "network" : { "tcp" : { "active\_opens" : 184270, "passive\_opens" : 7400580, "curr\_estab" : 115, "in\_segs" : 462492148, "out\_segs" : 341861048, "retrans\_segs" : 16346, "estab\_resets" : 1341, "attempt\_fails" : 72, "in\_errs" : 21, "out\_rsts" : 57676 } }, "fs" : { "timestamp" : 1401816162608, "total" : { "total\_in\_bytes" : 1731616178176, "free\_in\_bytes" : 1136018505728, "available\_in\_bytes" : 1048082395136 }, "data" : [ { "path" : "somedirectory", "mount" : "somedirectory", "dev" : "somedirectory", "total\_in\_bytes" : 1731616178176, "free\_in\_bytes" : 1136018505728, "available\_in\_bytes" : 1048082395136 } ] }, "transport" : { "server\_open" : 39, "rx\_count" : 178775, "rx\_size\_in\_bytes" : 699521746, "tx\_count" : 175506, "tx\_size\_in\_bytes" : 589101063 }, "http" : { "current\_open" : 0, "total\_opened" : 0 }, "fielddata\_breaker" : { "maximum\_size\_in\_bytes" : 25609319219, "maximum\_size" : "23.8gb", "estimated\_size\_in\_bytes" : 99488720, "estimated\_size" : "94.8mb", "overhead" : 1.03 } }, "eL4pkScqTfa\_7OUE6MWszA" : { "timestamp" : 1401816162553, "name" : "newsdataa30", "transport\_address" : "inet[/164.55.96.189:9300]", "host" : "newsdataa30", "ip" : ["inet[/164.55.96.189:9300]", "NONE" ], "attributes" : { "max\_local\_storage\_nodes" : "1", "master" : "true" }, "indices" : { "docs" : { "count" : 15355700, "deleted" : 2194810 }, "store" : { "size\_in\_bytes" : 630611669017, "throttle\_time\_in\_millis" : 243569 }, "indexing" : { "index\_total" : 42160, "index\_time\_in\_millis" : 640546, "index\_current" : 0, "delete\_total" : 0, "delete\_time\_in\_millis" : 0, "delete\_current" : 0 }, "get" : { "total" : 425528, "time\_in\_millis" : 31415, "exists\_total" : 424809, "exists\_time\_in\_millis" : 31388, "missing\_total" : 719, "missing\_time\_in\_millis" : 27, "current" : 0 }, "search" : { "open\_contexts" : 0, "query\_total" : 9147, "query\_time\_in\_millis" : 2560550, "query\_current" : 0, "fetch\_total" : 1143, "fetch\_time\_in\_millis" : 58306, "fetch\_current" : 0 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size\_in\_bytes" : 0, "total" : 1392, "total\_time\_in\_millis" : 373704, "total\_docs" : 325316, "total\_size\_in\_bytes" : 3561830482 }, "refresh" : { "total" : 12959, "total\_time\_in\_millis" : 139769 }, "flush" : { "total" : 179, "total\_time\_in\_millis" : 2505 }, "warmer" : { "current" : 0, "total" : 14888, "total\_time\_in\_millis" : 7423 }, "filter\_cache" : { "memory\_size\_in\_bytes" : 182403055, "evictions" : 0 }, "id\_cache" : { "memory\_size\_in\_bytes" : 0 }, "fielddata" : { "memory\_size\_in\_bytes" : 111598640, "evictions" : 0 }, "percolate" : { "total" : 6817, "time\_in\_millis" : 1126769, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 691375 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 2680, "memory\_in\_bytes" : 761091350 }, "translog" : { "operations" : 922, "size\_in\_bytes" : 0 } }, "os" : { "timestamp" : 1401816162638, "uptime\_in\_millis" : 4494521, "load\_average" : [0.52, 0.31, 0.23], "cpu" : { "sys" : 0, "user" : 0, "idle" : 98, "usage" : 0, "stolen" : 0 }, "mem" : { "free\_in\_bytes" : 1023049728, "used\_in\_bytes" : 66481917952, "free\_percent" : 47, "used\_percent" : 52, "actual\_free\_in\_bytes" : 32155111424, "actual\_used\_in\_bytes" : 35349856256 }, "swap" : { "used\_in\_bytes" : 280563712, "free\_in\_bytes" : 1816580096 } }, "process" : { "timestamp" : 1401816162638, "open\_file\_descriptors" : 697, "cpu" : { "percent" : 0, "sys\_in\_millis" : 138590, "user\_in\_millis" : 4471520, "total\_in\_millis" : 4610110 }, "mem" : { "resident\_in\_bytes" : 36342210560, "share\_in\_bytes" : 2912124928, "total\_virtual\_in\_bytes" : 680464547840 } }, "jvm" : { "timestamp" : 1401816162639, "uptime\_in\_millis" : 15236713, "mem" : { "heap\_used\_in\_bytes" : 21984266408, "heap\_used\_percent" : 68, "heap\_committed\_in\_bytes" : 32011649024, "heap\_max\_in\_bytes" : 32011649024, "non\_heap\_used\_in\_bytes" : 62779736, "non\_heap\_committed\_in\_bytes" : 92708864, "pools" : { "young" : { "used\_in\_bytes" : 1164777768, "max\_in\_bytes" : 1605304320, "peak\_used\_in\_bytes" : 1605304320, "peak\_max\_in\_bytes" : 1605304320 }, "survivor" : { "used\_in\_bytes" : 115874928, "max\_in\_bytes" : 200605696, "peak\_used\_in\_bytes" : 200605696, "peak\_max\_in\_bytes" : 200605696 }, "old" : { "used\_in\_bytes" : 20703613712, "max\_in\_bytes" : 30205739008, "peak\_used\_in\_bytes" : 25235641144, "peak\_max\_in\_bytes" : 30205739008 } } }, "threads" : { "count" : 417, "peak\_count" : 419 }, "gc" : { "collectors" : { "young" : { "collection\_count" : 1220, "collection\_time\_in\_millis" : 60254 }, "old" : { "collection\_count" : 4, "collection\_time\_in\_millis" : 2923 } } }, "buffer\_pools" : { "direct" : { "count" : 501, "used\_in\_bytes" : 96409873, "total\_capacity\_in\_bytes" : 96409873 }, "mapped" : { "count" : 10879, "used\_in\_bytes" : 629950077811, "total\_capacity\_in\_bytes" : 629950077811 } } }, "thread\_pool" : { "generic" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 15, "completed" : 20553 }, "index" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 48962 }, "get" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 221 }, "snapshot" : { "threads" : 5, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 5, "completed" : 13376 }, "merge" : { "threads" : 5, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 5, "completed" : 39881 }, "suggest" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "bulk" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 320 }, "optimize" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "warmer" : { "threads" : 4, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 4, "completed" : 14888 }, "flush" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 179 }, "search" : { "threads" : 96, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 96, "completed" : 10291 }, "percolate" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 6817 }, "management" : { "threads" : 5, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 5, "completed" : 32526 }, "refresh" : { "threads" : 9, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 9, "completed" : 12923 } }, "network" : { "tcp" : { "active\_opens" : 187257, "passive\_opens" : 7405296, "curr\_estab" : 125, "in\_segs" : 444968606, "out\_segs" : 336592894, "retrans\_segs" : 31766, "estab\_resets" : 1347, "attempt\_fails" : 94, "in\_errs" : 26, "out\_rsts" : 56295 } }, "fs" : { "timestamp" : 1401816162639, "total" : { "total\_in\_bytes" : 1731616178176, "free\_in\_bytes" : 1098595561472, "available\_in\_bytes" : 1010659450880 }, "data" : [ { "path" : "somedirectory", "mount" : "somedirectory", "dev" : "somedirectory", "total\_in\_bytes" : 1731616178176, "free\_in\_bytes" : 1098595561472, "available\_in\_bytes" : 1010659450880 } ] }, "transport" : { "server\_open" : 39, "rx\_count" : 187263, "rx\_size\_in\_bytes" : 725020029, "tx\_count" : 184359, "tx\_size\_in\_bytes" : 476329370 }, "http" : { "current\_open" : 0, "total\_opened" : 0 }, "fielddata\_breaker" : { "maximum\_size\_in\_bytes" : 25609319219, "maximum\_size" : "23.8gb", "estimated\_size\_in\_bytes" : 111598640, "estimated\_size" : "106.4mb", "overhead" : 1.03 } }, "V8RVx-3PQE2i6x4Jz-KyjQ" : { "timestamp" : 1401816162553, "name" : "newsdataa28", "transport\_address" : "inet[newsdataa28/164.55.96.187:9300]", "host" : "newsdataa28", "ip" : ["inet[newsdataa28/164.55.96.187:9300]", "NONE" ], "attributes" : { "max\_local\_storage\_nodes" : "1", "master" : "true" }, "indices" : { "docs" : { "count" : 15620937, "deleted" : 2080647 }, "store" : { "size\_in\_bytes" : 661959225415, "throttle\_time\_in\_millis" : 376994 }, "indexing" : { "index\_total" : 36962, "index\_time\_in\_millis" : 712752, "index\_current" : 0, "delete\_total" : 0, "delete\_time\_in\_millis" : 0, "delete\_current" : 0 }, "get" : { "total" : 211415, "time\_in\_millis" : 16067, "exists\_total" : 210827, "exists\_time\_in\_millis" : 16035, "missing\_total" : 588, "missing\_time\_in\_millis" : 32, "current" : 0 }, "search" : { "open\_contexts" : 0, "query\_total" : 9211, "query\_time\_in\_millis" : 2462277, "query\_current" : 0, "fetch\_total" : 1125, "fetch\_time\_in\_millis" : 72893, "fetch\_current" : 0 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size\_in\_bytes" : 0, "total" : 1367, "total\_time\_in\_millis" : 550372, "total\_docs" : 378105, "total\_size\_in\_bytes" : 5630682978 }, "refresh" : { "total" : 12317, "total\_time\_in\_millis" : 188853 }, "flush" : { "total" : 170, "total\_time\_in\_millis" : 2947 }, "warmer" : { "current" : 0, "total" : 14126, "total\_time\_in\_millis" : 6972 }, "filter\_cache" : { "memory\_size\_in\_bytes" : 183982668, "evictions" : 0 }, "id\_cache" : { "memory\_size\_in\_bytes" : 0 }, "fielddata" : { "memory\_size\_in\_bytes" : 107389383, "evictions" : 0 }, "percolate" : { "total" : 5120, "time\_in\_millis" : 1225884, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 518255 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 1865, "memory\_in\_bytes" : 788006138 }, "translog" : { "operations" : 823, "size\_in\_bytes" : 0 } }, "os" : { "timestamp" : 1401816162605, "uptime\_in\_millis" : 4496476, "load\_average" : [0.09, 0.13, 0.17], "cpu" : { "sys" : 0, "user" : 0, "idle" : 98, "usage" : 0, "stolen" : 0 }, "mem" : { "free\_in\_bytes" : 2885521408, "used\_in\_bytes" : 64619446272, "free\_percent" : 47, "used\_percent" : 52, "actual\_free\_in\_bytes" : 32229896192, "actual\_used\_in\_bytes" : 35275071488 }, "swap" : { "used\_in\_bytes" : 311009280, "free\_in\_bytes" : 1786134528 } }, "process" : { "timestamp" : 1401816162605, "open\_file\_descriptors" : 696, "cpu" : { "percent" : 0, "sys\_in\_millis" : 168280, "user\_in\_millis" : 4273640, "total\_in\_millis" : 4441920 }, "mem" : { "resident\_in\_bytes" : 36114485248, "share\_in\_bytes" : 2677698560, "total\_virtual\_in\_bytes" : 711765032960 } }, "jvm" : { "timestamp" : 1401816162606, "uptime\_in\_millis" : 15237177, "mem" : { "heap\_used\_in\_bytes" : 17395618416, "heap\_used\_percent" : 54, "heap\_committed\_in\_bytes" : 32011649024, "heap\_max\_in\_bytes" : 32011649024, "non\_heap\_used\_in\_bytes" : 62692400, "non\_heap\_committed\_in\_bytes" : 92835840, "pools" : { "young" : { "used\_in\_bytes" : 829437384, "max\_in\_bytes" : 1605304320, "peak\_used\_in\_bytes" : 1605304320, "peak\_max\_in\_bytes" : 1605304320 }, "survivor" : { "used\_in\_bytes" : 86595544, "max\_in\_bytes" : 200605696, "peak\_used\_in\_bytes" : 200605696, "peak\_max\_in\_bytes" : 200605696 }, "old" : { "used\_in\_bytes" : 16479585488, "max\_in\_bytes" : 30205739008, "peak\_used\_in\_bytes" : 23011882528, "peak\_max\_in\_bytes" : 30205739008 } } }, "threads" : { "count" : 415, "peak\_count" : 418 }, "gc" : { "collectors" : { "young" : { "collection\_count" : 1269, "collection\_time\_in\_millis" : 56592 }, "old" : { "collection\_count" : 1, "collection\_time\_in\_millis" : 370 } } }, "buffer\_pools" : { "direct" : { "count" : 519, "used\_in\_bytes" : 101124103, "total\_capacity\_in\_bytes" : 101124103 }, "mapped" : { "count" : 9738, "used\_in\_bytes" : 661268222992, "total\_capacity\_in\_bytes" : 661268222992 } } }, "thread\_pool" : { "generic" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 13, "completed" : 23343 }, "index" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 42212 }, "get" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 151 }, "snapshot" : { "threads" : 5, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 5, "completed" : 12569 }, "merge" : { "threads" : 5, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 5, "completed" : 42057 }, "suggest" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "bulk" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 247 }, "optimize" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "warmer" : { "threads" : 3, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 3, "completed" : 14126 }, "flush" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 2, "completed" : 170 }, "search" : { "threads" : 96, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 96, "completed" : 10336 }, "percolate" : { "threads" : 32, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 32, "completed" : 5120 }, "management" : { "threads" : 5, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 5, "completed" : 33230 }, "refresh" : { "threads" : 9, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 9, "completed" : 12289 } }, "network" : { "tcp" : { "active\_opens" : 200151, "passive\_opens" : 7522949, "curr\_estab" : 125, "in\_segs" : 508598169, "out\_segs" : 397943897, "retrans\_segs" : 75727, "estab\_resets" : 3559, "attempt\_fails" : 135, "in\_errs" : 24, "out\_rsts" : 60541 } }, "fs" : { "timestamp" : 1401816162606, "total" : { "total\_in\_bytes" : 1731616178176, "free\_in\_bytes" : 1067249401856, "available\_in\_bytes" : 979313291264 }, "data" : [ { "path" : "somediretory", "mount" : "somedirectory", "dev" : "somedirectory", "total\_in\_bytes" : 1731616178176, "free\_in\_bytes" : 1067249401856, "available\_in\_bytes" : 979313291264 } ] }, "transport" : { "server\_open" : 39, "rx\_count" : 222658, "rx\_size\_in\_bytes" : 511559857, "tx\_count" : 221087, "tx\_size\_in\_bytes" : 870671199 }, "http" : { "current\_open" : 0, "total\_opened" : 0 }, "fielddata\_breaker" : { "maximum\_size\_in\_bytes" : 25609319219, "maximum\_size" : "23.8gb", "estimated\_size\_in\_bytes" : 107389383, "estimated\_size" : "102.4mb", "overhead" : 1.03 } } } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a524088e-b78f-45cd-b2e7-ed98ac682bf3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a524088e-b78f-45cd-b2e7-ed98ac682bf3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 3, 2014, 7:48pm UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/2 "2014-06-03T19:48:29Z")
What ES version is this? Your segment count is very high (\>1000) which is not efficient. Maybe index.codec.bloom.load: false can help reducing heap mem usage. [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGLMa1zV-8JM5zejxsVoTHmW93Tbt\_Vs9hjD5t0NcOd4A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGLMa1zV-8JM5zejxsVoTHmW93Tbt_Vs9hjD5t0NcOd4A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Adam\_Georgiou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_georgiou/32/1409_2.png) [@Adam\_Georgiou](https://discuss.elastic.co/u/Adam_Georgiou)\ **Post date:** [June 4, 2014, 5:20pm UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/3 "2014-06-04T17:20:16Z")
Thanks for the response, Jörg. The version is 1.1.0, and I'll take a look at that bloom filter setting. -Adam On Tuesday, June 3, 2014 3:48:37 PM UTC-4, Jörg Prante wrote: > What ES version is this? > > Your segment count is very high (\>1000) which is not efficient. > > Maybe index.codec.bloom.load: false can help reducing heap mem usage. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html) > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aeaf66df-76a0-41ab-a17d-f41a01473912%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aeaf66df-76a0-41ab-a17d-f41a01473912%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jegansp](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@jegansp](https://discuss.elastic.co/u/jegansp)\ **Post date:** [June 5, 2014, 5:30am UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/4 "2014-06-05T05:30:25Z")
Is this related to [https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779) We are also facing this issue. Our version is also 1.1.0. After running fine for few days, the cluster slowly starts facing memory issues. At some point of time it spends almost 90% of the time in garbage collection with full GCs with each on taking more than 15 secs. Tuning jvm parameters didn't help us either. We have a two node cluster with 120+ indexes. In our case segments count is too high (one node has 16000+ and other has 25000+). Optimizing indexes API doesn't return either as indicated here [https://groups.google.com/forum/#!topic/elasticsearch/kqTRRADQBwc](https://groups.google.com/forum/#!topic/elasticsearch/kqTRRADQBwc) I am going to try with these settings to see if the situation improves. > But, not sure if this will merge existing segments. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 6, 2014, 11:44am UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/5 "2014-06-06T11:44:38Z")
Heap use is very dependant on your setup. How big are your indexes? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 5 June 2014 15:30, jegansp [jegansp@gmail.com](mailto:jegansp@gmail.com) wrote: > Is this related to > [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779) > > We are also facing this issue. Our version is also 1.1.0. After running > fine > for few days, the cluster slowly starts facing memory issues. At some point > of time it spends almost 90% of the time in garbage collection with full > GCs > with each on taking more than 15 secs. Tuning jvm parameters didn't help us > either. > > We have a two node cluster with 120+ indexes. In our case segments count is > too high (one node has 16000+ and other has 25000+). > > Optimizing indexes API doesn't return either as indicated here > [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/kqTRRADQBwc) > > I am going to try with these settings to see if the situation improves. > > [Elasticsearch configuration for high sustainable bulk feed · GitHub](https://gist.github.com/jprante/10666960) > > But, not sure if this will merge existing segments. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/What-s-using-memory-in-ElasticSearch-Details-to-follow-tp4056989p4057097.html](http://elasticsearch-users.115913.n3.nabble.com/What-s-using-memory-in-ElasticSearch-Details-to-follow-tp4056989p4057097.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1401946225557-4057097.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1401946225557-4057097.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZNcSnB0VD-OqcrUDzN24\_aS02K4PkCxD7Wwj8XxUAu8Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZNcSnB0VD-OqcrUDzN24_aS02K4PkCxD7Wwj8XxUAu8Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 6, 2014, 11:49am UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/6 "2014-06-06T11:49:34Z")
No, the settings will not merge existing segments unless you call \_optimize action via API. And take some patience. Thousands of segments take time - also, they need quite few memory resources to merge... I suggest backup your data first, to stay safe if the merging fails / aborts... Jörg On Thu, Jun 5, 2014 at 7:30 AM, jegansp [jegansp@gmail.com](mailto:jegansp@gmail.com) wrote: > Is this related to > [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779) > > We are also facing this issue. Our version is also 1.1.0. After running > fine > for few days, the cluster slowly starts facing memory issues. At some point > of time it spends almost 90% of the time in garbage collection with full > GCs > with each on taking more than 15 secs. Tuning jvm parameters didn't help us > either. > > We have a two node cluster with 120+ indexes. In our case segments count is > too high (one node has 16000+ and other has 25000+). > > Optimizing indexes API doesn't return either as indicated here > [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/kqTRRADQBwc) > > I am going to try with these settings to see if the situation improves. > > [Elasticsearch configuration for high sustainable bulk feed · GitHub](https://gist.github.com/jprante/10666960) > > But, not sure if this will merge existing segments. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/What-s-using-memory-in-ElasticSearch-Details-to-follow-tp4056989p4057097.html](http://elasticsearch-users.115913.n3.nabble.com/What-s-using-memory-in-ElasticSearch-Details-to-follow-tp4056989p4057097.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1401946225557-4057097.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1401946225557-4057097.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHyMHc242rWrLAihDEkz6GUfKBJbUWJEkqcsU2\_VmDxoA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHyMHc242rWrLAihDEkz6GUfKBJbUWJEkqcsU2_VmDxoA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jegansp](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@jegansp](https://discuss.elastic.co/u/jegansp)\ **Post date:** [June 6, 2014, 2:08pm UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/7 "2014-06-06T14:08:29Z")
Thanks for your replies Mark and Jorg. My index size is around 500GB. After using the settings (provided in my last post) initially the segments count came down to just around 200 (without doing optimize calls), but after some time it started increasing and now it stands around 2000+ in each node. Is this expected? I would surely try with the optimise APIs for each index. Is there any other settings I need to be aware of? Thanks, Jegan ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 7, 2014, 6:37pm UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/8 "2014-06-07T18:37:52Z")
Maybe the segment count is just counting new segments as they are created... can you look into the data folders to examine if the segment file count is still high? And can you verify if the settings are really active... not sure what's going on without seeing details. The \_optimize call takes a parameter max\_num\_segments, you should maybe start with 50 or so if you have 1000s of segments. When \_optimize runs, you can check the number of active segment merge threads in the nodes info for monitoring progress (or use a monitoring tool) Jörg On Fri, Jun 6, 2014 at 4:08 PM, jegansp [jegansp@gmail.com](mailto:jegansp@gmail.com) wrote: > Thanks for your replies Mark and Jorg. > > My index size is around 500GB. > > After using the settings (provided in my last post) initially the segments > count came down to just around 200 (without doing optimize calls), but > after > some time it started increasing and now it stands around 2000+ in each > node. > Is this expected? > > I would surely try with the optimise APIs for each index. Is there any > other > settings I need to be aware of? > > Thanks, > > Jegan > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/What-s-using-memory-in-ElasticSearch-Details-to-follow-tp4056989p4057223.html](http://elasticsearch-users.115913.n3.nabble.com/What-s-using-memory-in-ElasticSearch-Details-to-follow-tp4056989p4057223.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1402063709309-4057223.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1402063709309-4057223.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFM7GF\_bZ\_-b\_mS0tFj1jGntM%2BV8LAgTgF6wD4Z5SE7sQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFM7GF_bZ_-b_mS0tFj1jGntM%2BV8LAgTgF6wD4Z5SE7sQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:23am UTC](https://discuss.elastic.co/t/whats-using-memory-in-elasticsearch-details-to-follow/17893/9 "2017-07-06T01:23:59Z")