# Падает Эластик 5.2.1, те же запросы Elastiс 2.4.1 отрабатывает влёт **URL:** \ **Category:** Вопросы на русском языке\ **Created:** [February 16, 2017, 3:23pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357 "2017-02-16T15:23:41Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 3:23pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/1 "2017-02-16T15:23:41Z")
Всех приветствую! Есть 2 параллельно работающих кластера - старый, работающий на версии 2.4.1 и новый на 5.2.1 (ставил 5.0, 5.1, 5.2 версии). Данные на них записываются одинаково, через два output в logstash. Проблема заключается в том, что есть запросы, которые 2.4.1 отрабатывает за 4.5 секунды, а 5.х виснет глухо, сжирает весь HeapSize и либо отваливается, либо перестаёт отвечать несколько нод. Кластер 2.4.1 содержит 13 серверов (3 мастера и 10 дата-инстансов) 8CPU/64GbRam/916Gb SSD/Ubuntu 14.04/Java 1.8 Кластер 5.2.1 содержит 3 сервера (3 мастера и 9 дата-инстансов) 32CPU/128Gb/3x1Tb SSD + 1HDD/ ubuntu 16.04/ Java 1.8 На серверах с версией 2.4.1 запущен всего один процесс эластика. (30Гб памяти на процесс эластика) На серверах с версией 5.2.1 запущено 4 процесса эластика. (один мастер с 8Гб памяти на hdd и три дата-инстанса с 24Гб памяти и своим SSD на 1Тб) Суммарный объём данных в обоих кластерах: > ``` > Nodes: 12 > Disk Available: 1TB / 2TB (63.62%) > JVM Heap: 23.89% (57GB / 238GB) > Indices: 164 > Documents: 4,553,799,983 > Disk Usage: 3TB > Primary Shards: 658 > Replica Shards: 199 > > ``` Для примера есть запрос, который смотрит в индекс views-2017.02.11 объёмом: > ``` > 44,525,722 Documents > 33.8GB Primary Size > 33.8GB Total Size > 4 Total Shards > > ``` ``` curl -XGET 'http://CLUSTERNAME:9200/views-2017.02.11/_search?pretty' -d '{ "size" : 0, "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*", "fields" : [], "use_dis_max" : true, "tie_breaker" : 0.0, "default_operator" : "or", "auto_generate_phrase_queries" : false, "fuzziness" : "AUTO", "fuzzy_prefix_length" : 0, "fuzzy_max_expansions" : 50, "phrase_slop" : 0, "analyze_wildcard" : true, "escape" : false, "boost" : 1.0 } }, { "range" : { "@timestamp" : { "from" : 1486789200000, "to" : 1486799200000, "include_lower" : true, "include_upper" : true, "format" : "epoch_millis", "boost" : 1.0 } } } ], "disable_coord" : false, "adjust_pure_negative" : true, "boost" : 1.0 } }, "_source" : { "includes" : [], "excludes" : [] }, "aggregations" : { "2" : { "terms" : { "field" : "ip", "size" : 5, "min_doc_count" : 1, "shard_min_doc_count" : 0, "show_term_doc_count_error" : false, "order" : [ { "1" : "desc" }, { "_term" : "asc" } ] }, "aggregations" : { "1" : { "cardinality" : { "field" : "ip" } } } } } } }' ``` На 2.4.1 он отрабатывает: ``` # time sh elasticdie.sh { "took" : 4105, "timed_out" : false, "_shards" : { "total" : 4, "successful" : 4, "failed" : 0 }, "hits" : { "total" : 4241136, "max_score" : 0.0, "hits" : [] }, "aggregations" : { "2" : { "doc_count_error_upper_bound" : -1, "sum_other_doc_count" : 4241121, "buckets" : [ { "key" : "1.0.164.97", "doc_count" : 5, "1" : { "value" : 1 } }, { "key" : "1.0.167.140", "doc_count" : 4, "1" : { "value" : 1 } }, { "key" : "1.0.192.154", "doc_count" : 1, "1" : { "value" : 1 } }, { "key" : "1.0.192.56", "doc_count" : 2, "1" : { "value" : 1 } }, { "key" : "1.0.195.13", "doc_count" : 1, "1" : { "value" : 1 } } ] } } } real 0m4.298s user 0m0.004s sys 0m0.000s ``` На новом идёт тайм-аут: ``` # time sh elasticdie.sh curl: (52) Empty reply from server real 1m32.485s user 0m0.004s sys 0m0.000s ``` ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 3:25pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/2 "2017-02-16T15:25:23Z")
Конфиг Эластика (Пытался менять indices и thread\_pool - безрезультатно): > bootstrap.memory\_lock: true > cluster.name: "elastic" > node.name: "fatsod" > node.master: false > node.data: true > network.host: "fatsod" > network.bind\_host: 172.16.1.68 > transport.tcp.port: 9311 > transport.tcp.compress: true > http.port: 9211 > http.max\_content\_length: 100mb > http.cors.allow-origin: "\*" > http.cors.enabled: true > http.type: netty3 > discovery.zen.minimum\_master\_nodes: 3 > discovery.zen.ping.unicast.hosts: ["fatsod01:9300", "fatsod02:9300", "fatsod03:9300"] > thread\_pool.search.size: 20 > thread\_pool.search.queue\_size: 1000 > thread\_pool.bulk.size: 24 > thread\_pool.bulk.queue\_size: 3000 > thread\_pool.index.size: 20 > thread\_pool.index.queue\_size: 1000 > indices.store.throttle.max\_bytes\_per\_sec: 2048mb > indices.breaker.request.limit: 30% > indices.breaker.fielddata.limit: 85% > indices.fielddata.cache.size: 75% > indices.queries.cache.size: 50% > indices.requests.cache.expire: 60s > xpack.security.enabled: false В логах вижу только следующее (): > [2017-02-16T17:41:58,372][WARN][o.e.c.s.ClusterService] [fatsod11] cluster state update task [master\_failed ({fatsod02}{1X7Q8\_\_sQ2OoxG46NhL4Lg}{OiYz1VL6TOmFgnwSAN5lew}{fatsod02}{172.16.1.68:9300})] took [3.2m > ] above the warn threshold of 30s > [2017-02-16T17:41:58,380][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [fatsod11] fatal error in thread [elasticsearch[fatsod11][search][T#11]], exiting > java.lang.OutOfMemoryError: Java heap space > at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:99) ~[elasticsearch-5.2.1.jar:5.2.1] > ... 8 more Либо такая ошибка: > ==\> /storage/fatsod03/logs/elastic.log \<== > [2017-02-15T18:07:39,936][DEBUG][o.e.a.s.TransportSearchAction] [fatsod03] [views-2017.02.15][1], node[lKYVTneGTdyz7kwZSoWqbA], [P], s[STARTED], a[id=nu7bHtOkSUu0Xn2naQkYdw]: Failed to execute [SearchRequest{searchType=QUERY\_THEN\_FETCH, indices=[views-2017.02.15], indicesOptions=IndicesOptions[id=39, ignore\_unavailable=true, allow\_no\_indices=true, expand\_wildcards\_open=true, expand\_wildcards\_closed=false, allow\_alisases\_to\_multiple\_indices=true, forbid\_closed\_indices=true], types=, routing='null', preference='1487171196736', requestCache=null, scroll=null, source={ > "size" : 0, > "query" : { > "bool" : { > "must" : [ > { > "query\_string" : { > "query" : "\*", > "fields" : , > "use\_dis\_max" : true, > "tie\_breaker" : 0.0, > "default\_operator" : "or", > "auto\_generate\_phrase\_queries" : false, > "max\_determined\_states" : 10000, > "enable\_position\_increment" : true, > "fuzziness" : "AUTO", > "fuzzy\_prefix\_length" : 0, > "fuzzy\_max\_expansions" : 50, > "phrase\_slop" : 0, > "analyze\_wildcard" : true, > "escape" : false, > "split\_on\_whitespace" : true, > "boost" : 1.0 > } > }, > { > "range" : { > "@timestamp" : { > "from" : 1487170353810, > "to" : 1487171253812, > "include\_lower" : true, > "include\_upper" : true, > "format" : "epoch\_millis", > "boost" : 1.0 > } > } > } > ], > "disable\_coord" : false, > "adjust\_pure\_negative" : true, > "boost" : 1.0 > } > }, > "\_source" : { > "includes" : , > "excludes" : > }, > "aggregations" : { > "2" : { > "terms" : { > "field" : "ip", > "size" : 5, > "min\_doc\_count" : 1, > "shard\_min\_doc\_count" : 0, > "show\_term\_doc\_count\_error" : false, > "order" : [ > { > "1" : "desc" > }, > { > "\_term" : "asc" > } > ] > }, > "aggregations" : { > "1" : { > "cardinality" : { > "field" : "ip" > } > } > } > } > } > }}] lastShard [true] > org.elasticsearch.transport.RemoteTransportException: [fatsod09][172.16.1.69:9309][indices:data/read/search[phase/query]] > Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query] > Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [request] Data too large, data for [\] would be larger than limit of [10187558092/9.4gb] > at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97) ~[elasticsearch-5.2.1.jar:5.2.1] Господа, прошу Вашей помощи! Уже не знаю кого спрашивать... =( ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 3:34pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/3 "2017-02-16T15:34:45Z")
![](https://us1.discourse-cdn.com/elastic/original/2X/9/99441c4da7bc5896c914361e1e804194c4dacf2d.jpg) ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 16, 2017, 3:45pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/4 "2017-02-16T15:45:00Z")
А какой меппинг у поля "ip" в том и в другом кластере? ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 3:49pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/5 "2017-02-16T15:49:48Z")
> [@Vladimir\_Dyakov](#): > > views-2017.02.11 Старый: > ``` > "ip" : { > "type" : "string", > "index" : "not_analyzed" > }, > > ``` Новый, странно, но > ``` > "ip" : { > "type" : "keyword" > }, > > ``` Походу из-за > ``` > "dynamic_templates" : [ > { > "string_fields" : { > "match" : "*", > "match_mapping_type" : "string", > "mapping" : { > "index" : "not_analyzed", > "omit_norms" : true, > "type" : "keyword" > } > } > } > ], > > ``` ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 16, 2017, 4:43pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/6 "2017-02-16T16:43:56Z")
"string" больше не существует в 5.0, так что тут все нормально. Давайте разделим запрос на 3 части и будем каждую часть изучать отдельно. Запустите следующее 3 запроса и скажите, какие из них вызывают проблемы ```auto curl -XGET 'http://CLUSTERNAME:9200/views-2017.02.11/_search?pretty' -d '{ "size": 0, "query": { "bool": { "must": [{ "query_string": { "query": "*", "fields": [], "use_dis_max": true, "tie_breaker": 0.0, "default_operator": "or", "auto_generate_phrase_queries": false, "fuzziness": "AUTO", "fuzzy_prefix_length": 0, "fuzzy_max_expansions": 50, "phrase_slop": 0, "analyze_wildcard": true, "escape": false, "boost": 1.0 } }, { "range": { "@timestamp": { "from": 1486789200000, "to": 1486799200000, "include_lower": true, "include_upper": true, "format": "epoch_millis", "boost": 1.0 } } }], "disable_coord": false, "adjust_pure_negative": true, "boost": 1.0 } }, "_source": { "includes": [], "excludes": [] } } ' ``` ```auto curl -XGET 'http://CLUSTERNAME:9200/views-2017.02.11/_search?pretty' -d '{ "size": 0, "query": { "range": { "@timestamp": { "from": 1486789200000, "to": 1486799200000, "include_lower": true, "include_upper": true, "format": "epoch_millis", "boost": 1.0 } } }, "_source": { "includes": [], "excludes": [] }, "aggregations": { "2": { "terms": { "field": "ip", "size": 5, "min_doc_count": 1, "shard_min_doc_count": 0, "show_term_doc_count_error": false, "order": [{ "1": "desc" }, { "_term": "asc" }] } } } } ' ``` ```auto curl -XGET 'http://CLUSTERNAME:9200/views-2017.02.11/_search?pretty' -d '{ "size": 0, "query": { "range": { "@timestamp": { "from": 1486789200000, "to": 1486799200000, "include_lower": true, "include_upper": true, "format": "epoch_millis", "boost": 1.0 } } }, "_source": { "includes": [], "excludes": [] }, "aggregations": { "2": { "terms": { "field": "ip", "size": 5, "min_doc_count": 1, "shard_min_doc_count": 0, "show_term_doc_count_error": false, "collect_mode" : "breadth_first", "order": [{ "1": "desc" }, { "_term": "asc" }] }, "aggregations": { "1": { "cardinality": { "field": "ip" } } } } } } ' ``` ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 5:49pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/7 "2017-02-16T17:49:33Z")
Спасибо за поддержку! первый запрос отработал за 0.2 сек > # time sh 1.sh > > { > "took" : 81, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 4241136, > "max\_score" : 0.0, > "hits" : > } > } > real 0m0.212s > user 0m0.004s > sys 0m0.000s Второй запрос дал ошибку: > #time sh 1.sh > { > "error" : { > "root\_cause" : [ > { > "type" : "aggregation\_execution\_exception", > "reason" : "Invalid term-aggregator order path [1]. Unknown aggregation [1]" > } > ], > "type" : "search\_phase\_execution\_exception", > "reason" : "all shards failed", > "phase" : "query", > "grouped" : true, > "failed\_shards" : [ > { > "shard" : 0, > "index" : "views-2017.02.11", > "node" : "lKYVTneGTdyz7kwZSoWqbA", > "reason" : { > "type" : "aggregation\_execution\_exception", > "reason" : "Invalid term-aggregator order path [1]. Unknown aggregation [1]" > } > } > ], > "caused\_by" : { > "type" : "aggregation\_execution\_exception", > "reason" : "Invalid term-aggregator order path [1]. Unknown aggregation [1]" > } > }, > "status" : 500 > } > real 0m1.870s > user 0m0.000s > sys 0m0.004s Если убрать сортировку по первой агрегации ("order": [{"1": "desc"},) то выполняется за 1.9 сек: > # time sh 1.sh > > { > "took" : 1815, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 4241136, > "max\_score" : 0.0, > "hits" : > }, > "aggregations" : { > "2" : { > "doc\_count\_error\_upper\_bound" : 0, > "sum\_other\_doc\_count" : 4241121, > "buckets" : [ > { > "key" : "1.0.164.97", > "doc\_count" : 5 > }, > { > "key" : "1.0.167.140", > "doc\_count" : 4 > }, > { > "key" : "1.0.192.154", > "doc\_count" : 1 > }, > { > "key" : "1.0.192.56", > "doc\_count" : 2 > }, > { > "key" : "1.0.195.13", > "doc\_count" : 1 > } > ] > } > } > } > real 0m1.948s > user 0m0.004s > sys 0m0.000s Третий запрос положил одину ноду кластера: > # time sh 1.sh > > curl: (52) Empty reply from server > real 5m24.532s > user 0m0.008s > sys 0m0.004s Последние записи в логе ноды, которая ушла в ступор, при этом JAVA не упала и продолжала слушать порт не давая подсоединится к себе: > [2017-02-16T20:29:14,113][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][9959] overhead, spent [12.7s] collecting in the last [12.8s] > [2017-02-16T20:29:18,433][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][9960] overhead, spent [4.2s] collecting in the last [4.3s] > [2017-02-16T20:29:42,904][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][old][9961][16] duration [12.1s], collections [1]/[12.2s], total [12.1s]/[2.7m], memory [23.7gb]-\>[23.8gb]/[23.8gb], all\_pools {[young] [1.4gb]-\>[1.4g > b]/[1.4gb]}{[survivor] [159.8mb]-\>[179mb]/[191.3mb]}{[old] [22.1gb]-\>[22.1gb]/[22.1gb]} > [2017-02-16T20:29:42,905][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][9961] overhead, spent [12.1s] collecting in the last [12.2s] > [2017-02-16T20:30:04,701][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][old][9962][18] duration [24.1s], collections [2]/[24.1s], total [24.1s]/[3.1m], memory [23.8gb]-\>[23.7gb]/[23.8gb], all\_pools {[young] [1.4gb]-\>[1.4g > b]/[1.4gb]}{[survivor] [179mb]-\>[172.8mb]/[191.3mb]}{[old] [22.1gb]-\>[22.1gb]/[22.1gb]} > [2017-02-16T20:30:04,701][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][9962] overhead, spent [24.1s] collecting in the last [24.1s] > [2017-02-16T20:30:16,690][INFO][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][old][9963][20] duration [16.6s], collections [2]/[16.7s], total [16.6s]/[3.4m], memory [23.7gb]-\>[23.8gb]/[23.8gb], all\_pools {[young] [1.4gb]-\>[1.4g > b]/[1.4gb]}{[survivor] [172.8mb]-\>[180.6mb]/[191.3mb]}{[old] [22.1gb]-\>[22.1gb]/[22.1gb]} > [2017-02-16T20:30:16,691][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][9963] overhead, spent [16.6s] collecting in the last [16.7s] > [2017-02-16T20:30:31,904][INFO][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][old][9964][23] duration [20.2s], collections [3]/[20.2s], total [20.2s]/[3.7m], memory [23.8gb]-\>[23.8gb]/[23.8gb], all\_pools {[young] [1.4gb]-\>[1.4g > b]/[1.4gb]}{[survivor] [180.6mb]-\>[190.8mb]/[191.3mb]}{[old] [22.1gb]-\>[22.1gb]/[22.1gb]} > [2017-02-16T20:30:31,906][WARN][o.e.m.j.JvmGcMonitorService] [fatsod08] [gc][9964] overhead, spent [20.2s] collecting in the last [20.2s] ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 16, 2017, 6:14pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/8 "2017-02-16T18:14:01Z")
А вот такой вопрос еще, что вы пытаетесь этим запросом получить? И к какой ноде вы обращаетесь через curl? Это та нода, которая падает? ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 7:04pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/9 "2017-02-16T19:04:05Z")
Этот запрос делает Kibana, Из 3х физ серверов запрос либо на первй либо на второй либо на третий. Как правило инстансы отваливаются рандомно. Может отвалится на той же ноде, где и был запрос, может на другой. Как правило, это никак не выловить по закономерности. ![](https://us1.discourse-cdn.com/elastic/original/2X/1/10b131792494f4050882c9511daf81b63b9e980a.jpg) ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 7:13pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/10 "2017-02-16T19:13:39Z")
Сейчас выполнил запрос, который на картинке кибаны и опять всё отвалилось. На этот раз интереснее. На одном серере инстанс завис, а на другом упала сама ява со следующей ошибкой: > [2017-02-16T22:03:48,199][INFO][o.e.m.j.JvmGcMonitorService] [fatsod07] [gc][old][5336][27] duration [5.8s], collections [1]/[5.8s], total [5.8s]/[2.5m], memory [23.7gb]-\>[23.8gb]/[23.8gb], all\_pools {[young] [1.4gb]-\>[1.4gb]/ > [1.4gb]}{[survivor] [171.7mb]-\>[181.6mb]/[191.3mb]}{[old] [22.1gb]-\>[22.1gb]/[22.1gb]} > [2017-02-16T22:03:48,199][WARN][o.e.m.j.JvmGcMonitorService] [fatsod07] [gc][5336] overhead, spent [5.8s] collecting in the last [5.8s] > [2017-02-16T22:03:56,502][INFO][o.e.m.j.JvmGcMonitorService] [fatsod07] [gc][old][5337][28] duration [8.2s], collections [1]/[8.3s], total [8.2s]/[2.7m], memory [23.8gb]-\>[23.8gb]/[23.8gb], all\_pools {[young] [1.4gb]-\>[1.4gb]/ > [1.4gb]}{[survivor] [181.6mb]-\>[189.9mb]/[191.3mb]}{[old] [22.1gb]-\>[22.1gb]/[22.1gb]} > [2017-02-16T22:03:56,502][WARN][o.e.m.j.JvmGcMonitorService] [fatsod07] [gc][5337] overhead, spent [8.2s] collecting in the last [8.3s] > [2017-02-16T22:05:50,867][WARN][o.e.m.j.JvmGcMonitorService] [fatsod07] [gc][5338] overhead, spent [1.9m] collecting in the last [1.9m] > [2017-02-16T22:05:50,953][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [fatsod07] fatal error in thread [elasticsearch[fatsod07][refresh][T#6]], exiting > java.lang.OutOfMemoryError: Java heap space > at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:89) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.util.fst.FST.(FST.java:295) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.util.fst.Builder.(Builder.java:171) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$PendingBlock.compileIndex(BlockTreeTermsWriter.java:457) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:635) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.finish(BlockTreeTermsWriter.java:936) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:347) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:443) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32] > at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) ~[elasticsearch-5.2.1.jar:5.2.1] > at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) ~[elasticsearch-5.2.1.jar:5.2.1] > at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:693) ~[elasticsearch-5.2.1.jar:5.2.1] > at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) ~[elasticsearch-5.2.1.jar:5.2.1] > at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:835) ~[elasticsearch-5.2.1.jar:5.2.1] > at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:746) ~[elasticsearch-5.2.1.jar:5.2.1] ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 16, 2017, 7:42pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/11 "2017-02-16T19:42:46Z")
Я понимаю, что Kibana, но кто-то этот запрос в Kibana зачем-то сформулировал, правильно? Мне хотелось бы понять зачем. А можно как-нибудь на heapdump одного из oom-ных нод посмотреть? ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 16, 2017, 7:54pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/12 "2017-02-16T19:54:16Z")
> [@Igor\_Motov](#): > > Я понимаю, что Kibana, но кто-то этот запрос в Kibana зачем-то сформулировал, правильно? Мне хотелось бы понять зачем. Мой коллега ответил: > ``` > Yurii Gagarin, [16.02.17 21:37] > [In reply to Vladimir Dyakov] > пытался вывести количество уников через уникальные значения айпи+юзерагент > > ``` > ``` > Yurii Gagarin, [16.02.17 21:37] > видимо пытался не правильно) > > ``` > ``` > Yurii Gagarin, [16.02.17 21:38] > потом уже откинул лишние поля, получалось что она ругалась на метрику - уники по айпи, с агрегацией по уникам по айпи. > > ``` > ``` > Yurii Gagarin, [16.02.17 21:38] > запрос сама кибана делала > > ``` У меня вопрос тогда, если Вас не затруднит, скажите более детально, как замутить этот heapdump и завтра я его сделаю. Или это просто полный ДЕФОЛТНЫЙ лог с нод? =) ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 16, 2017, 8:51pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/13 "2017-02-16T20:51:46Z")
heapdump тут - [How to Capture a Heap Dump from a Running JVM - Logstash](https://discuss.elastic.co/t/how-to-capture-a-heap-dump-from-a-running-jvm-logstash/85) только в инетернет его не выкладываейте, потому что не понятно какими данными весь этот heap заполнен. Либо сами посмотрите куда память уходит, либо мне ссылку пришлите личным сообщением. Ваш запрос выдает количество уникальных IP для каждого IP. Если у вас в одной записи всегда один IP, то ответ на этот запрос будет всегда один IP для каждого IP. ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 20, 2017, 2:44pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/14 "2017-02-20T14:44:50Z")
Спасибо большое за отклик! Если Вас не затруднит, Вы не могли бы помочь с анализом лога? Я его Вам в личку скинул. Я в логе вижу только то, что вся память была забита типом byte[] (25Гб) ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 20, 2017, 3:39pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/15 "2017-02-20T15:39:47Z")
Да, дамп получили. Пытаемся разобраться, но пока что не очень понятно, что там происходит потому что это byte[] в локальной переменной и по дампу не понятно какой именно. А вы не поищите логи, у вас каких-нибудь ошибок с `java.lang.OutOfMemoryError` но с другим стеком еще не было? ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 20, 2017, 4:01pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/16 "2017-02-20T16:01:15Z")
Сегодня завис мастер, есть дамп. На нём закончились 8Гб выделенных, при этом процесс жил и в лог ничего не писал. По логу видно, что в 18:26:17 он завис и дампил, одно ядро процессора прогружено было на 100%. В 18:37:34 я его kill -9 убил, т.к. рестарт не работал через systemd и стартанул заново. Могу с него лог прислать, но это не воркер, хотя тоже должен рыбл ругнуться на большие данные и скинуть выполнение запроса, а не зависать, забирая с собой весь кластер. =( ``` [2017-02-20T18:23:58,263][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][286540] overhead, spent [25s] collecting in the last [26.5s] [2017-02-20T18:24:20,245][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][old][286541][18] duration [20.4s], collections [1]/[20.4s], total [20.4s]/[4.3m], memory [7.8gb]->[7.8gb]/[7.8gb], all_pools {[young] [1.4gb]->[1.4gb ]/[1.4gb]}{[survivor] [186.8mb]->[188.2mb]/[191.3mb]}{[old] [6.1gb]->[6.1gb]/[6.1gb]} [2017-02-20T18:24:20,246][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][286541] overhead, spent [20.4s] collecting in the last [20.4s] [2017-02-20T18:24:43,861][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][old][286542][19] duration [25.1s], collections [1]/[25.1s], total [25.1s]/[4.8m], memory [7.8gb]->[7.8gb]/[7.8gb], all_pools {[young] [1.4gb]->[1.4gb ]/[1.4gb]}{[survivor] [188.2mb]->[189.4mb]/[191.3mb]}{[old] [6.1gb]->[6.1gb]/[6.1gb]} [2017-02-20T18:24:43,862][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][286542] overhead, spent [25.1s] collecting in the last [25.1s] [2017-02-20T18:26:17,346][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][old][286543][21] duration [48.1s], collections [2]/[48.1s], total [48.1s]/[5.6m], memory [7.8gb]->[7.8gb]/[7.8gb], all_pools {[young] [1.4gb]->[1.4gb ]/[1.4gb]}{[survivor] [189.4mb]->[191.1mb]/[191.3mb]}{[old] [6.1gb]->[6.1gb]/[6.1gb]} [2017-02-20T18:26:17,346][WARN][o.e.m.j.JvmGcMonitorService] [fatsod03] [gc][286543] overhead, spent [48.1s] collecting in the last [48.1s] [2017-02-20T18:37:34,369][INFO][o.e.n.Node] [fatsod03] initializing ... [2017-02-20T18:37:34,591][INFO][o.e.e.NodeEnvironment] [fatsod03] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [419gb], net total_space [458.3gb], spins? [possibly], types [ext4] [2017-02-20T18:37:34,591][INFO][o.e.e.NodeEnvironment] [fatsod03] heap size [7.8gb], compressed ordinary object pointers [true] ``` ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 20, 2017, 4:05pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/17 "2017-02-20T16:05:38Z")
Мне вот такая ошибка нужна, только с другим стеком: ```auto [2017-02-16T22:05:50,953][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [fatsod07] fatal error in thread [elasticsearch[fatsod07][refresh][T#6]], exitingjava.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:89) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]...... ``` Вы не поищите по логам строку `OutOfMemoryError`, может, еще такие ошибки были? ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 20, 2017, 4:18pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/18 "2017-02-20T16:18:46Z")
Что-то типа такого? [http://pastebin.com/V5MQarzp](http://pastebin.com/V5MQarzp) ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 20, 2017, 5:30pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/19 "2017-02-20T17:30:18Z")
Да, это то что надо. Будем разбираться. ---
**Author:** ![Vladimir\_Dyakov](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@Vladimir\_Dyakov](https://discuss.elastic.co/u/Vladimir_Dyakov)\ **Post date:** [February 25, 2017, 1:40pm UTC](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357/20 "2017-02-25T13:40:38Z")
Игорь, здравствуйте! Хотел поинтересоваться, получилось ли что-то придумать по нашему вопросу? Может ещё какую-то информацию неоходимо Вам дать? [Next page](https://discuss.elastic.co/t/5-2-1-elasti-2-4-1/75357.md?page=2)