# Поиск по большим данным

**URL:** <https://discuss.elastic.co/t/topic/179950>\
**Category:** Вопросы на русском языке\
**Created:** [May 7, 2019, 11:30am UTC](https://discuss.elastic.co/t/topic/179950 "2019-05-07T11:30:52Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![KaKaKuRt](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@KaKaKuRt](https://discuss.elastic.co/u/KaKaKuRt)\
**Post date:** [May 7, 2019, 11:30am UTC](https://discuss.elastic.co/t/topic/179950/1 "2019-05-07T11:30:52Z")

</div>

Здравствуйте.  
Использую ELK стек для сбора событий безопасности Windows.  
Настройки кластера:

> **curl [http://xx.xx.xC:9200/\_cat/nodes?v](http://xx.xx.xC:9200/_cat/nodes?v)**  
> ip heap.percent ram.percent cpu load\_1m load\_5m load\_15m node.role master name  
> xx.xx.xA 30 99 28 3.80 3.59 4.17 mdi - ELK-MD1 (дата-нода, CPU 12, ОЗУ 64 ГБ, HDD 2 ТБ)  
> xx.xx.xD 19 34 0 0.02 0.07 0.08 - - ELK-K (Kibana, CPU 6, ОЗУ 8 ГБ, HDD 200 ГБ)  
> xx.xx.xC 14 63 1 0.01 0.02 0.00 mi \* ELK-M (мастер-нода, CPU 12, ОЗУ 12 ГБ, HDD 100 ГБ)  
> xx.xx.xB 64 94 32 4.21 3.76 3.83 mdi - ELK-MD2 (дата-нода, CPU 12, ОЗУ 64 ГБ, HDD 2 ТБ)  
> **curl [http://xx.xx.xC:9200/\_cat/master](http://xx.xx.xC:9200/_cat/master)**  
> nwg2aXi1ScCiWRBAhbs3IA xx.xx.xC xx.xx.xC ELK-M

При поиске за 4 дня и более в Kibane возникает следующая ошибка:

> SearchError: Internal Server Error  
> at [http://xx.xx.xD:5601/bundles/commons.bundle.js:4:324908](http://xx.xx.xD:5601/bundles/commons.bundle.js:4:324908)  
> at processQueue ([http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:199687](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:199687))  
> at [http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:200650](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:200650)  
> at Scope.$digest ([http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:210412](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:210412))  
> at Scope.$apply ([http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:213219](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:213219))  
> at done ([http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:132717](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:132717))  
> at completeRequest ([http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:136329](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:136329))  
> at XMLHttpRequest.requestLoaded ([http://xx.xx.xD:5601/built\_assets/dlls/vendors.bundle.dll.js:427:135225](http://xx.xx.xD:5601/built_assets/dlls/vendors.bundle.dll.js:427:135225))

Есть предложения по настройки для оптимизации поиска? Хотелось бы выполнять поиск за месяцы.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [May 7, 2019, 1:23pm UTC](https://discuss.elastic.co/t/topic/179950/2 "2019-05-07T13:23:33Z")

</div>

Эта ошибка может означать все что угодно. В логах кибаны должна быть соответствующая запись, в которой должно быть больше информации о том, что на самом деле там происходит.

---

<div class="post-metadata">

**Author:** ![KaKaKuRt](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@KaKaKuRt](https://discuss.elastic.co/u/KaKaKuRt)\
**Post date:** [May 15, 2019, 5:25am UTC](https://discuss.elastic.co/t/topic/179950/3 "2019-05-15T05:25:24Z")

</div>

Извиняюсь за задержку.  
Включил логирование на Kiban-e.  
Но кроме 500 ошибки никакой полезной информации не увидел.

> {"type":"response","@timestamp":"2019-05-15T05:55:32Z","tags":[],"pid":2504,"method":"post","statusCode":500,"req":{"url":"/elasticsearch/\_msearch?rest\_total\_hits\_as\_int=true&ignore\_throttled=true","method":"post","headers":{"host":"xx.xx.xD:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86\_64; rv:66.0) Gecko/20100101 Firefox/66.0","accept":"application/json, text/plain, _/_","accept-language":"ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"[http://xx.xx.xD:5601/app/kibana","content-type":"application/x-ndjson","kbn-version":"7.0.1","content-length":"1287","connection":"keep-alive"},"remoteAddress":"ClientHost","userAgent":"ClientHost","referer":"http://xx.xx.xD:5601/app/kibana"},"res":{"statusCode":500,"responseTime":30022,"contentLength":9},"message":"POST](http://xx.xx.xD:5601/app/kibana%22,%22content-type%22:%22application/x-ndjson%22,%22kbn-version%22:%227.0.1%22,%22content-length%22:%221287%22,%22connection%22:%22keep-alive%22%7D,%22remoteAddress%22:%22ClientHost%22,%22userAgent%22:%22ClientHost%22,%22referer%22:%22http://xx.xx.xD:5601/app/kibana%22%7D,%22res%22:%7B%22statusCode%22:500,%22responseTime%22:30022,%22contentLength%22:9%7D,%22message%22:%22POST) /elasticsearch/\_msearch?rest\_total\_hits\_as\_int=true&ignore\_throttled=true 500 30022ms - 9.0B"}  
> {"type":"error","@timestamp":"2019-05-15T05:55:35Z","tags":[],"pid":2504,"level":"error","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n at /usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:436:11)\n at tryOnTimeout (timers.js:300:5)\n at listOnTimeout (timers.js:263:5)\n at Timer.processTimers (timers.js:223:10)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?rest\_total\_hits\_as\_int=true&ignore\_throttled=true","query":{"rest\_total\_hits\_as\_int":"true","ignore\_throttled":"true"},"pathname":"/elasticsearch/\_msearch","path":"/elasticsearch/\_msearch?rest\_total\_hits\_as\_int=true&ignore\_throttled=true","href":"/elasticsearch/\_msearch?rest\_total\_hits\_as\_int=true&ignore\_throttled=true"},"message":"Request Timeout after 30000ms"}

В логах ES (xx.xx.xA, xx.xx.xB, xx.xx.xC) ошибок нет.  
Но в кластер также входит нода с Kiban-oй (xx.xx.xD) для мониторинга. ES на ней установлен без ролей.

Где еще можно посмотреть в чем проблема?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [May 15, 2019, 3:01pm UTC](https://discuss.elastic.co/t/topic/179950/4 "2019-05-15T15:01:15Z")

</div>

А что в логах xx.xx.xD?

---

<div class="post-metadata">

**Author:** ![KaKaKuRt](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@KaKaKuRt](https://discuss.elastic.co/u/KaKaKuRt)\
**Post date:** [May 16, 2019, 3:09am UTC](https://discuss.elastic.co/t/topic/179950/5 "2019-05-16T03:09:55Z")

</div>

> xx.xx.xA 30 99 28 3.80 3.59 4.17 mdi - ELK-MD1 (дата-нода, CPU 12, ОЗУ 64 ГБ, HDD 2 ТБ)  
> xx.xx.xD 19 34 0 0.02 0.07 0.08 - - ELK-K (Kibana, CPU 6, ОЗУ 8 ГБ, HDD 200 ГБ)  
> xx.xx.xC 14 63 1 0.01 0.02 0.00 mi \* ELK-M (мастер-нода, CPU 12, ОЗУ 12 ГБ, HDD 100 ГБ)  
> xx.xx.xB 64 94 32 4.21 3.76 3.83 mdi - ELK-MD2 (дата-нода, CPU 12, ОЗУ 64 ГБ, HDD 2 ТБ)

Это сервер на котором развернута Kibana и ES без ролей.  
**xx.xx.xD** 19 34 0 0.02 0.07 0.08 - - **ELK-K (Kibana, CPU 6, ОЗУ 8 ГБ, HDD 200 ГБ**)  
ES

 ![ELK](https://us1.discourse-cdn.com/elastic/original/3X/f/2/f24b01233093bddda4cdcc70dda25402be2b6d1c.png)

Kibana

 ![Kibana](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d88aa86b577a6abfb73cea9cdbd918ab475952c8.png)

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [May 16, 2019, 7:36pm UTC](https://discuss.elastic.co/t/topic/179950/6 "2019-05-16T19:36:46Z")

</div>

А что в логах elasticsearch на xx.xx.xD?

---

<div class="post-metadata">

**Author:** ![KaKaKuRt](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@KaKaKuRt](https://discuss.elastic.co/u/KaKaKuRt)\
**Post date:** [May 17, 2019, 4:48am UTC](https://discuss.elastic.co/t/topic/179950/7 "2019-05-17T04:48:04Z")

</div>

> [@Igor\_Motov](#):
>
> А что в логах elasticsearch на xx.xx.xD?

Ошибок в логах ES на xx.xx.xD нет.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [May 17, 2019, 1:25pm UTC](https://discuss.elastic.co/t/topic/179950/8 "2019-05-17T13:25:14Z")

</div>

Странно, в логах ошибки 500 обычно что-то оставляют. Я думаю, что ошибка в том, что индексы за разные даты имеют разный мэппинг, и это вызывает проблемы при обработке запроса.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 14, 2019, 1:25pm UTC](https://discuss.elastic.co/t/topic/179950/9 "2019-06-14T13:25:19Z")

</div>

Closed on June 14, 2019, 1:25pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.