# Данные из Logstash не попадают в Elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch/176263>\
**Category:** Вопросы на русском языке\
**Created:** [April 10, 2019, 3:53pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263 "2019-04-10T15:53:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![orsa](https://avatars.discourse-cdn.com/v4/letter/o/5daacb/32.png) [@orsa](https://discuss.elastic.co/u/orsa)\
**Post date:** [April 10, 2019, 3:53pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/1 "2019-04-10T15:53:11Z")

</div>

Доброго дня Коллеги,  
на одном из обновлений "сломался" Logstash, данные которые шли через него перестали попадать в ES (прямая запись в ES работает прекрасно)  
LS ставился из deb пакетов, он работает как сервис.  
если его остановить и запустить из cli с тем же кофигом - данные идут и обрабатываются нормально.  
как только LS стартует как служба - данных нет.  
подозреваю что по путались связи между конфигурационными файлами, pipeline и тд.  
Но пока картинка не сложилась.  
куда стоит посмотреть?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [April 10, 2019, 4:22pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/2 "2019-04-10T16:22:23Z")

</div>

> [@orsa](#):
>
> куда стоит посмотреть?

в логи Logstash

> **[Logstash Directory Layout | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/dir-layout.html#deb-layout)**

---

<div class="post-metadata">

**Author:** ![orsa](https://avatars.discourse-cdn.com/v4/letter/o/5daacb/32.png) [@orsa](https://discuss.elastic.co/u/orsa)\
**Post date:** [April 10, 2019, 4:47pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/3 "2019-04-10T16:47:04Z")

</div>

Игорь спасибо за ответ, посмотрел  
ничего криминального, стандартный старт системы.  
LS начинает слушать порты прописанные в секции input, и открывает конекторы секции output.  
Данные в сам ES не попадают.

```
[2019-04-10T15:52:44,338][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.7.1"}
[2019-04-10T15:53:02,928][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.dela
y"=>50}
[2019-04-10T15:53:04,079][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,437][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,525][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,530][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,570][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,621][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,638][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,646][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,646][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,663][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,679][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,694][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,707][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,708][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,728][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,739][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,751][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,762][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,762][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,769][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:06,206][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-04-10T15:53:06,310][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xe2922c5 run>"}
[2019-04-10T15:53:06,441][INFO][logstash.inputs.udp] Starting UDP listener {:address=>"0.0.0.0:2515"}
[2019-04-10T15:53:06,651][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-04-10T15:53:06,744][INFO][logstash.inputs.udp] UDP listener started {:address=>"0.0.0.0:2515", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-04-10T15:53:06,858][INFO][org.logstash.beats.Server] Starting server on port: 5044
[2019-04-10T15:53:08,254][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

может вопрос прав, ибо из cli он запускался с sudo  
`sudo /usr/share/logstash/bin/logstash -f test.in-out.conf`

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [April 10, 2019, 4:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/4 "2019-04-10T16:52:45Z")

</div>

> [@orsa](#):
>
> стандартный старт системы.

А вы уверены, что это не логи, которые остались после последнего запуска через командную строку?

> [@orsa](#):
>
> может вопрос прав,

Ну это легко проверить. Посмотрите на файлы и директории по списку из предыдущего ответа. Кто ими владеет и какие на них стоят права.

---

<div class="post-metadata">

**Author:** ![orsa](https://avatars.discourse-cdn.com/v4/letter/o/5daacb/32.png) [@orsa](https://discuss.elastic.co/u/orsa)\
**Post date:** [April 10, 2019, 10:33pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/5 "2019-04-10T22:33:01Z")

</div>

лог рестарта LS как сервиса по команде  
`sudo systemctl restart logstash.service && sudo tail -f /var/log/logstash/logstash-plain.log`

```
 [WARN][logstash.runner] SIGTERM received. Shutting down.
 [WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>32, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.8-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>29, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
 [ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
 [INFO][logstash.pipeline] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xe2922c5 run>"}
 [INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.7.1"}
 [INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
 [INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
 [WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
 [INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
 [WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
 [INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
 [INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
 [INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x141c02ea run>"}
 [INFO][logstash.inputs.udp] Starting UDP listener {:address=>"0.0.0.0:2515"}
 [INFO][org.logstash.beats.Server] Starting server on port: 5044
 [INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
 [INFO][logstash.inputs.udp] UDP listener started {:address=>"0.0.0.0:2515", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
 [INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

теперь лог старта LS из cli

```auto
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN] 2019-04-10 22:22:36.339 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO] 2019-04-10 22:22:36.352 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.7.1"}
[INFO] 2019-04-10 22:22:49.062 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO] 2019-04-10 22:22:49.914 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN] 2019-04-10 22:22:50.196 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO] 2019-04-10 22:22:50.556 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN] 2019-04-10 22:22:50.695 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO] 2019-04-10 22:22:50.697 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO] 2019-04-10 22:22:50.702 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN] 2019-04-10 22:22:50.706 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO] 2019-04-10 22:22:50.714 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN] 2019-04-10 22:22:50.714 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO] 2019-04-10 22:22:50.717 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO] 2019-04-10 22:22:51.154 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO] 2019-04-10 22:22:51.186 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7f40ce3e run>"}
[INFO] 2019-04-10 22:22:51.426 [[main]<beats] Server - Starting server on port: 5044
[INFO] 2019-04-10 22:22:51.446 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:2514"}
[INFO] 2019-04-10 22:22:51.464 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO] 2019-04-10 22:22:51.625 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:2514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[INFO] 2019-04-10 22:22:52.089 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
    {

```

права сейчас проверю отдельно

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [April 11, 2019, 7:16am UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/6 "2019-04-11T07:16:46Z")

</div>

Скорее всего, Не может у вас Logstash, работающий как сервер конфигурацию свою найти. У вас они в `/etc/logstash` находятся?

---

<div class="post-metadata">

**Author:** ![orsa](https://avatars.discourse-cdn.com/v4/letter/o/5daacb/32.png) [@orsa](https://discuss.elastic.co/u/orsa)\
**Post date:** [April 11, 2019, 10:31am UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/7 "2019-04-11T10:31:12Z")

</div>

да ,конфигурация там, точнее по стандартному pipeline  
/etc/logstash/conf.d/

в файле прописан путь  
administrator@dp-elk02:~$ sudo cat /etc/logstash/pipelines.yml

```
- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

```

Игорь меня что смущает,  
LS как сервис открывает входящие и исходящие соединения  
то что прописано в секциях input и output  
и дальше что то стопориться

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [April 11, 2019, 5:39pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/8 "2019-04-11T17:39:32Z")

</div>

> [@orsa](#):
>
> Игорь меня что смущает,  
> LS как сервис открывает входящие и исходящие соединения  
> то что прописано в секциях input и output  
> и дальше что то стопориться

Да. Согласен. Это действительно странно.

А какая была версия и на что вы ее обновили?

Откуда вы получаете информацию? И если это beats то какая там версия?

---

<div class="post-metadata">

**Author:** ![orsa](https://avatars.discourse-cdn.com/v4/letter/o/5daacb/32.png) [@orsa](https://discuss.elastic.co/u/orsa)\
**Post date:** [April 14, 2019, 11:30pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/9 "2019-04-14T23:30:07Z")

</div>

Добрый вечер Игорь,  
про версионность уже восстановить сложно, ибо это вспомогательные логи и с ними работают нечасто, плюс чистятся индексы  
высоко вероятно что сбой произошел в рамках обновления ветки 6.6  
6.6.0 - \> 6.6.1 -\> 6.6.3

через LS идет syslog сетевого оборудования.  
это либо настройки на оборудовании отправлять информацию на порт 2514, либо на локальный rsyslog (514), который в свою очередь пересылает на порт LS

все beat работают напрямую с ES и тут сбоев нет.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 12, 2019, 11:30pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch/176263/10 "2019-05-12T23:30:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.