Данные из Logstash не попадают в Elasticsearch

orsa · April 10, 2019, 3:53pm

Доброго дня Коллеги,
на одном из обновлений "сломался" Logstash, данные которые шли через него перестали попадать в ES (прямая запись в ES работает прекрасно)
LS ставился из deb пакетов, он работает как сервис.
если его остановить и запустить из cli с тем же кофигом - данные идут и обрабатываются нормально.
как только LS стартует как служба - данных нет.
подозреваю что по путались связи между конфигурационными файлами, pipeline и тд.
Но пока картинка не сложилась.
куда стоит посмотреть?

Igor_Motov · April 10, 2019, 4:22pm

в логи Logstash

orsa · April 10, 2019, 4:47pm

Игорь спасибо за ответ, посмотрел
ничего криминального, стандартный старт системы.
LS начинает слушать порты прописанные в секции input, и открывает конекторы секции output.
Данные в сам ES не попадают.

[2019-04-10T15:52:44,338][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2019-04-10T15:53:02,928][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.dela
y"=>50}
[2019-04-10T15:53:04,079][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,437][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,525][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,530][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,570][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,621][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,638][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,646][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,646][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,663][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,679][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,694][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,707][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,708][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,728][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,739][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,751][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,762][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,762][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,769][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:06,206][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-04-10T15:53:06,310][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xe2922c5 run>"}
[2019-04-10T15:53:06,441][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2515"}
[2019-04-10T15:53:06,651][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-04-10T15:53:06,744][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2515", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-04-10T15:53:06,858][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-04-10T15:53:08,254][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

может вопрос прав, ибо из cli он запускался с sudo
sudo /usr/share/logstash/bin/logstash -f test.in-out.conf

Igor_Motov · April 10, 2019, 4:52pm

А вы уверены, что это не логи, которые остались после последнего запуска через командную строку?

Ну это легко проверить. Посмотрите на файлы и директории по списку из предыдущего ответа. Кто ими владеет и какие на них стоят права.

orsa · April 10, 2019, 10:33pm

лог рестарта LS как сервиса по команде
sudo systemctl restart logstash.service && sudo tail -f /var/log/logstash/logstash-plain.log

 [WARN ][logstash.runner          ] SIGTERM received. Shutting down.
 [WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>32, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.8-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>29, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
 [ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
 [INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xe2922c5 run>"}
 [INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
 [INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
 [INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
 [WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
 [INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
 [WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
 [INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
 [INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
 [INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x141c02ea run>"}
 [INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2515"}
 [INFO ][org.logstash.beats.Server] Starting server on port: 5044
 [INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
 [INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2515", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
 [INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

теперь лог старта LS из cli

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-04-10 22:22:36.339 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-04-10 22:22:36.352 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.7.1"}
[INFO ] 2019-04-10 22:22:49.062 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-04-10 22:22:49.914 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-04-10 22:22:50.196 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-04-10 22:22:50.556 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-04-10 22:22:50.695 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-04-10 22:22:50.697 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2019-04-10 22:22:50.702 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-04-10 22:22:50.706 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-04-10 22:22:50.714 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-04-10 22:22:50.714 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-04-10 22:22:50.717 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2019-04-10 22:22:51.154 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2019-04-10 22:22:51.186 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7f40ce3e run>"}
[INFO ] 2019-04-10 22:22:51.426 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2019-04-10 22:22:51.446 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:2514"}
[INFO ] 2019-04-10 22:22:51.464 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-04-10 22:22:51.625 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:2514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[INFO ] 2019-04-10 22:22:52.089 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
    {

права сейчас проверю отдельно

Igor_Motov · April 11, 2019, 7:16am

Скорее всего, Не может у вас Logstash, работающий как сервер конфигурацию свою найти. У вас они в /etc/logstash находятся?

orsa · April 11, 2019, 10:31am

да ,конфигурация там, точнее по стандартному pipeline
/etc/logstash/conf.d/

в файле прописан путь
administrator@dp-elk02:~$ sudo cat /etc/logstash/pipelines.yml

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

Игорь меня что смущает,
LS как сервис открывает входящие и исходящие соединения
то что прописано в секциях input и output
и дальше что то стопориться

Igor_Motov · April 11, 2019, 5:39pm

Да. Согласен. Это действительно странно.

А какая была версия и на что вы ее обновили?

Откуда вы получаете информацию? И если это beats то какая там версия?

orsa · April 14, 2019, 11:30pm

Добрый вечер Игорь,
про версионность уже восстановить сложно, ибо это вспомогательные логи и с ними работают нечасто, плюс чистятся индексы
высоко вероятно что сбой произошел в рамках обновления ветки 6.6
6.6.0 - > 6.6.1 -> 6.6.3

через LS идет syslog сетевого оборудования.
это либо настройки на оборудовании отправлять информацию на порт 2514, либо на локальный rsyslog (514), который в свою очередь пересылает на порт LS

все beat работают напрямую с ES и тут сбоев нет.

system · May 12, 2019, 11:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.