Доброго дня Коллеги,
на одном из обновлений "сломался" Logstash, данные которые шли через него перестали попадать в ES (прямая запись в ES работает прекрасно)
LS ставился из deb пакетов, он работает как сервис.
если его остановить и запустить из cli с тем же кофигом - данные идут и обрабатываются нормально.
как только LS стартует как служба - данных нет.
подозреваю что по путались связи между конфигурационными файлами, pipeline и тд.
Но пока картинка не сложилась.
куда стоит посмотреть?
в логи Logstash
Игорь спасибо за ответ, посмотрел
ничего криминального, стандартный старт системы.
LS начинает слушать порты прописанные в секции input, и открывает конекторы секции output.
Данные в сам ES не попадают.
[2019-04-10T15:52:44,338][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2019-04-10T15:53:02,928][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.dela
y"=>50}
[2019-04-10T15:53:04,079][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,437][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,525][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,530][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,570][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,621][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,638][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,646][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,646][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,663][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,679][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,694][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,707][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,708][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,728][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:04,739][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-10T15:53:04,751][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-10T15:53:04,762][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-10T15:53:04,762][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es
_version=>6}
[2019-04-10T15:53:04,769][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-10T15:53:06,206][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-04-10T15:53:06,310][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xe2922c5 run>"}
[2019-04-10T15:53:06,441][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2515"}
[2019-04-10T15:53:06,651][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-04-10T15:53:06,744][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2515", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-04-10T15:53:06,858][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-04-10T15:53:08,254][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
может вопрос прав, ибо из cli он запускался с sudo
sudo /usr/share/logstash/bin/logstash -f test.in-out.conf
А вы уверены, что это не логи, которые остались после последнего запуска через командную строку?
Ну это легко проверить. Посмотрите на файлы и директории по списку из предыдущего ответа. Кто ими владеет и какие на них стоят права.
лог рестарта LS как сервиса по команде
sudo systemctl restart logstash.service && sudo tail -f /var/log/logstash/logstash-plain.log
[WARN ][logstash.runner ] SIGTERM received. Shutting down.
[WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>32, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.8-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>29, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
[ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xe2922c5 run>"}
[INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.1"}
[INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x141c02ea run>"}
[INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2515"}
[INFO ][org.logstash.beats.Server] Starting server on port: 5044
[INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2515", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
теперь лог старта LS из cli
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-04-10 22:22:36.339 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-04-10 22:22:36.352 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.7.1"}
[INFO ] 2019-04-10 22:22:49.062 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-04-10 22:22:49.914 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-04-10 22:22:50.196 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-04-10 22:22:50.556 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-04-10 22:22:50.695 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-04-10 22:22:50.697 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2019-04-10 22:22:50.702 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-04-10 22:22:50.706 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-04-10 22:22:50.714 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-04-10 22:22:50.714 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-04-10 22:22:50.717 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2019-04-10 22:22:51.154 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2019-04-10 22:22:51.186 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7f40ce3e run>"}
[INFO ] 2019-04-10 22:22:51.426 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2019-04-10 22:22:51.446 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:2514"}
[INFO ] 2019-04-10 22:22:51.464 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-04-10 22:22:51.625 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:2514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[INFO ] 2019-04-10 22:22:52.089 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
права сейчас проверю отдельно
Скорее всего, Не может у вас Logstash, работающий как сервер конфигурацию свою найти. У вас они в /etc/logstash
находятся?
да ,конфигурация там, точнее по стандартному pipeline
/etc/logstash/conf.d/
в файле прописан путь
administrator@dp-elk02:~$ sudo cat /etc/logstash/pipelines.yml
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
Игорь меня что смущает,
LS как сервис открывает входящие и исходящие соединения
то что прописано в секциях input и output
и дальше что то стопориться
Да. Согласен. Это действительно странно.
А какая была версия и на что вы ее обновили?
Откуда вы получаете информацию? И если это beats то какая там версия?
Добрый вечер Игорь,
про версионность уже восстановить сложно, ибо это вспомогательные логи и с ними работают нечасто, плюс чистятся индексы
высоко вероятно что сбой произошел в рамках обновления ветки 6.6
6.6.0 - > 6.6.1 -> 6.6.3
через LS идет syslog сетевого оборудования.
это либо настройки на оборудовании отправлять информацию на порт 2514, либо на локальный rsyslog (514), который в свою очередь пересылает на порт LS
все beat работают напрямую с ES и тут сбоев нет.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.