Logstash node not showing up in Kibana monitoring (Logstash constantly restarting)

[vdere (Viraj)]

One of our logstash node is not showing in the monitoring tab. So how can I log into specific logstash nodes to verify if x-pack is running? Assuming that is what is causing this problem. Or there might be any other issue with this?

Edit: I just checked and can see that logstash is constantly restarting. What could be the issue here?

[st3inbeiss (Pius Dittli)]

Could you post an excerpt of the log or a relevant error message?

It's kinda hard to tell what's wrong without any information.

[vdere (Viraj)]

Go those logs using systemctl status logstash -l

Oct 30 10:07:20 delk-stg1-log03 systemd[1]: Started logstash.
Oct 30 10:07:30 delk-stg1-log03 logstash[24955]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Oct 30 10:07:30 delk-stg1-log03 logstash[24955]: [2019-10-30T10:07:30,734][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Oct 30 10:07:30 delk-stg1-log03 logstash[24955]: [2019-10-30T10:07:30,742][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.2"}
Oct 30 10:07:31 delk-stg1-log03 logstash[24955]: [2019-10-30T10:07:31,360][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
Oct 30 10:07:31 delk-stg1-log03 logstash[24955]: [2019-10-30T10:07:31,360][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
Oct 30 10:07:32 delk-stg1-log03 logstash[24955]: [2019-10-30T10:07:32,527][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:r6_stgl3, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1870, column 1 (byte 73155) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]}

[st3inbeiss (Pius Dittli)]

There seems to be an issue with the config of the pipeline r6_stgl3

Is this a pipeline you manually created or modified?

My guess would be that there is an incorrect block of {} brackets somewhere in your config. The offending block ends/begins in line 1870 of your config.

[vdere (Viraj)]

There was a bad config file added during the last restart of logstash which resulted in this issue.
Now it is resolved. Moved the bad config to disabled folder and restarted logstash.