[logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError",

[Hanneu]

Hello experts,
Getting below error on logstash (LS) logs for all of the nodes.

[2020-05-21T12:08:42,231][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 4016, column 2 (byte 159543) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:42:in block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:92:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:92:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:317:in block in converge_state'"]}

Elasticsearch version: 6.5.4
Logstash version: 6.5.4
Kibana verison: 6.5.4

LS service has been up and running on all LS nodes. Attached is the logstash.yml, input.conf, output.conf and ruby filter files. Any help is greatly appreciated.

Can't attach the files.
input.conf looks like below:-
input {
beats {
port => "5044"
client_inactivity_timeout => "600"
}
}

Output.conf

[ptamba]

@Hanneu

Expected one of #, input, filter, output at line 4016, column 2 (byte 159543) after ", :backtrace=>

one of your config files is having syntax issues. looks like you put a whole bunch of .conf files in a same directory making logstash sees the error at line 4016

[Hanneu]

Hello Ptamba,
Yes indeed there are lot of .conf files under /etc/logstash/conf.d. Could you please recommend IF these .conf files need to be moved to a different directory?
[root@polelk04 logstash]# cd /etc/logstash/conf.d/
[root@polelk04 conf.d]# ls -ltr
total 212
-rw-r--r--. 1 root root 2574 Jun 22 2018 eccgw_filter.conf
-rw-r--r--. 1 root root 368 Jun 22 2018 accesslog.conf
-rw-r--r--. 1 root root 594 Jun 22 2018 tpcdb_error.conf
-rw-r--r--. 1 root root 4667 Oct 5 2018 epe_tpc_filter.conf
-rw-r--r--. 1 root root 11524 Oct 8 2018 filter_OCC_MSG.conf
-rw-r--r--. 1 root root 12510 Oct 15 2018 filter_OCC_MSG_ts.conf
-rw-r--r--. 1 root root 11772 Oct 15 2018 filter_OCC_DM_ts.conf
-rw-r--r--. 1 root root 5162 Oct 24 2018 filter_CCN.conf
-rw-r--r--. 1 root root 1766 Oct 24 2018 filter_SDP.conf
-rw-r--r--. 1 root root 7058 Oct 24 2018 filter_CCN_ts.conf
-rw-r--r--. 1 root root 15088 Oct 24 2018 filter_SDP_ts.conf
-rw-r--r--. 1 root root 20713 Oct 24 2018 filter_AIR_ts.conf
drwxr-xr-x. 3 root root 4096 Nov 27 2018 bkup
-rw-r--r--. 1 root root 49992 Nov 27 2018 filter_AIR.conf
-rw-r--r--. 1 root root 2119 Dec 17 2018 filter_cc.conf
drwxr-xr-x. 2 root root 100 Jul 25 2019 certs
-rw-r--r--. 1 root root 9156 Sep 12 2019 filter_OCC_dm.conf
-rw-r--r--. 1 root root 4869 May 21 11:11 output.conf
-rw-r--r--. 1 root root 498 May 21 14:52 output_cc.conf
-rw-r--r--. 1 root root 471 May 21 14:58 output_elk.conf
-rw-r--r--. 1 root root 95 May 22 08:49 input.conf

Any help is greatly appreciated.

[Hanneu]

Hello experts,
Any inputs or suggested workaround on this issue?

[ptamba]

you could try running logstash from CLI with config test and increase the log level to debug or trace

logstash -f <path_to_config_di> --config.test_and_exit --log.level trace