He given configuration is invalid. Reason: Expected one of #, input, filter, output

[kenorb]

I've got the following syntax error when running Logstash:

$ sudo -u logstash /usr/share/logstash/bin/logstash -V
logstash 7.1.0

$ sudo -u logstash /usr/share/logstash/bin/logstash -t -f /etc/logstash/logstash.yml
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-10-04 13:51:54.891 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2019-10-04 13:51:56.049 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 214, column 1 (byte 6380) after # Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
#   pipeline:
#     batch:
#       size: 125
#       delay: 5
#
# Or as flat keys:
#
#   pipeline.batch.size: 125
#   pipeline.batch.delay: 5
...
[ERROR] 2019-10-04 13:51:56.093 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit


$ tail /etc/logstash/logstash.yml
# * debug
# * trace
#
# log.level: info
#path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []

Basically I've commented out all the lines and Logstash still complaining about the syntax. Any ideas why Logstash errors?

Also I can see the following error in the logs:

[2019-10-04T13:41:57,572][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 214, column 1 (byte 6380) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}

[ben.west]

It looks like you are passing the logstash configuration file as pipeline configuration.
The -f option should point towards your pipeline config (a file which contains input {} filter {} and output {} blocks).

The logstash config file is specified via commandline using the option --path.settings and should be the directory in which your logstash.yml is.

[kenorb]

Thank you, I've specified --path.settings, but it still complains about the syntax, but this time doesn't tell which file it is.

# sudo -u logstash /usr/share/logstash/bin/logstash -t --path.settings /etc/logstash
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-10-04T14:19:57,172][FATAL][logstash.runner          ] The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 46, column 1 (byte 1000) after 
[2019-10-04T14:19:57,211][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Logs (logstash-plain.log):

[2019-10-04T14:21:11,555][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-10-04T14:21:13,893][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 46, column 1 (byte 1000) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-10-04T14:21:14,033][INFO ][logstash.runner          ] Logstash shut down.

Ok, I've found which file it is (/etc/logstash/conf.d/foo.conf), based on the line.

After the config fixes, the following command works correctly:

sudo -u logstash /usr/share/logstash/bin/logstash -t --path.settings /etc/logstash

[2019-10-04T14:24:22,219][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Thanks.