[2020-06-18T10:31:56,332][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
[2020-06-18T10:31:56,424][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2020-06-18T10:31:56,484][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
If I set path.config: "/etc/logstash/conf.d/01-wazuh.conf" in logstash.yml this is output:
[2020-06-18T10:36:35,235][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-06-18T10:36:35,246][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.7.1"}
[2020-06-18T10:36:35,338][DEBUG][logstash.agent ] Setting up metric collection
[2020-06-18T10:36:35,475][DEBUG][logstash.instrument.periodicpoller.os] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-06-18T10:36:35,958][DEBUG][logstash.instrument.periodicpoller.jvm] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-06-18T10:36:36,210][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2020-06-18T10:36:36,220][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
This is the expected behavior, it means no error and your logstash work as expected,
I saw that you run with this command
/usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash --config.debug
It should be like the last command:
/usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash/conf.d/ --config.debug
After all set, please remove the debug option. Use systemd start/stop of logstash, then check use systemctl status logstash.
Regards,
Fadjar Tandabawana
/usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash --config.debug--->
Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
/usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash/conf.d/ --config.debug---->
ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/conf.d/pipelines.yml
usage
Sorry, my mistake
You right.. the first to check the config.
So, what is the problem for your config?
The result of the log of the logstash is the expected behavior until there are explicit ERROR message occurred.
Regards,
Fadjar Tandabawana
@fadjar340 I'm bercome crazy!!!!
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/01-wazuh.conf works fine!!!!
/usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml--->
[ERROR] 2020-06-18 10:56:49.875 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 19, column 1 (byte 320) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in
compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:inblock in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:inblock in converge_state'"]}
[INFO ] 2020-06-18 10:56:50.552 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
This is definitely wrong and you got the ERROR message.
Also
/usr/share/logstash/bin/logstash -f /etc/logstash/pipeline.yml <=wrong
The logstash.yml and pipeline.yml will read when the --path.settings .
If you want to config the input, filter and output just use -f /etc/logstash/conf.d/
So, please don't use pipeline.yml as process configuration.
Better, if you want to use default pipeline.id: main, just put into logstash.yml and put the process configuration into /etc/logstash/conf.d/ accordingly.
Regards,
Fadjar Tandabawana
This is my actual logstash.yml:
pipeline.id: main
path.config: "/etc/logstash/conf.d/"
If you put the process config in the default folder /etc/logstash/conf.d/01-wazuh.conf ,
just disable the path.config: "/etc/logstash/conf.d/", because the default logstash config for processing in that folder, instead you put into another folder, then you need to use pipeline.yml.
I wish your logstash works as expected...
Regards,
Fadjar Tandabawana
I've removed path.config than /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml--->
[WARN ] 2020-06-18 11:13:39.323 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-18 11:13:39.340 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.1"}
[ERROR] 2020-06-18 11:13:41.110 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 19, column 1 (byte 320) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:inblock in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:inblock in converge_state'"]}
[INFO ] 2020-06-18 11:13:41.627 [Api Webserver] agent - Successfully started
Try just run the logstash from systemctl..
@fadjar340 This is complete /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml output:
root@elk1:~# /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-18 11:13:39.323 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-18 11:13:39.340 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.1"}
[ERROR] 2020-06-18 11:13:41.110 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 19, column 1 (byte 320) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:inblock in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:inblock in converge_state'"]}
[INFO ] 2020-06-18 11:13:41.627 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-06-18 11:13:46.433 [LogStash::Runner] runner - Logstash shut down.
Try this in the logstash.yml
pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
Same bad output. 
I'm using logstash.version"=>"7.7.1 on OpenJDK Runtime Environment (build 11.0.7+10-post-Debian-3deb10u1)
Did you following this:
https://documentation.wazuh.com/3.9/installation-guide/installing-elastic-stack/transform_logstash.html
I think you need to follow exact procedures....
Yes I've followed https://documentation.wazuh.com/3.12/installation-guide/installing-elastic-stack/transform_logstash.html
from your logstash.yml
# The ID of the pipeline.
#
pipeline.id: main
your logstash.yml has pipeline definition, with the id "main", and your pipelines.yml also has pipeline definition with id "main". I suggest just choose one of two.
there should be a default logstash.yml in /etc/logstash. you can backup your existing logstash.yml and restore the default logstash.yml in place.
never use logstash -f with logstash.yml. that file is logstash settings file, not logstash pipeline configuration file. read here for the difference , as well as information for pipelines.yml
you only pass -f to pipeline configuration files (either files or directory). read here for various options in running Logstash from CLI.
Assuming you're not making any changes to systemd, and you're using deb / rpm system : when you run logstash (without any parameters) from CLI, or using systemctl, logstash will read settings. from /etc/logstash. there's no need to pass --path.settings options
I've replaced logstash.yml and pipelines.yml with defaults
https://paste.debian.net/1152655/
https://paste.debian.net/1152657/
than systemctl start logstash ---> same error
[2020-06-18T13:03:22,291][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2020-06-18T13:03:22,344][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.7.1"}
[2020-06-18T13:03:22,479][DEBUG][logstash.agent ] Setting up metric collection
[2020-06-18T13:03:22,584][DEBUG][logstash.instrument.periodicpoller.os] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-06-18T13:03:23,058][DEBUG][logstash.instrument.periodicpoller.jvm] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-06-18T13:03:23,293][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2020-06-18T13:03:23,303][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
what does 01-wazuh.conf looks like ?
curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/v3.9.5/extensions/logstash/7.x/01-wazuh-remote.conf