t=That was to the command line to start which looks like you did.
By the output above, it still not working right ?
[root@elastic logstash]# less /var/log/logstash/logstash-plain.log
[2019-10-28T01:43:33,508][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2019-10-28T01:43:33,549][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2019-10-28T01:43:34,381][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.1"}
[2019-10-28T01:43:34,423][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"ae98c94a-ed77-449e-9cec-66ba2e036929", :path=>"/var/lib/logstash/uuid"}
[2019-10-28T01:43:35,932][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 1, column 7 (byte 7) after input ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[2019-10-28T01:43:36,493][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-10-28T01:43:41,249][INFO ][logstash.runner ] Logstash shut down.
[2019-10-28T01:44:13,990][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.1"}
[2019-10-28T01:44:15,636][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 1, column 7 (byte 7) after input ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}This case, I have followed that documents. Elasticsearch and Kibana are done. excepted Logstash is not working.
on Step 9. Enable TLS for Logstash on node1
Nope you can not connect to the Elasticsearch response code 403. so now you can start to debug your actual issues. You can probably set back to log.level: info from here on out.
Thats log file is from the previous runs ... with systemctl. That is still probably not right... did you re-install on top of an old install?
I would debug all the configuration by running by command line then I would check the cat /etc/systemd/system/logstash.service file it should look something like the following my guess it is missing the "--path.settings" "/etc/logstash"
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
Yep, I have re-install the Logstash. What happened with that scenario ? Could I remove and install logstash again? If that, give me the command line that can delete all the configuration files of logstash.
Yeah you probably need to get all cleaned up ... I am not an expert but use the package installer that you used do a complete uninstall (save your configs) then reinstall
First time, I have ran yum remove logstash
On step 5 (I followed)
@stephenb Hi, I have reinstall the Logstash
[root@elastic logstash]# systemctl start logstash
[root@elastic logstash]# systemctl status logstash
â— logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-10-30 01:54:30 EDT; 16s ago
Main PID: 23345 (java)
CGroup: /system.slice/logstash.service
└─23345 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX...
Oct 30 01:54:30 elastic.nbc.org.kh.local systemd[1]: logstash.service holdoff time over, scheduling restart.
Oct 30 01:54:30 elastic.nbc.org.kh.local systemd[1]: Stopped logstash.
Oct 30 01:54:30 elastic.nbc.org.kh.local systemd[1]: Started logstash.
Oct 30 01:54:30 elastic.nbc.org.kh.local logstash[23345]: OpenJDK 64-Bit Server VM warning: If the number...s=N
Hint: Some lines were ellipsized, use -l to show in full.
[root@elastic logstash]# less /var/log/logstash/logstash-plain.log
[2019-10-30T01:50:33,908][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2019-10-30T01:50:34,070][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-10-30T01:51:10,940][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2019-10-30T01:51:10,961][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-10-30T01:51:44,279][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2019-10-30T01:51:44,327][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-10-30T01:52:18,258][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/us/var/log/logstash/logstash-plain.log
@stephenb Could you help check the above logs ?
HI @pbona
I am a volunteer here, am happy we could get logstash running but I don't think I can help debug every line. It would seem it is time that you should start to be able to read the debug logs and then work on the fix.
in this case the error is pretty clear.
An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>,
Why that directory is not writeable I am unclear, I am not sure if this is a new system, how it was installed, why the directory permissions are wrong, but they appear to be wrong.
In addition I am not clear that you are looking at the correct set of logs the
The systemctl timestamp is Wed 2019-10-30 01:54:30 and the logs you show from the logs in /var/log/logstash/logstash-plain.log timestamp is
[2019-10-30T01:50:33,908] 3 minutes earlier so I am not even sure they are from the same run.
My Suggestion is to get the ELK stack up and running with the all defaults first (unsecured) and then try to secure it after you have it all running. Trying to do all the security with the initial setup when you are unfamiliar with the stack can make that process more difficult.
Good Luck.
1 Like
I will try to fix it again...
For the defaults (unsecured), did you the step by step document?
Right on the documents home page
https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html
How to make its writable ?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.