# Everyone faced this issue? **URL:** \ **Category:** Logstash\ **Created:** [October 29, 2019, 2:22am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576 "2019-10-29T02:22:52Z")\ **Posts on this page:** 16\ **Page:** 2
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 3:49am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/21 "2019-10-30T03:49:27Z")
> [@pbona](#): > > Add `path.settings /etc/logstash` to logstash.conf or logstash.yml ? t=That was to the command line to start which looks like you did. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:51am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/22 "2019-10-30T03:51:48Z")
By the output above, it still not working right ? ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:53am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/23 "2019-10-30T03:53:34Z")
``` [root@elastic logstash]# less /var/log/logstash/logstash-plain.log [2019-10-28T01:43:33,508][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"} [2019-10-28T01:43:33,549][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"} [2019-10-28T01:43:34,381][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.1"} [2019-10-28T01:43:34,423][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=>"ae98c94a-ed77-449e-9cec-66ba2e036929", :path=>"/var/lib/logstash/uuid"} [2019-10-28T01:43:35,932][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 1, column 7 (byte 7) after input ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]} [2019-10-28T01:43:36,493][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-10-28T01:43:41,249][INFO][logstash.runner] Logstash shut down. [2019-10-28T01:44:13,990][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.1"} [2019-10-28T01:44:15,636][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 1, column 7 (byte 7) after input ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]} ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:57am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/24 "2019-10-30T03:57:18Z")
This case, I have followed that documents. Elasticsearch and Kibana are done. excepted Logstash is not working. > **[How to setup TLS for Elasticsearch, Kibana, Logstash and Filebeat with...](https://www.elastic.co/blog/how-to-setup-tls-for-elasticsearch-kibana-logstash-filebeat-with-offline-install-in-linux)** > > How to transmit fully-encrypted log data from Filebeat to logstash to elasticsearch to kibana, along with offline install, in Linux OS. on Step 9. Enable TLS for Logstash on node1 ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 4:01am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/25 "2019-10-30T04:01:17Z")
> [@pbona](#): > > LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL '[https://172.16.5.199:9200/logstash](https://172.16.5.199:9200/logstash)' Nope you can not connect to the Elasticsearch response code 403. so now you can start to debug your actual issues. You can probably set back to `log.level: info` from here on out. Thats log file is from the previous runs ... with `systemctl`. That is still probably not right... did you re-install on top of an old install? I would debug all the configuration by running by command line then I would check the cat `/etc/systemd/system/logstash.service` file it should look something like the following my guess it is missing the `"--path.settings" "/etc/logstash"` ``` [Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 4:05am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/26 "2019-10-30T04:05:05Z")
Yep, I have re-install the Logstash. What happened with that scenario ? Could I remove and install logstash again? If that, give me the command line that can delete all the configuration files of logstash. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 4:09am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/27 "2019-10-30T04:09:31Z")
Yeah you probably need to get all cleaned up ... I am not an expert but use the package installer that you used do a complete uninstall (save your configs) then reinstall ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 4:11am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/28 "2019-10-30T04:11:22Z")
First time, I have ran `yum remove logstash` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 4:16am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/29 "2019-10-30T04:16:23Z")
> **[Configuring SSL, TLS, and HTTPS to secure Elasticsearch, Kibana, Beats, and...](https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash#step-2-6-4)** > > Feeling insecure about your Elastic Stack security? Run through these step-by-step instructions for setting up TLS encryption and https on Elasticsearch, Kibana, Logstash, and Beats to shore up your stack's defenses. Highly recommended for end-to-end... On step 5 (I followed) ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 5:56am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/30 "2019-10-30T05:56:46Z")
@stephenb Hi, I have reinstall the Logstash ``` [root@elastic logstash]# systemctl start logstash [root@elastic logstash]# systemctl status logstash ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-10-30 01:54:30 EDT; 16s ago Main PID: 23345 (java) CGroup: /system.slice/logstash.service └─23345 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX... Oct 30 01:54:30 elastic.nbc.org.kh.local systemd[1]: logstash.service holdoff time over, scheduling restart. Oct 30 01:54:30 elastic.nbc.org.kh.local systemd[1]: Stopped logstash. Oct 30 01:54:30 elastic.nbc.org.kh.local systemd[1]: Started logstash. Oct 30 01:54:30 elastic.nbc.org.kh.local logstash[23345]: OpenJDK 64-Bit Server VM warning: If the number...s=N Hint: Some lines were ellipsized, use -l to show in full. [root@elastic logstash]# less /var/log/logstash/logstash-plain.log [2019-10-30T01:50:33,908][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `
'"]} [2019-10-30T01:50:34,070][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2019-10-30T01:51:10,940][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `
'"]} [2019-10-30T01:51:10,961][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2019-10-30T01:51:44,279][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `
'"]} [2019-10-30T01:51:44,327][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2019-10-30T01:52:18,258][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in `block in validate_all'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/us/var/log/logstash/logstash-plain.log ``` Source: [https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash#step-2-6-4](https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash#step-2-6-4) ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 10:09am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/31 "2019-10-30T10:09:49Z")
@stephenb Could you help check the above logs ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 1:49pm UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/32 "2019-10-30T13:49:10Z")
HI @pbona I am a volunteer here, am happy we could get logstash running but I don't think I can help debug every line. It would seem it is time that you should start to be able to read the debug logs and then work on the fix. in this case the error is pretty clear. `An unexpected error occurred! {:error=>#,` Why that directory is not writeable I am unclear, I am not sure if this is a new system, how it was installed, why the directory permissions are wrong, but they appear to be wrong. In addition I am not clear that you are looking at the correct set of logs the The systemctl timestamp is `Wed 2019-10-30 01:54:30` and the logs you show from the logs in `/var/log/logstash/logstash-plain.log` timestamp is `[2019-10-30T01:50:33,908]` 3 minutes earlier so I am not even sure they are from the same run. My Suggestion is to get the ELK stack up and running with the all defaults first (unsecured) and _then_ try to secure it after you have it all running. Trying to do all the security with the initial setup when you are unfamiliar with the stack can make that process more difficult. Good Luck. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 31, 2019, 12:14am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/33 "2019-10-31T00:14:48Z")
I will try to fix it again... For the defaults (unsecured), did you the step by step document? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 31, 2019, 5:00am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/34 "2019-10-31T05:00:48Z")
Right on the documents home page [https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html](https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html) ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 1, 2019, 2:13am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/35 "2019-11-01T02:13:25Z")
> [@stephenb](#): > > An unexpected error occurred! {:error=\>#\, How to make its writable ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 29, 2019, 2:18am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/36 "2019-11-29T02:18:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/everyone-faced-this-issue/205576.md?page=1)