# Log stash error **URL:** \ **Category:** Logstash\ **Created:** [November 1, 2019, 10:09am UTC](https://discuss.elastic.co/t/log-stash-error/206141 "2019-11-01T10:09:51Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 1, 2019, 10:09am UTC](https://discuss.elastic.co/t/log-stash-error/206141/1 "2019-11-01T10:09:51Z")
below is the error i get while running logstash > Sending Logstash logs to C:/busapps/rrsb/gbl1/logstash/7.0.0/logs which is now configured via log4j2.properties > [2019-11-01T10:04:00,538][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified > [2019-11-01T10:04:00,703][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.0.0"} > [2019-11-01T10:04:13,129][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 69, column 5 (byte 1460) after output {\r\n\tstdout {}\r\n \tif ("total" in [tags]) {\r\n \t\telasticsearch {\r\n \t\t\thosts =\> ["localhost:9200"]\r\n \t\t\tindex =\> "totalexecution-%{+YYYY}"\r\n\t\t\t\tuser =\> elastic\r\n\t\t\t\tpassword =\> 3wUwULD3QJaKke\r\n\t\t\t\r\n \t\t", :backtrace=\>["C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "C:/busapps/rrsb/gbl1/logstash/7.0.0/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]} > [2019-11-01T10:04:15,907][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} > [2019-11-01T10:04:19,360][INFO][logstash.runner] Logstash shut down. [config file](https://notepad.pw/j2v8702) config file has 68 lines , but log stash is showing error in line number 69 . am not sure whether log stash is picking the config file. log stash version used is 7.0 . please help. P.S my previous config file was bigger , it had a else section in the out put. i removed just to identify whether there is really problem with line number 69 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 1, 2019, 2:11pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/2 "2019-11-01T14:11:42Z")
I would put double quotes around the username and password. What comes immediately after that? ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 4, 2019, 11:35am UTC](https://discuss.elastic.co/t/log-stash-error/206141/3 "2019-11-04T11:35:34Z")
are u asking the pipline folder configration which logstash uses to apply filters etc., ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2019, 12:36pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/4 "2019-11-04T12:36:16Z")
No, I am asking what comes immediately after the password in your logstash configuration. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 4, 2019, 12:46pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/5 "2019-11-04T12:46:40Z")
let me give some more info . below is the error we are getitng while "file beats" sends te log to the ELK server `2019-11-04T12:43:06.072Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://win000587.aze.michelin.com:5044)): dial tcp 10.221.100.180:5044: connectex: No connection could be made because the target machine actively refused it.` and to your question what comes after the password section ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/93313f04430ce9c2a96882e32a88269061e99a86.png) i even tried to copy and paste the in some working configration > "password =\>" why beacuse , some times the =\> this one gives problem . i readin some threads ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2019, 1:20pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/6 "2019-11-04T13:20:37Z")
I do not understand how that configuration could produce that error. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 4, 2019, 1:47pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/7 "2019-11-04T13:47:35Z")
you mean to say that everything is fine ? are you are doubtful about something ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2019, 3:09pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/8 "2019-11-04T15:09:17Z")
You initial post included a configuration and an error message. I am saying I cannot understand how that configuration could result in the error message that your post included. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 4, 2019, 3:30pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/9 "2019-11-04T15:30:24Z")
there are two error message i posted . one i obtained in form the log stash log file . that is in the post itself. the error message in the conversation thread is one more evidence i got from file beats . ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 4, 2019, 3:31pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/10 "2019-11-04T15:31:37Z")
am not saying that config file is one issue , am just posting what are all the facts i have . not sure still this error is because of config file or some thing else is the reason ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2019, 3:37pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/11 "2019-11-04T15:37:49Z")
> [@Seetharaman\_K](#): > > :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 69, column 5 (byte 1460) after output {\r\n\tstdout {}\r\n \tif ("total" in [tags]) {\r\n \t\telasticsearch {\r\n \t\t\thosts =\> ["localhost:9200"]\r\n \t\t\tindex =\> "totalexecution-%{+YYYY}"\r\n\t\t\t\tuser =\> elastic\r\n\t\t\t\tpassword =\> 3wUwULD3QJaKke\r\n\t\t\t\r\n \t\t" That exception is unquestionably because of the config file. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 4, 2019, 3:56pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/12 "2019-11-04T15:56:57Z")
ok here are some more observations , - even if i change the cofigration (like removing stdout , removing one more output section still the number of lines in the error was the same. ideally logstash service should pick the new file in $logshtash/bin/pipelines \*telnet to 5044 from beats server to logstash server is not working ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2019, 4:40pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/13 "2019-11-04T16:40:59Z")
If you get exception=\>"LogStash::ConfigurationError" then the pipeline is not running, so I would not expect it to be listening on port 5044. Try running with --config.debug --log.level debug --config.test\_and\_exit on the command line. The configuration will get printed out after a message that says [DEBUG][logstash.config.pipelineconfig] Merged config. Please post the configuration that gets printed. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 5, 2019, 6:36am UTC](https://discuss.elastic.co/t/log-stash-error/206141/14 "2019-11-05T06:36:53Z")
as you asked : below is the final part i posted this as image . otherwise the entire debug result is in the below link [Logstash output in command line - DebugMode](https://notepad.pw/9ecj0n2q) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/4813e3d40ff15df612b69eba4ac1c8d479dde7cc.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 5, 2019, 3:31pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/15 "2019-11-05T15:31:55Z")
```auto [DEBUG][logstash.runner] *path.config: "C:\\busapps\\rrsb\\gbl1\\logstash\\7.0.0\\bin\\pipelines" ``` That is a directory. logstash will concatenate all of the files in that directory to form the configuration. Every file. No exceptions. If there is a java heap dump in the directory then logstash will try to parse it as a configuration file. Other messages then logged are ```auto Config string {:protocol=>"file", :id=>"C:/busapps/rrsb/gbl1/logstash/7.0.0/bin/pipelines/logstash - Copy.conf"} Config string {:protocol=>"file", :id=>"C:/busapps/rrsb/gbl1/logstash/7.0.0/bin/pipelines/logstash.conf"} Config string {:protocol=>"file", :id=>"C:/busapps/rrsb/gbl1/logstash/7.0.0/bin/pipelines/logstash_bkp.conf"} ``` So it merges those three files to form the configuration. The error is at line 68, which is in the first file. Adding quotes around the passwords in logstash - Copy.conf fixes the errors, but you probably want to move the backup files to a different directory. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 5, 2019, 4:52pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/16 "2019-11-05T16:52:16Z")
trying to keep only one config file and trying to rerun . will post the results in few minutes. thanks . ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 5, 2019, 5:43pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/17 "2019-11-05T17:43:45Z")
seems i have corrected the configuration as you suggested . why i am sure , because got the below screen after i tested the config ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/4/a4c0ee092595efdc5b7dcf70f99e9afeb4944e55.png) , but when logstash service runs i again got following error > [2019-11-05T17:57:17,176][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) > [2019-11-05T17:57:17,176][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>1} > [2019-11-05T17:57:17,176][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) > [2019-11-05T17:57:17,176][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>1} > [2019-11-05T17:57:17,184][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) > [2019-11-05T17:57:17,188][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>1} > [2019-11-05T17:57:17,188][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) > [2019-11-05T17:57:17,188][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) > [2019-11-05T17:57:17,188][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) > [2019-11-05T17:57:17,188][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>3} two kind of post discuss about this. 1.disk space issue (which is not in my case ) 2.[this post talks about locked indices](https://discuss.elastic.co/t/logstash-error-retrying-failed-action-with-response-code-403/174864/2) not sure how to identify the actual problem ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 5, 2019, 6:13pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/18 "2019-11-05T18:13:56Z")
> [@Seetharaman\_K](#): > > [2019-11-05T17:57:17,176][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"index [totalexecution-2019] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) If elasticsearch has set the index to be read-only by far the most common reason is that disk utilization reached 95%. Even if utilization comes back down the index will remain read-only. When elasticsearch set the index to be read-only it will have logged the reason why. Check your elasticsearch logs. ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 5, 2019, 6:19pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/19 "2019-11-05T18:19:03Z")
checking will get back in few mins thanks for your observation and guidance so far 🙂 ---
**Author:** ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Post date:** [November 5, 2019, 6:33pm UTC](https://discuss.elastic.co/t/log-stash-error/206141/20 "2019-11-05T18:33:54Z")
Below is the elastic log . i did have a look at that . could not locate exactly where elastic is setting index to be read only [elastic log](https://notepad.pw/39f0gf2u) how ever some words here and there i can find . like , > Desired survivor size 17432576 bytes, new threshold 1 (max 6) > > - age 1: 21928840 bytes, 21928840 total in some lines above mentioned threshold is going up "6" . not sure whether that is the place where index is becoming read only. also one observation , as the above logstash problem existed for more than a month , lots of logs have heaped under this index"totalexecution-2019" (which is the indexwe saw in the error). so we should take that also in account while zeroing the problem. [Next page](https://discuss.elastic.co/t/log-stash-error/206141.md?page=2)