# Grokparsefailure output error
**URL:** https://discuss.elastic.co/t/grokparsefailure-output-error/209350
**Category:** Logstash
**Created:** [November 25, 2019, 5:09pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350 "2019-11-25T17:09:06Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Barushka](https://discuss.elastic.co/u/Barushka)
#### Post date: [November 25, 2019, 5:09pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/1 "2019-11-25T17:09:06Z")
My logstash shuts down because of this error:
[2019-11-25T10:53:06,400][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Something is wrong with your configuration.", :backtrace=\>["C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/config/mixin.rb:86:in `config_init'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/outputs/base.rb:60:in `initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:232:in `initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in `initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:239:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:137:in `buildOutput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/java\_pipeline.rb:23:in `initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]}
Here is my config file. What am i doing wrong?
input {
file {
path =\> "C:/ELK/WITSML/WITSML\_logs/rigcloud\_witsml/two/RigCLOUD\_WITSML\_Store\_ALL\_log\_two.log"
start\_position =\> "beginning"
}
}
filter {
if [message] =~ "Begin Request" {
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{NOTSPACE:method} - %{GREEDYDATA:witsml\_begin\_request}. URI: %{NOTSPACE:log\_message}, Referrer: , UserHostAddress: %{IP:user\_host\_ip}, UserHostName: %{IP:user\_host\_name}, TotalBytes: %{NUMBER:WITSML\_request\_bytes}"}
}
}
if [message] =~ "End Request" {
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{NOTSPACE:method} - %{GREEDYDATA:witsml\_end\_request}. URI: %{NOTSPACE:log\_message}, Referrer: , UserHostAddress: %{IP:user\_host\_ip}, UserHostName: %{IP:user\_host\_name}, TotalBytes: %{NUMBER:request\_bytes}, Elapsed Time: %{NUMBER:WITSML\_elapsed\_time}"}
}
}
if [message] =~ "apiURI" {
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{NOTSPACE:method} -%{GREEDYDATA:api\_request} %{NOTSPACE:log\_message}, Username: %{USERNAME}, StatusCode: %{WORD:status\_code}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{NOTSPACE:method} - Authentication Response for user: %{NOTSPACE:user\_name} %{GREEDYDATA:api\_request\_no\_elapsed\_time}: http:%{GREEDYDATA:api\_request\_details}"}
}
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{NOTSPACE:method} - Authorization Response for user: %{NOTSPACE:user\_name}. %{GREEDYDATA:api\_request} http:%{GREEDYDATA:api\_request\_details}, StatusCode: %{WORD:status\_code}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method} - %{NOTSPACE:api\_request\_no\_elapsed\_time}: %{GREEDYDATA:api\_query\_string}"}
}
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method} - %{NOTSPACE:api\_request}: %{GREEDYDATA:api\_query\_string}, StatusCode: %{WORD:status\_code}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method} - %{NOTSPACE:api\_request\_no\_elapsed\_time}: %{GREEDYDATA:api\_query\_string}, StatusCode: %{WORD:status\_code}"}
}
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method}. %{NOTSPACE:api\_request}: http:%{GREEDYDATA:api\_query\_string}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
}
}
output {
```
if "_grokparsefailure" in [tags] {
file {"path" => "C:/ELK/WITSML/WITSML_logs/rigcloud_witsml/two/api_grokparsefailure.txt"}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "witsml_api_twoo"
template => "C:/ELK/mytemplate.json"
}
}
```
}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 25, 2019, 6:29pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/2 "2019-11-25T18:29:18Z")
> [@Barushka](#):
>
> :exception=\>"LogStash::ConfigurationError", :message=\>"Something is wrong with your configuration."
There should be a more specific error message immediately preceding this one.
---
### Author:  [@Barushka](https://discuss.elastic.co/u/Barushka)
#### Post date: [November 26, 2019, 6:11pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/3 "2019-11-26T18:11:51Z")
there sure is. here is a full version of the error:
Sending Logstash logs to C:/ELK/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-11-25T10:52:57,081][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-11-25T10:52:57,114][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.1.1"}
[2019-11-25T10:53:06,383][ERROR][logstash.outputs.file] Unknown setting '"path"' for file
[2019-11-25T10:53:06,400][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Something is wrong with your configuration.", :backtrace=\>["C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/config/mixin.rb:86:in `config_init'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/outputs/base.rb:60:in `initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:232:in `initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in `initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:239:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:137:in `buildOutput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/java\_pipeline.rb:23:in `initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]}
[2019-11-25T10:53:06,881][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
[2019-11-25T10:53:11,586][INFO][logstash.runner] Logstash shut down.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 26, 2019, 6:36pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/4 "2019-11-26T18:36:20Z")
Remove the double quotes around path
```
file {"path" => "C:/ELK/..." }
```
should be
```
file { path => "C:/ELK/..." }
```
---
### Author:  [@Barushka](https://discuss.elastic.co/u/Barushka)
#### Post date: [December 6, 2019, 7:12pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/5 "2019-12-06T19:12:56Z")
I did and still nothing. Now it shows that logstash runs ok and elastic is fine but they do not talk to each other, thus I have no index formed in my kibana
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 6, 2019, 7:57pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/6 "2019-12-06T19:57:42Z")
If you enable log.level trace then filewatch should show you whether it is pushing event onto the pipeline. If it is then check what is in api\_grokparsefailure.txt.
---
### Author:  [@Barushka](https://discuss.elastic.co/u/Barushka)
#### Post date: [December 6, 2019, 8:12pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/7 "2019-12-06T20:12:36Z")
> [@Badger](#):
>
> log.level trace
here is about filewatch. looks fine I think , and nothing is in my grokfailure.txt file
[2019-12-06T13:32:37,887][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=\>"main"}
[2019-12-06T13:32:38,075][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2019-12-06T13:32:38,100][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}
[2019-12-06T13:32:39,540][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 6, 2019, 8:17pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/8 "2019-12-06T20:17:51Z")
Use log.level trace. filewatch will indicate what length it thinks the file is, how much of it it has read, when it flushes events to the pipeline etc. You should see a bunch of messages like [these](https://discuss.elastic.co/t/when-logstash-shutdown-renam-file-and-data-can-be-lost-or-duplicated/192669/3).
---
### Author:  [@Barushka](https://discuss.elastic.co/u/Barushka)
#### Post date: [December 18, 2019, 6:26pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/9 "2019-12-18T18:26:12Z")
Managed to fix it and now I am facing another issue. I am trying to catch the number of the events in one big event (linking the whole message by tracking ID). I am interested mainly in this info only. I used logstash conditionals to filter and output my data. Everything works ok except that my conditionals are executed partially. It catches the big event ( beginning and the end) but not the small events that happens in a big event. I checked all my groks and they all work in the debugger but for some reason logstash sends all my small events in groakparsefailure file. I wonder why is it? I assume it would compare my data, filter it and if any of the groks matches, give me the info. Am I missing something?
config file :
input {
file {
path =\> "C:/ELK/WITSML/WITSML\_logs/rigcloud\_witsml/three/RigCLOUD\_WITSML.log"
start\_position =\> "beginning"
}
}
filter {
if [indicator] == "Begin Request" {
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{DATA:method} - Begin Request. %{GREEDYDATA:indicator}, Referrer: , UserHostAddress: %{IP:user\_host\_ip}, UserHostName: %{IP:user\_host\_name}, TotalBytes: %{NUMBER:WITSML\_request\_bytes}"}
}
}
if [indicator] == "apiURI" {
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{DATA:method}: %{USERNAME:user\_name} %{GREEDYDATA:indicator}: %{GREEDYDATA:api\_request}, Header Values, token: %{DATA:token}, useridnbr: %{BASE10NUM}, livetoken: %{DATA:livetoken}, userrole: %{USERNAME:userrole}, userid: %{USERNAME:userid}"}
}
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{DATA:method}: %{USERNAME:user\_name}. %{NOTSPACE:indicator}: %{GREEDYDATA:api\_request}, StatusCode: %{WORD:status\_code}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method} - %{NOTSPACE:indicator}: %{GREEDYDATA:api\_request}, %{GREEDYDATA:api\_query\_srtring}"}
}
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method} - %{NOTSPACE:indicator}: %{GREEDYDATA:api\_request}, QueryString: %{GREEDYDATA:api\_query\_string}, StatusCode: %{WORD:status\_code}"}
}
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method}. %{NOTSPACE:indicator}: %{GREEDYDATA:api\_request}, QueryString: %{DATA:api\_query\_string}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{GREEDYDATA:method}. %{NOTSPACE:indicator}: %{GREEDYDATA:api\_request}, QueryString: %{DATA:api\_query\_string}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
grok {
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{NOTSPACE:method} -%{GREEDYDATA:indicator} %{NOTSPACE:api\_request}, Username: %{USERNAME:user\_name}, StatusCode: %{WORD:status\_code}, Elapsed Time: %{NUMBER:api\_elapsed\_time}"}
}
}
if [indicator] == "End Request" {
grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601} %{NOTSPACE:tracking\_ID} %{NOTSPACE:thread\_ID} %{WORD:log\_level} %{DATA:method} - End Request. %{GREEDYDATA:indicator}, Referrer: , UserHostAddress: %{IP:user\_host\_ip}, UserHostName: %{IP:user\_host\_name}, TotalBytes: %{NUMBER:request\_bytes}, Elapsed Time: %{NUMBER:WITSML\_elapsed\_time}"}
}
}
}
output {
```
if "_grokparsefailure" in [tags] {
file {
path => "C:/ELK/WITSML/WITSML_logs/rigcloud_witsml/three/api_grokparsefailure.txt"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "api_th"
template => "C:/ELK/mytemplate.json"
}
}
stdout
{
codec => rubydebug
}
```
}
data sample:
2019-10-22 18:42:26,850 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.AuthenticationModule.OnBeginRequest.0 - Begin Request. URI: /WitsmlStore.svc, Referrer: , UserHostAddress: 10.41.540.677, UserHostName: 10.41.540.677, TotalBytes: 1144
2019-10-22 18:42:26,858 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.AuthenticationModule.OnAuthenticateRequest.0 - Authorization Header: Basic Y2ggaXRoYW55YS52YW5ggbWE6Q2s4ODEgODg3OTY=
2019-10-22 18:42:26,861 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Authentication.GetCredentials.0 - Auth Header Value: Y2gggXRgYW55YS52YW5hbWE6Q2s4ODEgODc3OTY=
2019-10-22 18:42:26,862 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.Authentication.GetCredentials.0 - Auth Creds, username: bla.bla
2019-10-22 18:42:26,871 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Authentication.AuthenticateUsingAPI.0 - payload username: bla.bla
2019-10-22 18:42:31,004 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Security.Authentication.AuthenticateUsingAPI.0 - Response StatusCode: OK
2019-10-22 18:42:31,006 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Security.Authentication.AuthenticateUsingAPI.0 - Fetching headers from response
2019-10-22 18:42:31,022 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.Security.Authentication.AuthenticateUsingAPI.0 - Authentication Response for user: bla.bla. apiURI: [http://10.167.65.6:9096/api/v1/security/authenticate](http://10.167.65.6:9096/api/v1/security/authenticate), Header Values, token: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0=, useridbbr: 16789, livetoken: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0=, userrole: int\_user, userid: bla.bla
2019-10-22 18:42:31,023 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.Authentication.AuthenticateUsingAPI.0 - apiURI: [http://10.122.67.4:9077/api/v1/security/authenticate](http://10.122.67.4:9077/api/v1/security/authenticate), Username: bla.bla, StatusCode: OK, Elapsed Time: 4161 ms
2019-10-22 18:42:31,038 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Security.Authentication.SetPrincipal.0 - Creating claims and identity
2019-10-22 18:42:31,066 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Authentication.SetPrincipal.0 - Claims count: 6, Claims: ClaimTypes.NameIdentifier: bla.bla, ClaimTypes.Thumbprint: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0aO1V0noM5oShM=, ClaimTypes.UserData: 16069, ClaimTypes.Rsa: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0aO1V0noM5oShM=, ClaimTypes.Role: int\_user, ClaimTypes.Expiration: 6370736657384956067-7
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 15, 2020, 6:26pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350/10 "2020-01-15T18:26:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.