Grokparsefailure output error

Elastic Stack Logstash
1

My logstash shuts down because of this error:
[2019-11-25T10:53:06,400][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/config/mixin.rb:86:in config_init'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/outputs/base.rb:60:in initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:232:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:in initialize'", "org/logstash/plugins/PluginFactoryExt.java:239:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:137:in buildOutput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Here is my config file. What am i doing wrong?

input {
file {
path => "C:/ELK/WITSML/WITSML_logs/rigcloud_witsml/two/RigCLOUD_WITSML_Store_ALL_log_two.log"
start_position => "beginning"
}
}

filter {
if [message] =~ "Begin Request" {
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - %{GREEDYDATA:witsml_begin_request}. URI: %{NOTSPACE:log_message}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:WITSML_request_bytes}"}
}
}

if [message] =~ "End Request" {
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - %{GREEDYDATA:witsml_end_request}. URI: %{NOTSPACE:log_message}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:request_bytes}, Elapsed Time: %{NUMBER:WITSML_elapsed_time}"}
}
}

if [message] =~ "apiURI" {
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} -%{GREEDYDATA:api_request} %{NOTSPACE:log_message}, Username: %{USERNAME}, StatusCode: %{WORD:status_code}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - Authentication Response for user: %{NOTSPACE:user_name} %{GREEDYDATA:api_request_no_elapsed_time}: http:%{GREEDYDATA:api_request_details}"}
}
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - Authorization Response for user: %{NOTSPACE:user_name}. %{GREEDYDATA:api_request} http:%{GREEDYDATA:api_request_details}, StatusCode: %{WORD:status_code}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method} - %{NOTSPACE:api_request_no_elapsed_time}: %{GREEDYDATA:api_query_string}"}
}
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method} - %{NOTSPACE:api_request}: %{GREEDYDATA:api_query_string}, StatusCode: %{WORD:status_code}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method} - %{NOTSPACE:api_request_no_elapsed_time}: %{GREEDYDATA:api_query_string}, StatusCode: %{WORD:status_code}"}
}
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method}. %{NOTSPACE:api_request}: http:%{GREEDYDATA:api_query_string}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
}
}

output {

if "_grokparsefailure" in [tags] {
    file {"path" => "C:/ELK/WITSML/WITSML_logs/rigcloud_witsml/two/api_grokparsefailure.txt"}
}
else {
    elasticsearch {
    hosts => ["localhost:9200"]
index => "witsml_api_twoo"
template => "C:/ELK/mytemplate.json"
 }
}

}

2

There should be a more specific error message immediately preceding this one.

3

there sure is. here is a full version of the error:

Sending Logstash logs to C:/ELK/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-11-25T10:52:57,081][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-11-25T10:52:57,114][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-11-25T10:53:06,383][ERROR][logstash.outputs.file ] Unknown setting '"path"' for file
[2019-11-25T10:53:06,400][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/config/mixin.rb:86:in config_init'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/outputs/base.rb:60:in initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:232:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:in initialize'", "org/logstash/plugins/PluginFactoryExt.java:239:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:137:in buildOutput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/ELK/logstash-7.1.1/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-11-25T10:53:06,881][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-11-25T10:53:11,586][INFO ][logstash.runner ] Logstash shut down.

4

Remove the double quotes around path

file {"path" => "C:/ELK/..." }

should be

file { path  => "C:/ELK/..." }
5

I did and still nothing. Now it shows that logstash runs ok and elastic is fine but they do not talk to each other, thus I have no index formed in my kibana

6

If you enable log.level trace then filewatch should show you whether it is pushing event onto the pipeline. If it is then check what is in api_grokparsefailure.txt.

7

here is about filewatch. looks fine I think , and nothing is in my grokfailure.txt file
[2019-12-06T13:32:37,887][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-12-06T13:32:38,075][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-12-06T13:32:38,100][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-12-06T13:32:39,540][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

8

Use log.level trace. filewatch will indicate what length it thinks the file is, how much of it it has read, when it flushes events to the pipeline etc. You should see a bunch of messages like these.

9

Managed to fix it and now I am facing another issue. I am trying to catch the number of the events in one big event (linking the whole message by tracking ID). I am interested mainly in this info only. I used logstash conditionals to filter and output my data. Everything works ok except that my conditionals are executed partially. It catches the big event ( beginning and the end) but not the small events that happens in a big event. I checked all my groks and they all work in the debugger but for some reason logstash sends all my small events in groakparsefailure file. I wonder why is it? I assume it would compare my data, filter it and if any of the groks matches, give me the info. Am I missing something?

config file :
input {
file {
path => "C:/ELK/WITSML/WITSML_logs/rigcloud_witsml/three/RigCLOUD_WITSML.log"
start_position => "beginning"
}
}

filter {
if [indicator] == "Begin Request" {
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{DATA:method} - Begin Request. %{GREEDYDATA:indicator}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:WITSML_request_bytes}"}
}
}

if [indicator] == "apiURI" {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{DATA:method}: %{USERNAME:user_name} %{GREEDYDATA:indicator}: %{GREEDYDATA:api_request}, Header Values, token: %{DATA:token}, useridnbr: %{BASE10NUM}, livetoken: %{DATA:livetoken}, userrole: %{USERNAME:userrole}, userid: %{USERNAME:userid}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{DATA:method}: %{USERNAME:user_name}. %{NOTSPACE:indicator}: %{GREEDYDATA:api_request}, StatusCode: %{WORD:status_code}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method} - %{NOTSPACE:indicator}: %{GREEDYDATA:api_request}, %{GREEDYDATA:api_query_srtring}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method} - %{NOTSPACE:indicator}: %{GREEDYDATA:api_request}, QueryString: %{GREEDYDATA:api_query_string}, StatusCode: %{WORD:status_code}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method}. %{NOTSPACE:indicator}: %{GREEDYDATA:api_request}, QueryString: %{DATA:api_query_string}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{GREEDYDATA:method}. %{NOTSPACE:indicator}: %{GREEDYDATA:api_request}, QueryString: %{DATA:api_query_string}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} -%{GREEDYDATA:indicator} %{NOTSPACE:api_request}, Username: %{USERNAME:user_name}, StatusCode: %{WORD:status_code}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
}
if [indicator] == "End Request" {
grok { match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{DATA:method} - End Request. %{GREEDYDATA:indicator}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:request_bytes}, Elapsed Time: %{NUMBER:WITSML_elapsed_time}"}
}
}
}

output {

if "_grokparsefailure" in [tags] {
    file {
   path => "C:/ELK/WITSML/WITSML_logs/rigcloud_witsml/three/api_grokparsefailure.txt"
   }
}
else {
    elasticsearch {
    hosts => ["localhost:9200"]
   index => "api_th"
   template => "C:/ELK/mytemplate.json"
 }
}

stdout
{
 codec => rubydebug
}

}

data sample:

2019-10-22 18:42:26,850 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.AuthenticationModule.OnBeginRequest.0 - Begin Request. URI: /WitsmlStore.svc, Referrer: , UserHostAddress: 10.41.540.677, UserHostName: 10.41.540.677, TotalBytes: 1144
2019-10-22 18:42:26,858 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.AuthenticationModule.OnAuthenticateRequest.0 - Authorization Header: Basic Y2ggaXRoYW55YS52YW5ggbWE6Q2s4ODEgODg3OTY=
2019-10-22 18:42:26,861 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Authentication.GetCredentials.0 - Auth Header Value: Y2gggXRgYW55YS52YW5hbWE6Q2s4ODEgODc3OTY=
2019-10-22 18:42:26,862 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.Authentication.GetCredentials.0 - Auth Creds, username: bla.bla
2019-10-22 18:42:26,871 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Authentication.AuthenticateUsingAPI.0 - payload username: bla.bla
2019-10-22 18:42:31,004 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Security.Authentication.AuthenticateUsingAPI.0 - Response StatusCode: OK
2019-10-22 18:42:31,006 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Security.Authentication.AuthenticateUsingAPI.0 - Fetching headers from response
2019-10-22 18:42:31,022 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.Security.Authentication.AuthenticateUsingAPI.0 - Authentication Response for user: bla.bla. apiURI: http://10.167.65.6:9096/api/v1/security/authenticate, Header Values, token: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0=, useridbbr: 16789, livetoken: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0=, userrole: int_user, userid: bla.bla
2019-10-22 18:42:31,023 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] INFO RigCLOUD.WITSML.Authentication.AuthenticateUsingAPI.0 - apiURI: http://10.122.67.4:9077/api/v1/security/authenticate, Username: bla.bla, StatusCode: OK, Elapsed Time: 4161 ms
2019-10-22 18:42:31,038 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Security.Authentication.SetPrincipal.0 - Creating claims and identity
2019-10-22 18:42:31,066 [b713994b-87b0-4e41-bf94-f710d4d989d6] [17] DEBUG RigCLOUD.WITSML.Authentication.SetPrincipal.0 - Claims count: 6, Claims: ClaimTypes.NameIdentifier: bla.bla, ClaimTypes.Thumbprint: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0aO1V0noM5oShM=, ClaimTypes.UserData: 16069, ClaimTypes.Rsa: 6lnVU0jwnefMEJLZkKO/xaRU1l9DQ0aO1V0noM5oShM=, ClaimTypes.Role: int_user, ClaimTypes.Expiration: 6370736657384956067-7

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.