Grok parse failed, works good with grok debugger

Elastic Stack Logstash
1

Can someone please help me with this. I tried almost all possible ways but no luck. My grok debugger works perfect but get "beats_input_codec_plain_applied, failedparse" error in Kibana from logstash.

This is the grok I am using:

input {
beats {
port => 5054
}
}

filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logdate}}"}
# if we can't parse the log message then add a tag to it to identify it
tag_on_failure => [ "failedparse" ]
}
# date match on the date of the log entry using a custom format match, not specifying a target so this date result will go into the @timestamp field
date {
match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
}
# date match on the date of the log entry using a custom format match, specify the target to put this date result in the logdate field
date {
match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
target => "logdate"
}
}

output {
elasticsearch {
# output to elasticsearh, use templates to parse the data and save in an index noliouat-request then append the year and month
hosts => ["localhost:9200"]
manage_template => true
index => "uat-dmall-%{+YYYY.MM}"
}
}

=============================================================
Sample Log file:

2017-10-26 19:17:59,894 [org.springframework.jms.listener.DefaultMessageListenerContainer#3-1] INFO (com.app.platform.server.dataservices.services.execmng.ConnectivityUpdaterImpl:92) - Done updating connectivity: es_hostname
2017-10-26 19:18:04,216 [ExecutionServerStatusTask-41322] DEBUG (com.app.platform.server.dataservices.services.execmng.ExecutionServerStatusMonitor:92) - Update execution servers status
2017-10-26 19:17:53,633 [JobDisposer-1] INFO (com.app.platform.server.dataservices.services.execution.ExecutionServiceImpl:1262) - disposing job [2162707].

Appreciate a quick turn around.

2

There are two } in your grok expression.

3

I see there are the equal number of opening and closing { }. Can you advise what I am overlooking?

4

The grok expression, i.e. the string between the double quotes, is %{TIMESTAMP_ISO8601:logdate}}. That's one { and two }:

5

Thanks..figured that out. But I still have Parse failed error from my grok... Can you please suggest the best grok pattern to handle this....

=======================================================================

input {
beats {
port => 5054
}
}

filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:tempdate} [%{NOTSPACE:thread} %{WORD:type}%{SPACE}(%{JAVACLASS:program}:%{NUMBER:programline}) - %{GREEDYDATA:logmessage}"}
# if we can't parse the log message then add a tag to it to identify it
tag_on_failure => [ "failedparse" ]
}
mutate {
split => { "tempadate" => "," }
add_field => { "date" => "%{tempdate[0]}" }
remove_field => [ "tempdate" ]
}
# date match on the date of the log entry using a custom format match, not specifying a target so this date result will go into the @timestamp field
date {
match => [ "date", "YYYY-MM-dd HH:mm:ss" ]
}
}

======================================================================================
Sample log file:

2017-10-30 14:34:13,362 [JobDisposer-1] INFO (com.platform.server.dataservices.services.execution.ExecutionServiceImpl:1262) - disposing job [1441868].
2017-10-30 14:34:45,958 [ExecutionServerStatusTask-13983] DEBUG (com.platform.server.dataservices.services.execmng.ExecutionServerStatusMonitor:92) - Update servers status
2017-10-30 14:34:46,514 [Finalizer] DEBUG (com.platform.connection.PoolingClientConnectionManager:275) - Connection manager is shutting down
2017-10-30 14:34:48,703 [eventsProcessingHandler[PerJobExecutorsManager_garbageCollector]-1] DEBUG (com..platform.server.dataservices.services.executors.PerJobExecutorsManagerImpl:232) - Shutting down executors for finished jobs and cleaning up resources...
2017-10-30 14:34:48,703 [AgentHttpRequestsHandler-[PerJobExecutorsManager_garbageCollector]-1] DEBUG (com.platform.server.dataservices.services.executors.PerJobExecutorsManagerImpl:232) - Shutting down executors for finished jobs and cleaning up resources...
2017-10-30 14:34:48,713 [org.springframework.jms.listener.DefaultMessageListenerContainer#3-1] INFO (com.platform.server.dataservices.services.execmng.ConnectivityUpdaterImpl:83) - Updating connectivity. current details: Execut

==================================================================

The issue I think is that the parse is failing while handling the major difference in between lines with [.......] and [...[....]] Please advise on how this can be handled.

6

Please post your configuration as preformatted text so that backslashes don't get stripped.

[%{NOTSPACE:thread}

Where's the closing square bracket?

mutate {
split => { "tempadate" => "," }
add_field => { "date" => "%{tempdate[0]}" }
remove_field => [ "tempdate" ]
}

Why on earth are you doing this?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.