Grokparsefailure and don't understand why

Elastic Stack Elasticsearch
#1

I have a message that is driving me nuts and I don't know how to fix it.
For some reason I'm getting a _grokparsefailure for every one the entries I
have for one of our blade enclosures. Everything that I can see shows that
it's actually working the right way and doing everything I ask it to.

I did test this out with the grokdebug https://grokdebug.herokuapp.com/
and it works exactly the way I expect it to. I also did the config check
and it came back fine as well.

There is nothing in the logstash logs that show any reason why it's
failing. In fact I stripped my test machine of all of the other config
information and only have this one thing hitting.

The raw messages are below. Please note that there is no year in the
message, don't ask me why.

<12> Nov 13 10:06:00 8.10.255.10 OA: Administrator logged into the Onboard
Administrator from 10.121.255.205
<12> Nov 13 17:16:10 8.10.255.10 OA: Administrator logged out of the
Onboard Administrator

My config looks like this.

input {
redis {
host => "8.8.8.8"
data_type => "list"
type => "redis-input"
key => "logstash"
}
syslog {
type => "HPOAEvents"
port => 1551
}
}

filter {
if [type] == "HPOAEvents" {
dns {
reverse => [ "host" ]
action => "replace"
}
mutate {
add_tag => [ "HPOnboardAdminEventlog" ]
}
}
}

filter {
if [type] == "HPOAEvents" {
grok {
match => [
"message", "<%{POSINT:syslog_pri}>
(?%{MONTH} %{MONTHDAY} %{TIME}) %{IPORHOST:syslog_hostname}
%{GREEDYDATA:syslog_message}"
]
}
mutate {
replace => [ "@source_host", "%{host}" ]
replace => [ "@source_host_ip",
"%{syslog_hostname}" ] #ip comes across into this field so just moving it
to the right place.
rename => [ "syslog_message", "@message" ]
rename => [ "timestamp", "@time" ]
}
}
}

output {
stdout { }
elasticsearch {
cluster => "logstash"
protocol => "http"
}
}

Here is the output.
{
"_index": "logstash-2014.11.14",
"_type": "HPOAEvents",
"_id": "uVyt2i8CQfiSkGyowdfqlw",
"_score": null,
"_source": {
"message": "<12> Nov 13 10:06:01 8.10.255.10 OA: Administrator logged
out of the Onboard Administrator",
"@version": "1",
"@timestamp": "2014-11-14T15:06:00.274Z",
"type": "HPOAEvents",
"host": "oa1enc1rk5.mydomain.com",
"tags": [
"_grokparsefailure",
"HPOnboardAdminEventlog"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice",
"syslog_pri": "12",
"syslog_hostname": "8.8.8.8",
"@message": "OA: Administrator logged out of the Onboard Administrator",
"@time": "Nov 14 10:06:01",
"@source_host": "oa1enc1rk5.mydomain.com",
"@source_host_ip": "8.8.8.8"
},
"sort": [
1415977560274,
1415977560274

Is there somewhere else I should be looking for the parse failure?

Thanks in advance,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/333804ad-2354-4abf-9a46-7e797dae3949%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#2

Try to add a tag_on_failure on each grok filter to identify which grok
filter is failing.
I had the same issue and by explicitly setting a tag on each grok I could
determine the one causing issues.

Op vrijdag 14 november 2014 16:23:41 UTC+1 schreef Billy F:

I have a message that is driving me nuts and I don't know how to fix it.
For some reason I'm getting a _grokparsefailure for every one the entries I
have for one of our blade enclosures. Everything that I can see shows that
it's actually working the right way and doing everything I ask it to.

I did test this out with the grokdebug https://grokdebug.herokuapp.com/
and it works exactly the way I expect it to. I also did the config check
and it came back fine as well.

There is nothing in the logstash logs that show any reason why it's
failing. In fact I stripped my test machine of all of the other config
information and only have this one thing hitting.

The raw messages are below. Please note that there is no year in the
message, don't ask me why.

<12> Nov 13 10:06:00 8.10.255.10 OA: Administrator logged into the Onboard
Administrator from 10.121.255.205
<12> Nov 13 17:16:10 8.10.255.10 OA: Administrator logged out of the
Onboard Administrator

My config looks like this.

input {
redis {
host => "8.8.8.8"
data_type => "list"
type => "redis-input"
key => "logstash"
}
syslog {
type => "HPOAEvents"
port => 1551
}
}

filter {
if [type] == "HPOAEvents" {
dns {
reverse => [ "host" ]
action => "replace"
}
mutate {
add_tag => [ "HPOnboardAdminEventlog" ]
}
}
}

filter {
if [type] == "HPOAEvents" {
grok {
match => [
"message", "<%{POSINT:syslog_pri}>
(?%{MONTH} %{MONTHDAY} %{TIME}) %{IPORHOST:syslog_hostname}
%{GREEDYDATA:syslog_message}"
]
}
mutate {
replace => [ "@source_host", "%{host}" ]
replace => [ "@source_host_ip",
"%{syslog_hostname}" ] #ip comes across into this field so just moving it
to the right place.
rename => [ "syslog_message", "@message" ]
rename => [ "timestamp", "@time" ]
}
}
}

output {
stdout { }
elasticsearch {
cluster => "logstash"
protocol => "http"
}
}

Here is the output.
{
"_index": "logstash-2014.11.14",
"_type": "HPOAEvents",
"_id": "uVyt2i8CQfiSkGyowdfqlw",
"_score": null,
"_source": {
"message": "<12> Nov 13 10:06:01 8.10.255.10 OA: Administrator logged
out of the Onboard Administrator",
"@version": "1",
"@timestamp": "2014-11-14T15:06:00.274Z",
"type": "HPOAEvents",
"host": "oa1enc1rk5.mydomain.com",
"tags": [
"_grokparsefailure",
"HPOnboardAdminEventlog"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice",
"syslog_pri": "12",
"syslog_hostname": "8.8.8.8",
"@message": "OA: Administrator logged out of the Onboard
Administrator",
"@time": "Nov 14 10:06:01",
"@source_host": "oa1enc1rk5.mydomain.com",
"@source_host_ip": "8.8.8.8"
},
"sort": [
1415977560274,
1415977560274

Is there somewhere else I should be looking for the parse failure?

Thanks in advance,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bd15cb8d-f1a1-470e-b662-f56bab950938%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#3

Great suggestion, I didn't know about that function. And it's something I
will incorporate into my larger config that has multiple filters.

But....even with the addition of tag_on_failure it still returns nothing
except the _grokparsefailure. There are no other grok in this config
except this one.

I know it's not the best idea, but my only other option is......

remove_tag => "_grokparsefailure"

Like I said, the actual filter and everything else works just fine.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ff5d923-f415-49fd-b224-7851868c32d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#4