Logstash _grokparsefailure . Unable to find issue

Elastic Stack Logstash
1

Hello Loggers,

I have been receiveing a _grokparsefailure tag in Logstash but cannot find the issue. My Logstash Grok filter looks like this:

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:input}" }
}

I have no idea what the issue is. My log contains an ISO8601 timestamp "2016-09-28T09:44:22,533-0700 data-blah blah \r" and a carriage return character at the end that is not visible. Should'nt GREEDYDATA pickup the carriage return symbol as well? I have also tried using my timestamp with a space intead of T "2016-09-28 09:44:22,533-0700" but I get the same issue. The space between the timestamp and GREDDYDATA exist, which is why a have a space after my timestamp in my grok filter. Thanks in advance guys, chromehris.

2

Also, I want to mention that I also have a KV filter and a Mutate filter, but I believe "_grokparsefailure" is only caused by a grok filter correct?

3

Works for me:

$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:input}" }
  }
}
$ echo "2016-09-28T09:44:22,533-0700 data-blah blah\r" | logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "2016-09-28T09:44:22,533-0700 data-blah blah\r",
      "@version" => "1",
    "@timestamp" => "2016-09-29T20:29:50.414Z",
          "host" => "bertie",
     "timestamp" => "2016-09-28T09:44:22,533-0700",
         "input" => " data-blah blah\r"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

but I believe "_grokparsefailure" is only caused by a grok filter correct?

Yes.

4

Thanks Magnus! Man, sorry about that, I forgot to place a space between TImestamp & Greedydata.

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:input}" }
}

Also, the "\r" is invisible as text. It's a "Carriage Return" special character. I am thinking it is this special character that gives the issue.

5

Also, the "\r" is invisible as text. It's a "Carriage Return" special character.

Yes, of course.

I am thinking it is this special character that gives the issue.

What issue? You may want to use the mutate filter's strip option to remove the trailing carriage return.

6

This is my full filter which removes the "\r" carraige return special character:

filter {
if [type] == "businesslog" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:input}" }
}
kv {
source => "input"
field_split => "|"
value_split => "="
}
mutate {
remove_field => [ "input" ]
gsub => [
"*", "\r", ""
]
}
}
}

Then, I tried removing the special character before my grok filter:

filter {
if [type] == "businesslog" {
if [message] =~ "\r" {
mutate {
gsub => ["", "\r", ""]
}
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:input}" }
}
kv {
source => "input"
field_split => "|"
value_split => "="
}
# mutate {
# remove_field => [ "input" ]
# gsub => [
# "", "\r", ""
# ]
# }
}
}

I still get a grok parse failure. My logs look like:

2016-09-28 12:44:22,533-0700 key=00000|key=9392000|key=99383|key=7438932883\r

The "\r" at the end symbolizes the special character. I am not sure what could be cuasing this _grokparsefailure anymore.

7

Also tried this filter, in case the message field did not exists yet:

filter {
if [type] == "businesslog" {
# if [message] =~ "\r" {
mutate {
gsub => ["", "\r", ""]
}
# }
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:input}" }
}
kv {
source => "input"
field_split => "|"
value_split => "="
}
# mutate {
# remove_field => [ "input" ]
# gsub => [
# "", "\r", ""
# ]
# }
}
}

8

And this one, specifically stating the message field in gsub before the grok filter:

filter {
if [type] == "businesslog" {
#if [message] =~ "\r" {
mutate {
gsub => ["message", "\r", ""]
}
# }
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:input}" }
}
kv {
source => "input"
field_split => "|"
value_split => "="
}
# mutate {
# remove_field => [ "input" ]
# gsub => [
# "*", "\r", ""
# ]
# }
}
}

After all these attempts, I still receive a _grokparsefailure

9

Did you try the mutate filter's strip option?

10

Isn't strip only for whitespace & newline?

I tried:

filter {
if [type] == "businesslog" {
#if [message] =~ "\r" {
mutate {
strip => ["message"]
}
# mutate {
# gsub => ["message", "\r", ""]
# }
# }
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:input}" }
}
kv {
source => "input"
field_split => "|"
value_split => "="
}
mutate {
remove_field => [ "input" ]
#gsub => [
# "*", "\r", ""
#]
}
}
}

Still _grokparsefailure

Could it be because the "message" field does not exists yet at the time that I am using the mutate filter on it?

11

I have even removed the "/r" from my logs, so I changed my filter to:

filter {
if [type] == "businesslog" {
mutate {
strip => ["message"]
#gsub => [ "message", "\r", "" ]
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:input}" }
}
kv {
source => "input"
field_split => "|"
value_split => "="
}
mutate {
remove_field => [ "input" ]
#gsub => [
# "*", "\r", ""
#]
}
}
}

I still get a _grokparsefailure, even though all my data is coming in accurately. It'sjust uncomftrable to have that _grokparsefailure.

12

Are you sure you don't have any other grok filters laying around in your configuration files? Remember that Logstash reads all files in /etc/logstash/conf.d. As I showed in a previous example the grok filter does work and does not add a _grokparsefailure tag. It seems you're either not showing us all of your configuration or you're not using the configuration you think you are. Replicating my example could be a good start.

1 Like
13

I indeed am using multiple config files under /conf.d, one is an output and the others are input-&-filter each . My configuration however passes the configtest. How can multiple configuration files affect a _grokparsefailure tag in a certain message that comes from a config file which does not create such a tag? Since I most certainly know that my configuration file that I have originally mentioned should not cause a _grokparsefailure. I have tries different filters and I still receive this _grokparsefailure, so this tag must be coming from another source.

14

My configuration however passes the configtest.

That basically just means it's syntactically correct. Whether it does what you expect it to is a very different matter.

How can multiple configuration files affect a _grokparsefailure tag in a certain message that comes from a config file which does not create such a tag?

Another file could contain a grok filter that isn't under a condition, meaning it applies to all events.

file A:

filter {
  grok {
    match => ["message", "this pattern doesn't match anything"]
  }
}

file B:

filter {
  if [type] == "sometype" {
    grok {
      match => ["message", "this pattern matches my log file"]
    }
  }
}

In this example, every single event will get the _grokparsefailure tag since the grok filter in file A will apply to all events but will match none of them.

1 Like
15

If I have:
File A
filter {
if [app] == "app1" {
mutate {
add_field => { "type" => "type1" }
}
grok {
match => ["message", "this pattern matches my log file"]
}
}
}

File B
filter {
if [app] == "app2" {
mutate {
add_field => { "type" => "type1" }
}
grok {
match => ["message", "this pattern matches my log file"]
}
}
}

Will there be any issues with both of these having the same type, even though I first split them by a different "app" field value?

16

This should be okay, but keep in mind that add_field appends to any existing values so you'll probably end up with a type field that's a two-element array. Prefer the mutate filter's replace option.