Why _grokparsefailure?

[trelo]

Hi, here is my config:

input {
  stdin {}
  file {
    codec => multiline {
      pattern => "^{%{DATESTAMP_EVENTLOG}"
      what => "previous"
      negate => true
      multiline_tag => ""
    }
    path => "/tmp/testdata"
    start_position => "beginning"
    sincedb_path => "/dev/null"
#    break_on_match => true
  }
}

filter {
  mutate {
    gsub => ["message","\n",""]
}
  mutate {
    remove_field => [ "host", "@version", "path", "tags", "@timestamp", "@version"]
}
  grok {
    match => [ "message", "{%{DATESTAMP_EVENTLOG:date},%{DATA:sda},{%{DATA:qwe},%{INT:cxzc}},%{INT:iueqiwueiquweiq},%{INT:vcxv},%{INT:gff},%{INT:ifjidjfidjfijd},%{INT:ret},%{DATA:nbvn},\\"%{DATA:nbvn}\\",%{INT:vxcv},{\\"P\\",{6,{%{GREEDYDATA:zxckeowkek}},{%{GREEDYDATA:xczx}}}},\\"%{DATA:sdasdew}\\",%{INT:cxzsa},%{INT:nvbre},%{INT:fgwe34c},%{INT:nhrtyew},%{INT:dqrtq}%{GREEDYDATA:drop}" ]
  }
}


output {
    stdout { codec => rubydebug }
#    file { path => "/tmp/debug.out" }
}

Output:

{
       "tags" => [
        [0] "_grokparsefailure"
"message" => "{20181004010443,N,{0,0},640,12,3,20340532,3,s,\"\",0,{\"P\",{6,{\"S\",\"ДАННЫЕ\"},{\"S\",\"LOPS\\ДАННЫЕ\"}}},\"\",1,17,17,258497671,0,{0}},"
}

But why the result is _grokparsefailure? I tested it with Kibana grok debugger and it's working.

[trelo]

Please, help!

it works if send data to stdin, but doesn't with data from file. Why?

In file I've multiline events, they handled by multiline codec to be single line:

{20181004010443,N,{0,0},640,12,3,20340532,3,s,\"\",0,{\"P\",{6,{\"S\",\"ДАННЫЕ\"},{\"S\",\"LOPS\\ДАННЫЕ\"}}},\"\",1,17,17,258497671,0,{0}},

[Album] Imgur

[shwesinhan (ssh)]

i also have faced multi-line problem
i remove line breaks since app level eg Regex.Replace(str, @"\n|\r", ""); then pass to logstash
it works, no more _grokparsefailure on multiline logs

[trelo]

It's impossible in my case. I've more than 1TB plain log already.

[trelo]

The problem was with stdout rubydebug. With other outputs works well.