Multiline and Ingest Node

Elastic Stack Elasticsearch
1

Hi,

I am having Filebeat 5 sending multiline events to an Elastic Ingest node, which in turn, runs the grok processor.
However the grok processor, takes only the first line of the data.
The end of each line is a stack trace start message, which I would like to have all data, and using GREEDYDATA pattern.

Is it possible to ingest multiline events with the Ingest node ?

Thanks,

Ori

2

@ori.rubinfeld. just to be sure, you are receiving documents of this nature?

{
"message": "first line\nsecond line\nthird line"
}
3

Hi,

I am receiving lines like this:

27/08/2016 1:00:23 PM|some text1|some text2|first line
second line
third line
.....

Using the GREEDYDATA at the end, I was able to extract only the first line.

I found a workaround for now, but would like to have another solution:
I am replacing using gsub all the followings, before the GROK:

\n --> @@n@@
\r --> @@r@@
\t --> @@t@@

Then, after extracting the message, I am doing another replace to the original character using again the gsub processor.

@@n@@ --> \n
@@r@@ --> \r
@@t@@ --> \t

For now it is working great!!!
Data is being presented correctly in Kibana.

But I would like to be able just to use the GREEDYDATA like in Logstash to have all the Stack trace, and not adding these uses of gsub processor.

Ori

4

since GREEDYDATA does not span new-lines, you can introduce a new pattern that does

{
  "grok" : {
    "field" : "myField",
    "patterns" : [ "%{GREEDYMULTILINE:allMyData}" ]
    "pattern_definitions" : {
      "GREEDYMULTILINE" : "(.|\n)*"
    }
  }
}

This is pretty much a catch-all though, no different than the original value. I do not fully know the goal, so this may or may not help. Regardless, glad you were able to work around it with the help from other processors!

1 Like
5

Thanks a lot!!!!!
I will test it and update.
Will take a while, since currently not on it.

Ori

6

By the way, with the Logstash, GREEDYDATA takes all lines.
Why isn't it the same here ?
Shouldn't it just take the whole string (Including newline characters and carriage return) ?

Ori

7

I suppose that is how the regex engine treats .*