Multiline Logstah file txt

[Daniel_Bergmann]

I have a TXT file that I want to import into Elasticsearch via LogStash,

it is a fixed type of columns, in which I used grok and if to separate the columns of this file, that's fine, but I want to use the multiline to get 'n' lines that will be inserted / consolidated in the same document in Elastic.
Each line of the file the first column is the type line, (it can be 1,2, or 3) and I would like to consolidate everything below 1 an event in the logstash, it closes the event when the next line is 1, it follows an example of lines in the txt file:

LINE 1:
1 ....
LINE 2:
2 ....
LINE 3:
2 ....
LINE 4:
3 ....
LINE 5:
1 ....
LINE 6:
1 ....
LINE 7:
2 ....
LINE 8:
1 ....

in the output it would be like this:
LINE 1, 2, 3, 4 - document elastic _id
LINE 5 - another document elastic _id
LINE 6 and 7 - another document elastic_id
LINE 8 - another document elastic_id

I have been researching but I can't find a correct pattern for this or not to do for Lostash but for a python reading the file and consolidating it into an array.

Can someone help me ?

Thank you
Daniel

[Badger]

Are "LINE 1:" etc just labels in your example or do they actually exist in the data?

Are you using a file input? If so, you might be able to do this with a multiline codec on the input. Alternatively an aggregate filter.

[Daniel_Bergmann]

Hi Badger, LINE 1: don´t exist only first column in line is (1,2 or 3)

yes a using file input.

Thank you

[Badger]

On the file input use

codec => multiline { pattern => '^1' negate => true what => previous auto_flush_interval => 1 }

then if you use mutate+split, with a literal newline in the configuration

    mutate { split => { "message" => "
" } }

you will get events like

{
"@timestamp" => 2020-08-03T20:40:30.716Z,
  "@version" => "1",
   "message" => [
    [0] "1 ....",
    [1] "2 ....",
    [2] "2 ....",
    [3] "3 ...."
],
      "tags" => [
    [0] "multiline"
],
      "host" => "myhost,
      "path" => "/home/user/foo.txt"
}

[Daniel_Bergmann]

Thank you it´s work, How do I apply grok to each line of the array [0] [1] [2] to separate the fields ?

Regards,
Daniel

[Badger]

I believe if you ask grok to match against a field that is an array it will iterate over the entries and try the pattern or patterns against each one. You may or may not like how it presents the results. I suggest you try it.

[Daniel_Bergmann]

Thank you answer, but howto make ? how scan array ?

Thanks