Create documents from array field

Elastic Stack Logstash
1

Hello all,

this is the output of my pipeline that process a file. I've succesfully extract the lines i want from the file, each file per day and n number of lines each day and split the lines into and array.

{
  "host" => "",
        "field1" => [
        [0] " 1/17/0 AA 3 MAR 20 SAB R-XXX 3801.00 EU",
        [1] " 1/17/0 BB 2 JAN 20 TER GCX 12.6000 US",
    ],
    "@timestamp" => 2020-01-23T20:36:42.037Z
}

What i need is create a document of each array element like this

{
  "field1" => "AA",
  "field2" => "3",
  "field3" => "MAR 20",
  "field4" => "SAB R-XXX",
  "field5" => "3000.00 EU",
  "@timestamp" => "2020-01-23T20:36:42.037Z"
}
{
  "field1" => "BB",
  "field2" => "2",
  "field3" => "JAN 20",
  "field4" => "TER GCX",
  "field5" => "12.6000 US",
  "@timestamp" => "2020-01-23T20:36:42.037Z"
}

Is it posible to achieve this?

Best regards

2

You can use a split filter to split an array into multiple events, then use dissect. Something like

dissect { mapping => { "field1" => "%{} %{field1} %{field2} %{+field2} %{field3} %{+field3} %{field4} %{+field4} %{field5}" } }

Note that field1 is both an input and an output. I am unsure what the dissect filter will do there. You may end up with field1 as an array (again).

3

Thank you Badger,

how is the split syntax?

split{
     split_in_documents => "field1"
}

Not working

4

See the documentation.

split { field => "field1" }

Note that I misread your original post and thought the field was named "field", not "field1", so I have updated my previous post.

1 Like
5

Thank you so much Badger

6

Badger,

split filter is working perfectly to split an array field in separate documents, i wonder why this is apearing in tags although is working.

"tags" => [
        [0] "_split_type_failure"
7

There should be a related error message in the logstash log.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.