Logstash config for splitting array into new events


(Александр Кобрин) #1

Hi there. I'm trying to split array into separate log files. Input is a JSON like this

{
  "data": [
    {
      "field1": "val1",
      "field2": "val2"
    },
    {
      "field1": "val1",
      "field2": "val2"
    },
    ...
  ]
}

And i use conf like this:

    input {
      file {
        path => [ "/usr/local/etc/logstash/multi/*.json"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        type => "kitlog-multi"
      }
    }
 
filter {
  json {
    source => "message"
    target => "message"
  }
  if [type] == "kitlog-multi"{
    split {
      field => "[data]"
    }
  }
}
 
output {
  if [type] == "kitlog-multi" {
    elasticsearch {
      hosts => ["127.0.0.1:9200"] 
      index => "kitlog-multi"
    }
  }
  
  stdout {}
}

But all objects from "data" are coming to ES as a single log anyways, they are just like comma-separated
message.data.field1 = value1, value2, ....
message.data.field2 = value1, value2, ...

Any ideas why it is not splitting correctly? Thanks in advance


(Lewis Barclay) #2

I mean to me this is expected behaviour, in your example you have two fields with the same name in the same object? You have two message.data.field1 so how do you expect it to behave?