Trying to separate key-value pairs in nested array (help with RUBY)

[Harshet_Jain (jainh)]

I have been trying to import my mySQL database into Elasticsearch through Logstash but I am stuck (for two days now) to separate three columns into separate fields.

I want to split these into different fields so it displays as:

"dr_behaviour_Patient Healer": "5", 
"dr_behaviour_Couldn’t Care Less": "5"

filter {
  mutate {
    split => {
        "dr_behaviour_rate" => "," 
        }
    }
  kv {
    source => "dr_behaviour_rate"
    prefix => "dr_behaviour"
    field_split_pattern => "/[/]"
    include_brackets => true
  }
}

This is the actual data that is interpreted by Logstash/Elasticsearch

This is how the data is being changed through the above query:

[OphyTe]

I think you have to play on the value-split options.

Try something like this :

 filter {
  kv {
    source => "dr_behaviour_rate"
    prefix => "dr_behaviour_"
    field_split => ","
    include_brackets => false
    value-split => "\["
    trim_key => "\s"
    trim_value => "\s"
  }
}

[Harshet_Jain (jainh)]

Thank you. This solved it. One issue remains. I am getting the data as:

"dr_behaviour_Patient Healer": "1]",

How do I remove the last "]"?

[OphyTe]

I thought the " include_brackets => false" will remove it but I think you can add these symbols in the trim options :

 filter {
  kv {
    source => "dr_behaviour_rate"
    prefix => "dr_behaviour_"
    field_split => ","
    include_brackets => false
    value-split => "\["
    trim_key => "\s\[\]"
    trim_value => "\s\[\]"
  }
}

[OphyTe]

Keep in mind that if your key has a "[" into it, this won't work ...

[Harshet_Jain (jainh)]

Yes, you are right. It works. I will ensure that the key will not have square brackets. Thank you very much!!!

If I have other fields with similar issues, should I add another set of kv filter with that field as the source? In this way, I will have three sets of KV filters with different source and prefix. for example,

 filter {
  kv {
    source => "dr_orientation"
    prefix => "dr_orientation_"
    field_split => ","
    include_brackets => false
    value-split => "\["
    trim_key => "\s\[\]"
    trim_value => "\s\[\]"
  }
}

[OphyTe]

Yes it should work but with only one filter block !

filter {
  kv {
    ...
  }
  kv {
    ...
  }
  kv {
    ...
  }
}