Trying to separate key-value pairs in nested array (help with RUBY)


(jainh) #1

I have been trying to import my mySQL database into Elasticsearch through Logstash but I am stuck (for two days now) to separate three columns into separate fields.

I want to split these into different fields so it displays as:

"dr_behaviour_Patient Healer": "5", 
"dr_behaviour_Couldn’t Care Less": "5"
filter {
  mutate {
    split => {
        "dr_behaviour_rate" => "," 
        }
    }
  kv {
    source => "dr_behaviour_rate"
    prefix => "dr_behaviour"
    field_split_pattern => "/[/]"
    include_brackets => true
  }
}

This is the actual data that is interpreted by Logstash/Elasticsearch

This is how the data is being changed through the above query:


#2

I think you have to play on the value-split options.

Try something like this :

 filter {
  kv {
    source => "dr_behaviour_rate"
    prefix => "dr_behaviour_"
    field_split => ","
    include_brackets => false
    value-split => "\["
    trim_key => "\s"
    trim_value => "\s"
  }
}

(jainh) #3

Thank you. This solved it. One issue remains. I am getting the data as:

"dr_behaviour_Patient Healer": "1]",

How do I remove the last "]"?


#4

I thought the " include_brackets => false" will remove it but I think you can add these symbols in the trim options :

 filter {
  kv {
    source => "dr_behaviour_rate"
    prefix => "dr_behaviour_"
    field_split => ","
    include_brackets => false
    value-split => "\["
    trim_key => "\s\[\]"
    trim_value => "\s\[\]"
  }
}

#5

Keep in mind that if your key has a "[" into it, this won't work ...


(jainh) #6

Yes, you are right. It works. I will ensure that the key will not have square brackets. Thank you very much!!! :slight_smile:

If I have other fields with similar issues, should I add another set of kv filter with that field as the source? In this way, I will have three sets of KV filters with different source and prefix. for example,

 filter {
  kv {
    source => "dr_orientation"
    prefix => "dr_orientation_"
    field_split => ","
    include_brackets => false
    value-split => "\["
    trim_key => "\s\[\]"
    trim_value => "\s\[\]"
  }
}

#7

Yes it should work but with only one filter block !

filter {
  kv {
    ...
  }
  kv {
    ...
  }
  kv {
    ...
  }
}

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.