Logstash parsing

Hi Team,

I have 2 Fields which is From and TO which contains set of values which is comma separated.
For example:

"from" : "Loin, Elephant, cat, movie, John"
"to" : "Loin, Elephant, cat, movie, John, USA "

Now we would like to get the difference/unique values between the From & TO field and dynamically assign in it new field. As we can "USA" is the difference/unique .

Please suggest me to get this in the logstash using ruby


Try something like this.


input {
  generator {
    lines => [
      '{"from": "Loin, Elephant, cat, butter, movie, John", "to": "Loin, Elephant, cat, movie, John, USA"}'
    codec => "json"
    count => 1

filter {
  ruby {
    code => '
      from_array = event.get("from").split(", ").map(&:strip).to_set
      to_array = event.get("to").split(", ").map(&:strip).to_set

      difference = from_array ^ to_array

      event.set("unique_fields", difference.to_a.join(", "))

output {
  stdout { codec => json_lines }


    "unique_fields": "USA, butter",
    "from": "Loin, Elephant, cat, butter, movie, John",
    "to": "Loin, Elephant, cat, movie, John, USA"
1 Like

Hi ,

Thanks for the help. small suggestion need , where the values "from" & "to" are dynamic. can we ignore the input Generator plugin and directly we can add the grok pattern with the ruby script ??

Please help me with this


The input generator is just to test the pipeline with the data. Replace the input with your input and as long as the data that's coming in looks like the data I used it should work.

Hi @aaron-nimocks ,

Thanks for the help, It is working as required. Solved my issue

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.