Logstash parsing

[dilipchiru (dilip bk)]

Hi Team,

I have 2 Fields which is From and TO which contains set of values which is comma separated.
For example:

"from" : "Loin, Elephant, cat, movie, John"
"to" : "Loin, Elephant, cat, movie, John, USA "

Now we would like to get the difference/unique values between the From & TO field and dynamically assign in it new field. As we can "USA" is the difference/unique .

Please suggest me to get this in the logstash using ruby

Thanks
DILIP BK

[aaron-nimocks]

Try something like this.

Conf

input {
  generator {
    lines => [
      '{"from": "Loin, Elephant, cat, butter, movie, John", "to": "Loin, Elephant, cat, movie, John, USA"}'
    ]
    codec => "json"
    count => 1
  }
}

filter {
  ruby {
    code => '
      from_array = event.get("from").split(", ").map(&:strip).to_set
      to_array = event.get("to").split(", ").map(&:strip).to_set

      difference = from_array ^ to_array

      event.set("unique_fields", difference.to_a.join(", "))
    '
  }
}

output {
  stdout { codec => json_lines }
}

Output

{
    "unique_fields": "USA, butter",
    "from": "Loin, Elephant, cat, butter, movie, John",
    "to": "Loin, Elephant, cat, movie, John, USA"
}

[dilipchiru (dilip bk)]

Hi ,

Thanks for the help. small suggestion need , where the values "from" & "to" are dynamic. can we ignore the input Generator plugin and directly we can add the grok pattern with the ruby script ??

Please help me with this

Thanks
DILIP BK

[aaron-nimocks]

The input generator is just to test the pipeline with the data. Replace the input with your input and as long as the data that's coming in looks like the data I used it should work.

[dilipchiru (dilip bk)]

Hi @aaron-nimocks ,

Thanks for the help, It is working as required. Solved my issue