Implement custom code in Filter plugin for Logstash

Elastic Stack Logstash
1

While ingesting the data we can using Filters like

filter {
mutate {
copy => { "source_field" => "dest_field" }
}
}

But we need to implement as listed below..

· Proper case
· Proper punctuation
· Avoid duplicate titles
· Line breaks should be html tag < br >, < p >
· Bullets should be bullets (not “_”, “-”, “;”,“~”)

Is it possible using out of the box filters or can we use our custom python code or any other language script to transform the data before storing into Elasticsearch ?

2

You can use:

  • mutate, uppercase, capitalize,lowercase,gsub, etc.
  • Avoid duplicate titles - not clear what does it means, ruby code could help
  • Line breaks, bullets,... gsub should help.
    Check other the filter plugins or use ruby code
3

Thanks.. Pls share some ruby code for example make the string into uppercase

4

You can use mutate or ruby code, it's up to you

input {
  generator {
       message => "Test message"
	   count => 1
  }
}
filter {

   mutate {
     copy => { "message" => "msgmutate" }
   }
   mutate { 
     lowercase => [ "msgmutate" ]
   }

    ruby {
     code => "event.set('message', event.get('message').upcase )"
	}

    mutate { remove_field => [  "host", "log" ] }

}

output {
    stdout { }
}

Output:

{
         "event" => {
        "original" => "Test message",
        "sequence" => 0
    },
       "message" => "TEST MESSAGE",
     "msgmutate" => "test message"
}
1 Like