Tutorial for creating custom filter

Elastic Stack Logstash
1

Hi All.

I'm new guys in logstash.
I briefly understand what is logstash and need to create my own filter.

Below is my situation.

  1. We have own solution and keep the data in Database.
  2. We built our 1st elastic server using river technology.
    • Bulk indexing for current data
    • Delta indexing in every 5 mins.
  3. We want to replace our river based technology to logstash

For #2, we have to find a solution. Especially, we need a solution for delta indexing at logstash.

Below is my approach.

  • At DB, register a trigger.The trigger log changing at logstash with primary key.
  • At logstash, get final data at DB using primary key and forwarding to elastic search.

I think we need to create custom filter to pull the data from DB and formatted for elastic search.

For this, is there any tutorial for creating custom filter ?
I also curios whether elastic search just re-index fields which in logstash output,
In river, elastic search re-build whole document using query result. So we have to join whole tables although just changed one field.

Thanks

2

Yes! See the official documentation: How to write a Logstash filter plugin