I'm new guys in logstash.
I briefly understand what is logstash and need to create my own filter.
Below is my situation.
- We have own solution and keep the data in Database.
- We built our 1st elastic server using river technology.
- Bulk indexing for current data
- Delta indexing in every 5 mins.
- We want to replace our river based technology to logstash
For #2, we have to find a solution. Especially, we need a solution for delta indexing at logstash.
Below is my approach.
- At DB, register a trigger.The trigger log changing at logstash with primary key.
- At logstash, get final data at DB using primary key and forwarding to elastic search.
I think we need to create custom filter to pull the data from DB and formatted for elastic search.
For this, is there any tutorial for creating custom filter ?
I also curios whether elastic search just re-index fields which in logstash output,
In river, elastic search re-build whole document using query result. So we have to join whole tables although just changed one field.