I am looking into a highly available solution to feed in AWS RDS transactions into my Elasticsearch repository. There are some articles on this topic. For example:
But this is not a flexible and agile in the long-term for changes. It seems that the best way to do this is through AWS DMS. The AWS documentation seems to imply using Logstash. I have seen the following article:
This article seems promising. But I have some concerns. For example, what happens if the node goes down. How can we save and keep track of the value for "tracking_column" which is stored on disk at .logstash_jdbc_last_run.
Can we store this value in a database because it seems that only one instance of Logstash can be used at a time with this setup.
Thanking you in advance for any help.