we have 2 logstash servers with same configurations, connecting to one database server through their jdbc input plugins.
The idea was to have a highly available logstash.
Both LS servers are using the same .logstash_jdbc_last_run file kept on shared file system. The schedules of jdbc input plugins are firing at different times, updating the .logstash_jdbc_last_run file and making sure same data wont be read again.
However, it seems LS is reading the .logstash_jdbc_last_run only on start up, so if LS1 updates the file, it is not read by LS2 on the next sql run.
Is there a way to make LS read the .logstash_jdbc_last_run on each run of sql statement?
I am not aware that it is possible to configure this. What happens if logstash A reads the file and starts loading the data but before it was able to write the file logstash B reads it and gets the data again?
The only idea I have is:
Configure both Logstash servers that each server has their own file - which is cleaner in my opinion.
In the Logstash pipeline add code that the document ID is generated from the content(e.g. the database id or something similar). This way, ElasticSearch would recognise that the entry already exists in the database and would update it instead of creating a new document.
You would still have the overhead of ingesting each message twice(most of the time). But you would have each message only once in ElasticSearch and you would still be highly available.
We control the schedules of both LS servers to make sure there is no overlapping. The data is ingested once a day, so creating a 3 hour offset between two schedules prevents the problem you described.
We are aware of the alternative you have suggested but as you said, there is the overhead of dealing with each message twice...
As you say, the last run file is only read at startup. The documentation says that if you have multiple statements they must use different last run files. That applies to multiple instances too.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.