Jdbc input plugin full and incremental updates

Elastic Stack Logstash
1

I'm using the jdbc input plugin to feed MySQL data into elasticsearch. I'm using sql_last_value to send incremental updates to elastic. Is there an automated way to send full updates to elasticsearch say every 24 hours and incremental updates every 5 min?

I know that we can use clean_run to do that, however the configuration file needs to be changed and is a manual process.

Please let me know your thoughts.

2

Use two inputs on different schedules?

3

I have two concerns taking this approach.

  • I have around 50 indices, so that approach would make the config complex.

  • What are implications of two output plugins trying to write to elastic at the same time?

I'm thinking if I should set up a cron to delete .logstash_jdbc_last_run file(s). Has anyone tried that? Any thoughts on this approach?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.