How to Configure Logstash to Forward Data When New Data is Available in Elasticsearch?

Dear All.

I am a beginner and I have the following questions to ask:

Environment: I have a program that fetches data from a network device API every five minutes and stores it into an Elasticsearch index. The data in the index is continuously overwritten, so the index size remains fixed. It takes about three minutes to fetch and fully store the data into the index. After that, I need to forward the data using Logstash via UDP. The data is fetched every 5 minutes and sent over UDP.

Current issue: Since it takes about 3 minutes to write data into the index, my Logstash pipeline only has 1 to 2 minutes to fetch the current data. If the data in the index hasn’t finished overwriting yet and Logstash starts running, it will miss the data. Is there a way to configure Logstash to send data only when new data has been fully written into the index?

Hi @Wei_Li,

Welcome! Some more information would be useful to help you.

  1. Can you confirm that this is timestamped data?
  2. Where are you forwarding the Elasticsearch data to using Logstash?
  3. Is there any particular reason you are overwriting the entire index rather than use ILM to delete data based on the timestamp?

You could probably use Logstash and the Elasticsearch input plugin to pull the data out. It's possible to provide a CRON schedule and query as part of the plugin configuration to get the data you need.

Hope that helps!