Execute policy synchronously without a `sleep` in my bash scripts

I've been following this guide to learn how to use enrich processor and ingestion pipelines:

I've got most things working so I tried a more realistic workflow that goes something like this:


/usr/share/logstash/bin/logstash -e 'input {file {path="/home/user/foo.csv"} } output { elasticsearch {...connection details...}}'

curl -X PUT -u elastic:mypass "https://example.com:9200/_enrich/policy/foo-policy" \
-H "Content-Type: application/json" \
-d "...some policy that depends on the `foo` index..."

sleep 60

curl -X PUT -u elastic:mypass "https://example.com:9200/_enrich/policy/foo-policy/_execute"

# ...define some pipelines that use my `foo-policy` and then use the pipelines...

Everything above works fine. But if I remove the line sleep 60, then my pipelines fail to enrich any data from the foo policy.

From what I can tell, after logstash has sent data to elastic, elastic still needs time to process them and store them. If I run my foo-policy/_execute too early, then the foo data won't be available for ingestion pipelines further down in my workflow.

Is there a way to tell elastic to foo-policy/_execute only after it has processed my latest batch of data?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.