I've been following this guide to learn how to use enrich processor and ingestion pipelines:
I've got most things working so I tried a more realistic workflow that goes something like this:
#!/bin/bash
/usr/share/logstash/bin/logstash -e 'input {file {path="/home/user/foo.csv"} } output { elasticsearch {...connection details...}}'
curl -X PUT -u elastic:mypass "https://example.com:9200/_enrich/policy/foo-policy" \
-H "Content-Type: application/json" \
-d "...some policy that depends on the `foo` index..."
sleep 60
curl -X PUT -u elastic:mypass "https://example.com:9200/_enrich/policy/foo-policy/_execute"
# ...define some pipelines that use my `foo-policy` and then use the pipelines...
Everything above works fine. But if I remove the line sleep 60, then my pipelines fail to enrich any data from the foo policy.
From what I can tell, after logstash has sent data to elastic, elastic still needs time to process them and store them. If I run my foo-policy/_execute too early, then the foo data won't be available for ingestion pipelines further down in my workflow.
Is there a way to tell elastic to foo-policy/_execute only after it has processed my latest batch of data?