Is it possible to get the value or verify value of “kafka_consumer_lag” directly in Logstash 7.17?
The idea is to verify:
if [kafka_consumer_lag][lag] == 0
In Logstash Output and if yes run http plugin with
url => http://localhost:5601/_enrich/policy/policy_name/_execute
To execute enrich policy every time source data are updated, instead of manual execution which is needed by the default Elastic design. So, the enrich processor pipeline automatically updates the final index with fresh data from the source.
Or is there another way how to automate enrich policy execution?
I know about Enrich processor: allow scheduling of policy executions · Issue #50071 · elastic/elasticsearch · GitHub “Enrich processor: allow scheduling of policy executions”, which is still not resolved.