Kafka_consumer_lag in Logstash

Elastic Stack Logstash
1

Hello,

Is it possible to get the value or verify value of “kafka_consumer_lag” directly in Logstash 7.17?

The idea is to verify:
if [kafka_consumer_lag][lag] == 0

In Logstash Output and if yes run http plugin with
url => http://localhost:5601/_enrich/policy/policy_name/_execute

To execute enrich policy every time source data are updated, instead of manual execution which is needed by the default Elastic design. So, the enrich processor pipeline automatically updates the final index with fresh data from the source.

Or is there another way how to automate enrich policy execution?

I know about Enrich processor: allow scheduling of policy executions · Issue #50071 · elastic/elasticsearch · GitHub “Enrich processor: allow scheduling of policy executions”, which is still not resolved.

Thank you

2

No, it is not possible.

The easiest way is to have a crontab to run the _execute, this is what I did in my case, every couple of hours I make a request to _execut the policy and update the enrich index if there are any changes.

3

Thank you for you reply @leandrojmp
Could you pls show me how did you set up the crontab?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.