Improve support for Master Data and Enrichment

Elastic Stack Kibana
1

If you are setting up anything that looks like a datawarehouse or whatever, sooner or later you are going to have a need for some kind of "lookup tables". The lack of support for such thing in Kibana is really frustrating. This need is not something farefetched or exotic, it's like pretty basic stuff so I really can't understand that this is not included in an enterprise product like Elastic-Kibana.
Currently there is the "Enrich" processor in pipelines (which only has basic documentation) which needs an "Enrichment Policy" to work. This Enrichment Policy is basically a lookup table. But then again

  • There is no UI in Kibana to create/manage these. Why?
  • After you have create the Enrichment Policy, each time the data in the source index changes you need to update the Enrichment Policy. Then why can't this be a continuous "Latest"-type transformation?

So, lot's of questions concerning a lacking feature which I'd consider to be standard stuff for any data platform.

2

Yeah, doing enrichments with only Elasticsearch can be complicated, there is still a lot of things to be improved.

There were improvements on doing enrichments during query time when using ESQL, but not much changed for doing enrichments during ingestion time.

This is one of the reasons for why Logstash is still required, you can easily do enrichments on Logstash during ingestion.

There is an open issue about allowing Scheduling the execution of Enrichment policies here: Enrich processor: allow scheduling of policy executions · Issue #50071 · elastic/elasticsearch · GitHub

1 Like
3

There's one. Under index management IIRC.

That's what ES|QL LOOKUP JOIN does from 8.18.