Problem Statement

I have 3 different files ( 3 diff csv files) - transaction data, card info data and user data. I need to merge all these data into single file (I mean to perform SQL type join based on a key). I am aware that we can’t perform joins in elasticsearch.

So what approach to solve this problem. I need to join these documents and inject the data into single index on Kibana.

My options – let me know if this can be possible.

Use logstash- ETL tool, do data structuring and transformation there ( is it possible to do joins in logstash? )
Or use join queries and mapping documents and inner_hits concepts in Kibana console ?
Which of the above is feasible or if any new recommendations to solve this problem?

I'd use Logstash to do it, have a look at the translate filter for eg.

Thanks a ton. Can I know if any example solving related issue using translate filter?

There's examples in the documentation, but also lots on here if you do a search :slight_smile:

Writing a python script on the Elasticsearch end can be easier than merging on the Logstash end?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.