Index relationship between distinct csv files

Elastic Stack Elasticsearch
1

Hi all,

I need to index parent-child documents coming from distinct csv files generated several times a day.

One application generates the "parent" csv files and 2 other applications generate the "children" csv files.
The relation between parent and children is made using functionnal fields in the csv files.

Today, each file is indexed with its own type using filebeat + logstash and all the parent/child relationship is handle by code in the search application I am working on.

But I think it would be better to handle the relationships in ES using parent/child or nesting objects.

What is the best way to achieve this purpose ? Is it possible using filebeat+logstash or should I develop a custom application to manage indexing ?

Thank you.

Olivier

2

You need to join the data in flight (which is hard), or do a two phase process of processing the parent with one pipeline and then another for the child of that parent.

3

Thank you for your answer. But can you explain what you mean by a two phases process ?
Do you mean having one logstash pipeline for parent documents indexing and another one for children documents ?

4

Yes.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.