Combining Data in Multiple CSVs to Single Index

[rukmalf (Rukmal Fernando)]

Hi,

I have 3 CSVs which I use to create 3 indexes. But the problem is the 1st and 3rd Indexes are connected via the 2nd Index (refer to the image)

End of the day I need to get a search result like A1, A2, A3,B1,B2,B3. This would have been easily done using normal DBs. But with ES it seems to be impossible. Hence I have been advised to create a single index with the 3 CSVs

Can this be done using logstash? if so how? If not what are my options?

BR
Rukmal

[Rahul_Kumar4 (Rahul Kumar)]

@rukmalf

Hence I have been advised to create a single index with the 3 CSVs

How is that going to solve your problem of getting a output schema like you want A1,A2,A3,B1,B2,B3?

One alternate way to try this is to follow the below steps:

Step 1: Index 3 CSVs each into an index of its own using Logstash. Lets call them index 1 (A1,A2,A3) , index 2 (A1,B1) and index 3 (B1, B2, B3).

Step 2: Reingest the docs in index 1 in a new Logstash pipeline by using the Elasticsearch input plugin and in the filter section of that pipeline use two consecutive Elasticsearch filter plugins to find the matching docs in index 2 and index 3 and pull the fields you want using the fields parameter.