Merge two log file using Elasticsearch script

Elastic Stack Elasticsearch
1

Hello Team,

Please find below scenarios.

File 1:
Session_ID : S001
From_City_Name : TYO
To_City_Name : LAX

File 2:
Session_ID : S001
Product_Id : P001

I want to merge these two log file based on the session ID. Is that possible?

Script will be : Select From_City_Name, To_City_Name, Product_Id where Session_ID = Session_ID;

Result raw will be : S001, P001, TYO, LAX

Regards,
Binuraj

2

This is something you need to do before indexing into elasticsearch so you'd better ask on the logstash forum.

I'm moving your question there.

3

There's nothing in stock Logstash to merge multiple input files in the way you describe. What might work is reading them independently and updating the ES index twice. You'd have to pick a well-defined document id (perhaps the Session_ID field?). The first time a given session id is seen it'll create the document and the next time it'll update it with the additional fields. However, I don't think Logstash's elasticsearch output does partial document updates so you'd have to use an elasticsearch filter to fetch the missing fields. Yuck. I think this is something you'll want to do outside of Logstash.

4

Hello Team,

Please find below scenarios.

File 1:
Session_ID : S001
From_City_Name : TYO
To_City_Name : LAX

File 2:
Session_ID : S001
Product_Id : P001

I want to merge these two log file based on the session ID. Is that possible?

Script will be : Select From_City_Name, To_City_Name, Product_Id where Session_ID = Session_ID;

Result raw will be : S001, P001, TYO, LAX

Regards,
Binuraj

5

This seems to be exactly the same question that has already been answered. What is it that is not clear?

6

I'm checking elasticsearch support this kind of scripting or not?

7

I'm checking elasticsearch support this kind of scripting or not?

And I responded to your question. If some part of the answer is unclear, please ask a specific question about that. Don't post the same question all over again.