Using join field

Elastic Stack Logstash
1

Hello everyone!

I have problem joining 2 documents in one index by using join type field.
I have input, filter and output files configured for Logstash. I have index and index pattern set up in Elasticsearch. I was able to successfully retrieve the messages with out using join field. So I am having trouble only when I am trying to use this field. (I want to create a monitor with a query that requires joining 2 documents)

I receive 2 messages and they can be joined using one field. I use amazon_es plugin in the output file for Logstash configuration. I have 2 indexes (1 that is working with out join field and 2 that is supposed to have join field). I send document to 2 indexes (Tried to send them separately - it didn't help). In 1 index I receive the message but the problem that join field is actually text type (As I understood for join field to work I am supposed to create new index with join field for it to be join type). In 2 index I either receive only 1 out of 2 messages or no messages at all or my Logstash crashes in the infinite loop of retrying to send messages. I tried a lot of different approaches but nothing worked for me and I think I might be missing some basic information.

So what I did is:

  1. I created a new index using dev tool in Kibana
  2. I updated the filter with 2 lines for my documents. event.set("join_field", "s3event") for parent document and event.set("join_field", { "name" => "lambda", "parent" => parsed_message["requestPayload"]["Records"][0]["Sns"]["MessageId"] }). So there is a field for parent document and I use a field with the same information for the child document.
  3. And then I configure output for the correct index that I just created and send the message.

I tried so many approaches and can't think of what can be the issue, but the last thing I see that in Kibana join field actually has unknown type or maybe I set up the index incorrectly?

image
image1582×578 47.5 KB

Here is the gist with my filter configuration, output (with out data) and index configuration. Filter and output of Logstash · GitHub

Maybe you have some suggestions or links where I can get some information, because it's a little bit hard for me to obtain information for this join field topic...

I will be thankful for any help and tips!