Logstash document_id for 3 joined tables

Elastic Stack Logstash
1

Hi, I am new to ELK stack search. I have created an index by joining 3 tables - message, message_recipient and message_attachments - using inner join on message and message_recipient and left outer join on message_attachment since message may or may not have attachments. Now I want to get combined output of these joins in one index and also avoid duplicates.

elasticsearch {
  hosts => ..
  index => ..
  document_id => "%{message_id}%{recipient_id}%{attachment_id}"
}

With this document_id, only those messages will be ingested for which there is one or more attachments but there are recipients of messages for which there are no attachments, I want those records as well. How to ingest those records ?

2

You could try using a fingerprint filter, that will create a hash of whichever fields exist, and you can use that as the document id.

3

Thank you for your suggestion, I will check fingerprint filter.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.