Ingest Email Bounces

[xzelkworld (Xinhuan Zheng)]

We have Email bounce. They are in Email Mailbox format, but they are stored on Linux. Currently we have to set up complicate infrastructure to handle such Email Mailbox format ingestion processing. Is there a modern way to ingest Email Mailbox file to logstash then store them into Elasticsearch?

Example bounce back message:

    Return-Path: <someone@xyz.com>
    Received: from ..
    Received: from ..
    To: <anotherone@abc.com>
    Message-ID: <id>
    Date: Wed, 8 Jul 2020 09:45:00 -0700
    From: "XYZ" <someone@xyz.com>
    Subject: bounce
    Content-Language: en-us
    MIME-Version: 1.0
    Content-Transfer-Encoding: 8bit
    Content-Type: multipart/alternative;
            boundary="----=Part.1234567"

    ------=Part.1234567
    Content-Transfer-Encoding: 8bit
    Content-Type: text/plain; charset="UTF-8"
    ...

Xinhuan

[duc00 (Mathieu Défosse)]

Hello Xinhuan,

There would definitely be ways to parse and process these kind of messages. I suggest you looking at Logstash or the Elasticsearch Ingest. They both provide extensive and flexible capabilities to do that. Logstash would run out of the Elasticsearch cluster and would then send the output to it and Ingest would be part of the cluster (with possibility to define dedicated Ingest nodes).

I had to make the choice between the two of them and I chose the Ingest solution to keep everything in one single place. This article discussing both helped me.

Let me know if this answers your question,
duc00