Issue in Controls

moep · January 23, 2023, 3:52pm

My main usecase is, that I can search/visualize the mailflow.
For examples:

which $custumer was writing an e-mail and who was answering.
responses from e-mailservers, with error codes (e.g. mailbox is full)

What I want to see in Kibana:

I would like to see a table and see the recipient and sender, but in my logs you will never see a recipient and a sender in one logline. Over the message id its possible to see, who is the sender/recipient.

stephenb · January 24, 2023, 5:13pm

Ahhhh we finally get to the most crucial fact... the sender and recipient are not in the same log lines/documents but linked by a message ID.... perhaps I Missed that earlier, but I don't think so.

In the future, sample data right at the beginning would help... as we do not understand your context.

So this is NOT straightforward to solve in Elastic... Let me take a look and get back, there are some techniques but I need to look a bit...

Pretty sure we can build a table but let me create some sample data from yours and try...

Also, I do not know what significance the "exim_flags": "<=", if any so I am not going to focus on that.

moep · January 24, 2023, 5:23pm

from the Exim documentation.

5. Log line flags
One line is written to the main log for each message received, and for each successful, unsuccessful, and delayed delivery. These lines can readily be picked out by the distinctive two-character flags that immediately follow the timestamp. The flags are:

    <=	message arrival
    (=	message fakereject
    =>	normal message delivery
    ->	additional address in same delivery
    >>	cutthrough message delivery
    *>	delivery suppressed by -N
    **	delivery failed; address bounced
    ==	delivery deferred; temporary problem

I use it as a kind of pre-processing. If I don't do this, I can't match if an email in the log was the sender or the recipient. I found this old gist on github, so it was'nt nessecary, to start by zero.

stephenb · January 24, 2023, 5:31pm

Ok... I see that helps a bit... what do you want to do with that in the table.

Funny most of us would use a clear identifier like the following which I think is a much better plan for elasticsearch and clarity instead of some mathematical symbols...

Symbol	Meaning	Code
<=	message arrival	MESSAGE_ARRIVAL
(=	message fakereject	MESSAGE_FAKE_REJECT
=>	normal message delivery
->	additional address in same delivery
>>	cutthrough message delivery
*>	delivery suppressed by -N
**	delivery failed; address bounced
==	delivery deferred; temporary problem

Also, keep in mind we are elasticsearch focused so we don't know every system out there ... more context you provide the better help we can provide...

As I like to say "Help Us, Help You"

I will take a look at the table later today see what I can do...

stephenb · January 25, 2023, 2:47am

Hi @moep

Ok so here is a potential solution... see if you can follow along... this is not perfect but perhaps a start.

Note this is a possible solution not saying it is perfect....

Sample Data

DELETE discuss-test

GET discuss-test

PUT discuss-test
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings": {
          "match_mapping_type": "string",
          "match":   "exim*",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}



POST discuss-test/_doc
{
  "exim_year": "2022",
  "exim_month": "12",
  "exim_day": "26",
  "exim_time": "08:15:43",
  "@timestamp": "2023-01-23T18:00:59.389Z",
  "exim_flags": "**",
  "exim_sender": [
	"jane.doe@elasticsearch.com",
	"jane.doe@elasticsearch.com"],
	"exim_msg_id": "1pIlON-0006A7-iA"
}

POST discuss-test/_doc
{
  "exim_time": "19:00:52",
  "@timestamp": "2023-01-23T18:01:59.389Z",
  "exim_day": "22",
  "exim_flags": "==",
  "exim_month": "01",
  "exim_recipient": "input@mail.net",
  "exim_msg_id": "1pIlON-0006A7-iA"
}

POST discuss-test/_doc
{
"exim_year": "2022",
"exim_month": "12",
"exim_day": "26",
"exim_time": "08:15:43",
"@timestamp": "2023-01-23T18:05:59.389Z",
"exim_flags": "**",
"exim_sender": "mr.doe@elasticsearch.com",
"exim_msg_id": "1pJdTc-0001AR-Ma"
}

POST discuss-test/_doc
{
  "exim_time": "19:00:52",
	"@timestamp": "2023-01-23T18:06:59.389Z",
  "exim_day": "22",
  "exim_flags": "==",
  "exim_month": "01",
  "exim_recipient": "recepient@mail.net",
  "exim_msg_id": "1pJdTc-0001AR-Ma"
}


POST discuss-test/_doc
{
	"exim_year": "2022",
	"exim_month": "12",
	"exim_day": "26",
	"exim_time": "08:15:43",
	"@timestamp": "2023-01-23T18:08:59.389Z",
	"exim_flags": "**",
	"exim_sender": ["another.doe@elasticsearch.com","another1.doe@elasticsearch.com"],
	"exim_msg_id": "1pJdTc-0001AR-ZZ"
}

Then I created a Data View....

Then I created a Lens -> Table

Overview

Row -> Time

Max Time

Last Recipient

Last Sender

moep · January 30, 2023, 3:04pm

awesome!
Should I open a new topic? You improved alot, but I can't search in Controls. I just get a list of all recipient/sender like:

jane.doe@elasticsearch.com
jon.doe@elasticsearch.com

I assume, that when type into the search doe I get two results. But I don't get anyone. I just get a result when I type the full mail.

stephenb · January 30, 2023, 3:15pm

I don't think what you are trying to do in the control will ever work the way you want unless you create an index that has both the sender and receiver in a single document... The control works of a single field so at best you could chose to filter on either the sender or receiver In 2 separate controls one sender, one receiver.

You will never get a control that magically matches them unless you create an index that has both sender and receiver in the same document.... Which is possible but that is a whole other advanced conversation.

moep · February 2, 2023, 7:06pm

@stephenb right now, your answer is the best answer, but there is one thing, where Im not sure, how to solve it:

The filter only shows the last entry at exim_recipient, which is ok in ~90 % cases, but when I'm writing to an e-mail list, the MSGID is for all recipients is similar. I was thinking about Formulas, but not sure if they work in a table (yes I was reading the docs, but I didnt found an example for me case).

thx