Logstash misinterprets incoming data

[hellotty (Kirill)]

Hello.
I am streaming mdaemon logs using filebeat and using multiline.pattern to concatenate lines into one event. Using logstash, I create fields "mail from", "mail to", "message id", etc.
Is it possible to make grok understand the end of the line "\ n"?

Json sample from kibana:

"Tue 2021-09-14 02:33:15.396: 01: [26785584] REMOTE message: pd9001000177274.msg\nTue 2021-09-14 02:33:15.396: 01: [26785584] * Session 26785584; child 0002\nTue From: email@email.ru\n"

This would make it easier for me to write groks, since they would not be so "specific".

[Badger]

You can anchor each pattern, or match everything up to a newline.

[mutt13y (Stuart Gall)]

\n should match the end of line assuming unix line ends ([\n\r]+) is more generic

Note you can put (?m) at the start of the grok to make . and \s match the new line this is way more efficient if you just want to match the whole thing rather than using \n as some sort of delimiter

Also note you can have multiple matches in a grok so you can do

match => {message => [
"(?-m)From: %{FROM:DATA}",
"(?-m}To: %{TO:DATA}",
]}

[hellotty (Kirill)]

Should this work?

grok {
match => { "message" => [
"(?-m)From: %{GREEDYDATA:FROM}",
"(?-m}To: %{GREEDYDATA:TO}"]
}

I want to give an example again

from this:

Tue 2021-09-28 17:55:32.120: 01: [28050198] REMOTE message: pd3501003456824.msg\nTue 2021-09-28 17:55:32.121: 01: [28050198] * Session 28050198; child 0001\nTue 2021-09-28 17:55:32.121: 01: [28050198] * From: mailfrom@mail.com\nTue 2021-09-28 17:55:32.121: 01: [28050198] * To: mailto@mail.com\nTue 2021-09-28 17:55:32.121: 01: [28050198] * Subject: ��������� �������\nTue 2021-09-28 17:55:32.121: 01: [28050198] * Message-ID: 1355832302.20210928175526@mail.com\nTue 2021-09-28 17:55:32.121: 01: [28050198] * Size: 1259099; <e:\mdaemon\queues\remote\pd3501003456824.msg>\nTue 2021-09-28 17:55:32.129: 05: [28050198] Resolvi/>

I want to get it:

[hellotty (Kirill)]

Thanks for the tip! As a result, I use the following configuration:

my custom grok

NEXT_LINE [^\\\n]*

logstash config file

if "mdaemon_smtp" in [tags] {
grok {
patterns_dir => ["/etc/logstash/patterns"]
break_on_match => false
match => {
"message" => [
"From: %{NEXT_LINE:mail_from}",
"To: %{NEXT_LINE:mail_to}",
"Message-ID: <%{NEXT_LINE:message_id}>"]}
}
}