# Issue with Parsing multiline log in together with filebeat multiline config and logstash
**URL:** https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566
**Category:** Logstash
**Created:** [May 27, 2020, 2:46pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566 "2020-05-27T14:46:45Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 2:46pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/1 "2020-05-27T14:46:45Z")
I have the below multiline log:
```auto
2020-05-27 11:59:17 ----------------------------------------------------------------------
2020-05-27 11:59:17 Got context
2020-05-27 11:59:17 Raw context:
EMAIL=abc.def@example.com
NAME=abc.def
PAGER=+11111111111111
DATE=2020-05-27
AUTHOR=
COMMENT=
ADDRESS=1.1.1.1
ALIAS=abc.example.com
ATTEMPT=1
2020-05-27 11:59:17 Previous service hard state not known. Allowing all states.
2020-05-27 11:59:17 Computed variables:
URL=abc.example.com
STATE=UP
2020-05-27 11:59:17 Preparing flexible notifications for abc.def
2020-05-27 11:59:17 channel with plugin sms
2020-05-27 11:59:17 - Skipping: set
2020-05-27 11:59:17 channel with plugin plain email
2020-05-27 11:59:20 --------------------------------------------------------------------
```
This is my logstash config:
```auto
input {
stdin { }
}
filter {
grok {
match => { "message" => "(?m)%{GREEDYDATA:data}"}
}
if [data] {
mutate {
gsub => [
"data", "^\s*", ""
]
}
mutate {
gsub => ['data', "\n", " "]
}
}
}
output {
stdout { codec => rubydebug }
}
```
And Filebeat config
```auto
multiline.pattern: '^[[:space:]][A-Za-z]* (?m)'
multiline.negate: false
multiline.match: after
```
What I need is -
After the multiline log has been processed by filebeat, the log will get split into multiple lines. One such line is
```auto
2020-05-27 11:59:17 Raw notification context:
EMAIL=abc.def@example.com
NAME=abc.def
PAGER=+11111111111111
DATE=2020-05-27
AUTHOR=
COMMENT=
ADDRESS=1.1.1.1
ALIAS=abc.example.com
ATTEMPT=1
```
Now, I want to extract each key value pair using the kv filter and it should show as like one single message like the above but then also have query\_string\_key=value assigned to it.
How to achieve this ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 3:21pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/2 "2020-05-27T15:21:34Z")
I do not know filebeat well, but that regexp for the multiline looks wrong to me.
Anyways, assuming filebeat delivers a [message] like
```
"2020-05-27 11:59:17 Raw context:\n EMAIL=abc.def@example.com\n NAME=abc.def\n PAGER=+11111111111111\n DATE=2020-05-27\n AUTHOR=\n COMMENT=\n ADDRESS=1.1.1.1\n ALIAS=abc.example.com\n ATTEMPT=1"
```
Your filters will reduce that to
```
"2020-05-27 11:59:17 Raw context: EMAIL=abc.def@example.com NAME=abc.def PAGER=+11111111111111 DATE=2020-05-27 AUTHOR= COMMENT= ADDRESS=1.1.1.1 ALIAS=abc.example.com ATTEMPT=1"
```
Using grok to copy the message field to another field is expensive. It would be cheaper to do
```
mutate { copy => { "message" => "data"}
```
Also, the two mutate filters can be combined
```
gsub => [
"data", "^\s*", "",
'data', "\n", " "
]
```
If you just add
```
kv { source => "data" }
```
then you will get additional fields
```
"DATE" => "2020-05-27",
"EMAIL" => "abc.def@example.com",
"ATTEMPT" => "1",
"ALIAS" => "abc.example.com",
"PAGER" => "+11111111111111",
"NAME" => "abc.def",
"AUTHOR" => "COMMENT=",
"ADDRESS" => "1.1.1.1"
```
Note that two consecutive keys that do not have values are not well handled.
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 4:24pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/3 "2020-05-27T16:24:10Z")
Thanks a lot for your response. I am a newbie in ELK.
In Kibana, using the logstash and filebeat config that I posted, I get 10 hits(results) with the data field like below:
```
t data 2020-05-27 18:06:20 --------------------------------------------------------------------
t data 2020-05-27 18:06:17 Previous service hard state not known. Allowing all states.
t data 2020-05-27 18:06:17 Computed variables:
URL=abc.example.com
STATE=UP
t data 2020-05-27 18:06:17 Preparing flexible notifications for abc.def
t data 2020-05-27 18:06:17 channel with plugin sms
t data 2020-05-27 18:06:17 - Skipping: set
t data 2020-05-27 18:06:17 channel with plugin plain email
t data 2020-05-27 18:06:17 ----------------------------------------------------------------------
t data 2020-05-27 18:06:17 Got context
t data 2020-05-27 18:06:17 Raw context:
EMAIL=abc.def@example.com
NAME=abc.def
PAGER=+11111111111111
DATE=2020-05-27
AUTHOR=
COMMENT=
ADDRESS=1.1.1.1
ALIAS=abc.example.com
ATTEMPT=1
```
With the Logstash filter config, that you have suggested -
```auto
filter {
mutate {
copy => { "message" => "data"}
gsub => [
"data", "^\s*", "",
"data", "\n", " "
]
}
kv { source => "data" }
}
```
I have not tried it yet, but I would like to know if our below assumption of the message that filebeat delivers is correct or not based on the above output in Kibana. How can I check that ?
```auto
"2020-05-27 11:59:17 Raw context:\n EMAIL=abc.def@example.com\n NAME=abc.def\n PAGER=+11111111111111\n DATE=2020-05-27\n AUTHOR=\n COMMENT=\n ADDRESS=1.1.1.1\n ALIAS=abc.example.com\n ATTEMPT=1"
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 4:40pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/4 "2020-05-27T16:40:42Z")
Look at the message and data fields in kibana (expand an event in the discover tab and look at the JSON tab).
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 4:56pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/5 "2020-05-27T16:56:08Z")
Thanks a lot for this tip about expanding the JSON. I tried the grok filter and I get the results perfectly fine.
Is there a way to ignore the below data as it does not contain useful key value pairs when processed by the KV filter so that this does not appear in Kibana:
```
t data 2020-05-27 18:06:20 --------------------------------------------------------------------
t data 2020-05-27 18:06:17 Previous service hard state not known. Allowing all states.
t data 2020-05-27 18:06:17 Preparing flexible notifications for abc.def
t data 2020-05-27 18:06:17 channel with plugin sms
t data 2020-05-27 18:06:17 - Skipping: set
t data 2020-05-27 18:06:17 channel with plugin plain email
t data 2020-05-27 18:06:17 ----------------------------------------------------------------------
t data 2020-05-27 18:06:17 Got context
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 6:29pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/6 "2020-05-27T18:29:32Z")
Those lines are single lines, not multiple lines combined. The multiline codec adds a tag (set by the multiline\_tag option) to events only if they are a combination of multiple lines. So if you want those lines to disappear you could use
```
if "multiline" not in [tags] { drop {} }
```
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 6:41pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/7 "2020-05-27T18:41:23Z")
Thanks a lot. All works. Regarding one of the values that I get for the key.
For example, this is the Filebeat message:
```auto
ADDRESS=1.1.1.1\n NAME=abc.def\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\n ATTEMPT=3\n
```
For the key STATE, I only see "wan" getting displayed and rest all the values are ignored. How to get all the values for this key as it is ?Do I have to split here?
I tried like this but no luck:
```auto
kv {
source => "data"
field_split => "\n\s*"
value_split => "="
}
```
Am i missing something ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 7:06pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/8 "2020-05-27T19:06:56Z")
Change your gsub to put double quotes around the value for each key...
```
gsub => [
"data", "^\s*", "",
"data", "(=)([^\n]*)\n", '\1"\2" ',
'data', "\n", " "
]
```
The third is still required to handle the junk at the start of the data field. This improves the result because the kv filter ignores fields with empty values, so
```
AUTHOR=\"\" COMMENT=\"\"
```
no longer results in that bogus AUTHOR field on the event.
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 7:25pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/9 "2020-05-27T19:25:06Z")
Sorry, but the value of the key STATE is still "wan"
```auto
{
"host" => "pqr",
"AUTHOR" => "\\n",
"STATE" => "wan",
"ALIAS" => "abc.example.com\\n",
"ATTEMPT" => "1",
"PAGER" => "+11111111111111\\n",
"data" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n NAME=abc.def\\n PAGER=+11111111111111\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ADDRESS=1.1.1.1\\n ALIAS=abc.example.com\\n ATTEMPT=1",
"EMAIL" => "abc.def@example.com\\n",
"COMMENT" => "\\n",
"@timestamp" => 2020-05-27T19:18:57.370Z,
"message" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n NAME=abc.def\\n PAGER=+11111111111111\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ADDRESS=1.1.1.1\\n ALIAS=abc.example.com\\n ATTEMPT=1",
"DATE" => "2020-05-27\\n",
"NAME" => "abc.def\\n",
"ADDRESS" => "1.1.1.1\\n",
"@version" => "1"
}
```
This is the updated filter:
```
filter {
mutate {
copy => { "message" => "data"}
gsub => [
"data", "\s*", "",
"data", "(=)([^\n]*)\n", '\1"\2" ',
'data', "\n", " "
]
}
kv {
source => "data"
field_split => "\n\s*"
value_split => "="
}
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 7:44pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/10 "2020-05-27T19:44:14Z")
Change the kv back to
```
kv { source => "data" }
```
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 7:48pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/11 "2020-05-27T19:48:26Z")
Still it remains the same.
```auto
{
"COMMENT" => "\\n",
"@version" => "1",
"host" => "pqr",
"ALIAS" => "abc.example.com\\n",
"NAME" => "abc.def\\n",
"@timestamp" => 2020-05-27T19:45:50.644Z,
"STATE" => "wan",
"AUTHOR" => "\\n",
"EMAIL" => "abc.def@example.com\\n",
"ATTEMPT" => "1",
"ADDRESS" => "1.1.1.1\\n",
"message" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n NAME=abc.def\\n
PAGER=+11111111111111\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots
/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ADDRESS=1.1.1.1\\n
ALIAS=abc.example.com\\n ATTEMPT=1",
"DATE" => "2020-05-27\\n",
"data" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n NAME=abc.def\\n
PAGER=+11111111111111\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots
/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ADDRESS=1.1.1.1\\n
ALIAS=abc.example.com\\n ATTEMPT=1",
"PAGER" => "+11111111111111\\n"
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 7:53pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/12 "2020-05-27T19:53:09Z")
> [@marco2005](#):
>
> `Raw context:\\n `
Your message contains \n in several places. That not a newline, it is the character \ followed by the character n. Changing \n in the gsubs to \\n might help.
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 7:58pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/13 "2020-05-27T19:58:16Z")
```
filter {
mutate {
copy => { "message" => "data"}
gsub => [
"data", "\s*", "",
"data", "(=)([^\\n]*)\\n", '\1"\2" ',
'data', "\\n", " "
]
}
kv {
source => "data"
}
}
```
Did this change in the filter without any luck .
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 8:09pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/14 "2020-05-27T20:09:48Z")
> [@marco2005](#):
>
> `"(=)([^\\n]*)\\n"`
Oh, that's not going to work. The intent of the pattern was "characters that are not newline followed by newline". For a pair of characters you will have to change it to a lookahead assertion. I cannot test it right now, but maybe
```
"data", "(=)(.*)(?=\\n)", '\1"\2" ',
```
Hopefully the .\* will not get greedy.
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 8:20pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/15 "2020-05-27T20:20:33Z")
Sorry. It doesn't work.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 8:54pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/16 "2020-05-27T20:54:30Z")
It was not working because for some fields the .\* was getting greedy. How about this?
```
gsub => [
"data", "(=)([^=]*)(?=\\n)", '\1"\2"',
"data", "\\n\s*", " "
]
```
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 8:58pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/17 "2020-05-27T20:58:36Z")
I used this now and the issue remains the same:
```
filter {
mutate {
copy => { "message" => "data"}
gsub => [
"data", "(=)([^=]*)(?=\\n)", '\1"\2"',
"data", "\\n\s*", " "
]
}
kv {
source => "data"
}
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 9:02pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/18 "2020-05-27T21:02:19Z")
The problem I have is that the rubydebug [message] field you show appears to have both \\n and unescaped newlines. I have no idea how rubydebug could possibly display that, so I do not know the value of your [message] field. I cannot help.
---
### Author:  [@marco2005](https://discuss.elastic.co/u/marco2005)
#### Post date: [May 27, 2020, 9:04pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/19 "2020-05-27T21:04:58Z")
Are you referring to this?
```auto
"message" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n NAME=abc.def\\n PAGER=+11111111111111\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ADDRESS=1.1.1.1\\n ALIAS=abc.example.com\\n ATTEMPT=1",
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 27, 2020, 9:22pm UTC](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566/20 "2020-05-27T21:22:20Z")
No, if you go 8 posts back from your post you will see a post that contains the rubydebug of a complete event. That has a newline embedded just before PAGER and others further on.
I am running out of ideas. This configuration
```
input { generator { count => 1 lines => ["2020-05-27 11:59:17 Raw context:\n EMAIL=abc.def@example.com\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\n DATE=2020-05-27\n AUTHOR=\n COMMENT=\n ATTEMPT=1"] } }
filter {
mutate { copy => { "message" => "data"}
}
if [data] {
mutate {
gsub => [
"data", "(=)([^=]*)(?=\\n)", '\1"\2"',
"data", "\\n\s*", " "
]
}
kv { source => "data" }
}
}
output { stdout { codec => rubydebug { metadata => false } } }
```
produces
```
"message" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ATTEMPT=1",
"data" => "2020-05-27 11:59:17 Raw context: EMAIL=\"abc.def@example.com\" STATE=\"wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\" DATE=\"2020-05-27\" AUTHOR=\"\" COMMENT=\"\" ATTEMPT=1",
"STATE" => "wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/",
"DATE" => "2020-05-27",
"EMAIL" => "abc.def@example.com",
"ATTEMPT" => "1"
```
I had to remove a couple of fields from the original message for that example because otherwise the browser introduces the same unescaped newlines that were in your post. However, even if I use the [message] value you gave then it works...
```
"message" => "2020-05-27 11:59:17 Raw context:\\n EMAIL=abc.def@example.com\\n NAME=abc.def\\n PAGER=+11111111111111\\n STATE=wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\\n DATE=2020-05-27\\n AUTHOR=\\n COMMENT=\\n ADDRESS=1.1.1.1\\n ALIAS=abc.example.com\\n ATTEMPT=1",
"data" => "2020-05-27 11:59:17 Raw context: EMAIL=\"abc.def@example.com\" NAME=\"abc.def\" PAGER=\"+11111111111111\" STATE=\"wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/\" DATE=\"2020-05-27\" AUTHOR=\"\" COMMENT=\"\" ADDRESS=\"1.1.1.1\" ALIAS=\"abc.example.com\" ATTEMPT=1",
"ADDRESS" => "1.1.1.1",
"ALIAS" => "abc.example.com",
"ATTEMPT" => "1",
"PAGER" => "+11111111111111",
"NAME" => "abc.def",
"STATE" => "wan abc site:def cake-agent jlp bl3 mississipi algebra dev app abc1 nano /nano/ack/divgen/mississipi/bots/linux/",
"DATE" => "2020-05-27",
"EMAIL" => "abc.def@example.com"
```
[Next page](https://discuss.elastic.co/t/issue-with-parsing-multiline-log-in-together-with-filebeat-multiline-config-and-logstash/234566.md?page=2)