# Problem with parsing multiline filter plugin
**URL:** https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906
**Category:** Logstash
**Created:** [June 14, 2018, 11:26am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906 "2018-06-14T11:26:42Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 14, 2018, 11:26am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/1 "2018-06-14T11:26:42Z")
Hi
i have to parse netstat ("Network Statistics") output using logstash
sample entries will be like bellow
> TIME: 16:14:30.14
>
> Active Connections
>
> Proto Local Address Foreign Address State
> TCP 0.0.0.0:66 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:5357 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:9937 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:12372 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:49664 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:49665 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:49666 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:49667 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:49668 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:49679 0.0.0.0:0 LISTENING
here i have to use **TIME: 16:14:30.14** for all lines and push it into db, so i used 'multiline' as bellow
> filter
> {
> multiline {
> pattern =\> "^%{WORD:}: %{TIME}"
> negate =\> true
> what =\> "previous"
> }
> grok {
>
> match =\>
> {
> message =\> '^%{WORD:}: %{TIME:time} %{GREEDYDATA} %{WORD:protocol}%{SPACE}%{IPORHOST:localaddress}:%{POSINT:port}%{SPACE}%{IPORHOST:ForeignAddress}:%{NUMBER:port2}%{SPACE}%{WORD:state}'
> }
> }
>
> }
Sadly the results from the above configuration are unexpected , after matching time it is taking all lines as single event , but i want to parse each line separately and push it into ES as individual doc's
please help me
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 14, 2018, 11:29am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/2 "2018-06-14T11:29:56Z")
The multiline filter has been deprecated, so you should not use it. You should always look to perform multiline processing as close to the source as possible. If you are using Filebeat, then that is where it should be done. If you need to do it in Logstash, you should instead use the multiline codec.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 14, 2018, 11:31am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/3 "2018-06-14T11:31:55Z")
But the functionalities are working fine, its an issue with
> pattern, negate,what
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 14, 2018, 11:38am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/4 "2018-06-14T11:38:51Z")
Multiline just groups lines, it does not take a line and add it to others. For this you probably need to group it all into one event, extract the timestamp and other data from the initial line before splitting up the event using a split or ruby filter.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 14, 2018, 11:44am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/5 "2018-06-14T11:44:25Z")
Could you please explain in-depth , i have not worked on ruby-filter , please give an example of how to group all data into single event and break them using ruby
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 14, 2018, 11:48am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/6 "2018-06-14T11:48:57Z")
Can you show what your expect the resulting events to look like?
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 14, 2018, 11:54am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/7 "2018-06-14T11:54:25Z")
input is as bellow..
> TIME: 16:14:30.14
>
> Active Connections
>
> Proto Local Address Foreign Address State
> TCP 0.0.0.0:80 0.0.0.0:0 LISTENING
> TCP 0.0.0.0:2 0.0.0.0:0 ESTABLISHED
>
> ..................................................................................................................
> TIME: 16:14:40.13
>
> Active Connections
>
> Proto Local Address Foreign Address State
> TCP 0.0.0.0:3 0.0.0.0:0 LISTENING
i want output as to be
> {
>
> time : 16:14:30.14
> Protocol : TCP
> localAdd : 0.0.0.0:80
> Foreign Address : 0.0.0.0:0
> state : LISTENING
> }
>
> {
> time : 16:14:30.14
> Protocol : TCP
> localAdd : 0.0.0.0:3
> Foreign Address : 0.0.0.0:0
> state : ESTABLISHED
> }
> {
> time :16:14:40.13
> Protocol : TCP
> localAdd : 0.0.0.0:2
> Foreign Address : 0.0.0.0:0
> state : LISTENING
> }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 14, 2018, 12:15pm UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/8 "2018-06-14T12:15:47Z")
Try something like this:
```auto
filter {
dissect {
mapping => {
"message" => "TIME: %{time}
%{}"
}
}
split {}
if [message] =~ /^(TCP|UDP)/ {
dissect {
mapping => {
"message" => "%{protocol} %{local_address} %{foreign_address} %{state}"
}
}
} else {
drop {}
}
}
```
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 14, 2018, 12:38pm UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/9 "2018-06-14T12:38:43Z")
Sorry!! i didn't understand the code , i tried to run it but it is giving some waring messages :
> [2018-06-14T18:06:43,046][WARN][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=\>"message", "event"=\>{"@version"=\>"1", "tags"=\>["\_dissectfailure"],
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 14, 2018, 1:18pm UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/10 "2018-06-14T13:18:51Z")
The first dissect pattern just parses out the time field. You can replace this with grok if you want.
The second filter splits the event by line, and all events keep the time filter that was previously parsed out.
The we drop all lines that does not start with TCP or UDP and parse the remaining lines using a dissect filter.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 14, 2018, 1:28pm UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/11 "2018-06-14T13:28:05Z")
your awesome ....you made my day ....
its working great ... thanks a lot !!!
but one small issue is in **time** nothing is coming it coming empty
{
"protocol" =\> "TCP",
"foreign\_address" =\> "0.0.0.0:0",
"time" =\> "",
"@timestamp" =\> 2018-06-14T13:31:16.898Z,
"local\_address" =\> "0.0.0.0:2",
"message" =\> "TCP 0.0.0.0:2 0.0.0.0:0 ESTABLISHED",
"state" =\> "ESTABLISHED",
"path" =\> "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/Logs/try.txt",
"host" =\> "avk03-Vostro-3800",
"@version" =\> "1"
}
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 15, 2018, 9:41am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/12 "2018-06-15T09:41:42Z")
Hi @Christian_Dahlqvist thanks for your help.
I am facing some spacing issues while using **dissect filter plugin**
when my log entry is as bellow
> TCP 0.0.0.0:66 0.0.0.0:0 LISTENING (without space in the beginning of line)
parser what you suggested me is working fine..
But when lines having spaces as bellow:
> TCP 0.0.0.0:66 0.0.0.0:0 LISTENING (with space in the beginning of line)
parser is throwing some warning messages like:
> [2018-06-15T15:01:31,539][WARN][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=\>"message", "event"
Could you please help me how to match "_white spaces_" when we are using **dissect filter plugin**
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 15, 2018, 10:05am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/13 "2018-06-15T10:05:27Z")
OK. Then try to change the latter part to something like this:
```auto
if [message] =~ /^\s+(TCP|UDP)/ {
dissect {
mapping => {
"message" => "%{?tmp->} %{Protocol->} %{localAdd->} %{Foreign Address->} %{state}"
}
}
} else {
drop {}
}
```
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 15, 2018, 10:12am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/14 "2018-06-15T10:12:41Z")
No, ☹ its giving wrong output
> {
> "Protocol" =\> "TCP",
> "localAdd" =\> "",
> "time" =\> "",
> "path" =\> "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/Logs/try.txt",
> "state" =\> " 0.0.0.0:445 0.0.0.0:0 LISTENING",
> "@timestamp" =\> 2018-06-15T10:08:19.779Z,
> "@version" =\> "1",
> "message" =\> " TCP 0.0.0.0:445 0.0.0.0:0 LISTENING",
> "host" =\> "avk03-Vostro-3800",
> "Foreign Address" =\> ""
> }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 15, 2018, 10:16am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/15 "2018-06-15T10:16:48Z")
I updated it as I noticed you have variable number of spaces in between fields as well. If it is still not working, read through the dissect documentation and go through the config step by step as described in [this blog post](https://www.elastic.co/blog/a-practical-introduction-to-logstash).
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 15, 2018, 10:20am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/16 "2018-06-15T10:20:06Z")
Thanks a lot... it is working as i expected 🙂
> {
> "message" =\> " TCP 0.0.0.0:445 0.0.0.0:0 LISTENING",
> "host" =\> "avk03-Vostro-3800",
> "Protocol" =\> "TCP",
> "Foreign Address" =\> "0.0.0.0:0",
> "state" =\> "LISTENING",
> "path" =\> "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/Logs/try.txt",
> "@version" =\> "1",
> "@timestamp" =\> 2018-06-15T10:18:52.702Z,
> "time" =\> "",
> "localAdd" =\> "0.0.0.0:445"
> }
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [June 15, 2018, 11:21am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/17 "2018-06-15T11:21:52Z")
Hi @Christian_Dahlqvist
Actually this parser is not reading lines Which contains **TIME** data
> {
> "state" =\> "LISTENING",
> "time" =\> "",
> "@version" =\> "1",
> "message" =\> " TCP 100.113.26.16:139 0.0.0.0:0 LISTENING",
> "Foreign Address" =\> "0.0.0.0:0",
> "host" =\> "avk03-Vostro-3800",
> "@timestamp" =\> 2018-06-15T11:17:49.699Z,
> "path" =\> "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/Logs/try.txt",
> "Protocol" =\> "TCP",
> "localAdd" =\> "100.113.26.16:139"
> }
> {
> "state" =\> "ESTABLISHED",
> "time" =\> "",
> "@version" =\> "1",
> "message" =\> " TCP 100.113.26.16:54120 111.221.29.101:443 ESTABLISHED",
> "Foreign Address" =\> "111.221.29.101:443",
> "host" =\> "avk03-Vostro-3800",
> "@timestamp" =\> 2018-06-15T11:17:49.701Z,
> "path" =\> "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/Logs/try.txt",
> "Protocol" =\> "TCP",
> "localAdd" =\> "100.113.26.16:54120"
> }
I have tried many ways went through documentation, but no luck in **message** only it is reading TCP|UDP lines , i am not getting why first part of dissect is not functioning ...!!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 13, 2018, 11:21am UTC](https://discuss.elastic.co/t/problem-with-parsing-multiline-filter-plugin/135906/18 "2018-07-13T11:21:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.