# Multiline filter/Codec question

**URL:** <https://discuss.elastic.co/t/multiline-filter-codec-question/81549>\
**Category:** Logstash\
**Created:** [April 7, 2017, 1:05am UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549 "2017-04-07T01:05:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxw2016](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@maxw2016](https://discuss.elastic.co/u/maxw2016)\
**Post date:** [April 7, 2017, 1:05am UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/1 "2017-04-07T01:05:28Z")

</div>

Hello,  
I'm working on exim email logs and I'm stuck with this codec.  
here is an example from the log:

2017-04-03 02:19:58 H=(localhost) [117.0.54.236] F=[test@earatt.net](mailto:test@earatt.net) rejected RCPT [smityrd@example.com](mailto:smityrd@example.com): Rejected message because 117.0.54.236 is in a black list at [huzkzg6n5flrulopcolvmnfhty.zen.dq.spamhaus.net](http://huzkzg6n5flrulopcolvmnfhty.zen.dq.spamhaus.net)  
2017-04-03 02:19:58 unexpected disconnection while reading SMTP command from (localhost) [117.0.54.236] (error: Connection reset by peer)  
2017-04-03 02:19:58 dovecot\_login authenticator failed for (ylmf-pc) [104.247.196.7]: 535 Incorrect authentication data ([set\_id=@example.com](mailto:set_id=@example.com))  
2017-04-03 02:19:58 no host name found for IP address 58.187.167.240  
2017-04-03 02:19:58 1cuy9W-000F1j-JS DKIM: [d=stratr.com](http://d=stratr.com) s=k1 c=relaxed/relaxed a=rsa-sha1 b=1024 [i=noreply@stratfor.com](mailto:i=noreply@stratfor.com) [verification succeeded]  
**2017-04-03 02:19:58 1cuy9W-000F1j-JS \<= [bounce-mc.us4\_7958185.306033-test=example.com@mail10.4.rsgsv.net](mailto:bounce-mc.us4_7958185.306033-test=example.com@mail10.4.rsgsv.net) [H=mail10.atl11.rsgsv.net](http://H=mail10.atl11.rsgsv.net) [205.201.133.10] P=esmtp S=21651 [id=7478.rsgsv.net](http://id=7478.rsgsv.net) T="How Japan Got Baseball"**  
**_2017-04-03 02:19:58 no host name found for IP address 46.29.251.135_**  
**2017-04-03 02:19:58 1cuy9W-000F1j-JS =\> test [test@example.com](mailto:test@example.com) R=mysql\_user T=mysql\_delivery**  
**2017-04-03 02:19:58 1cuy9W-000F1j-JS Completed**  
I was trying to use following logstash config:

filter {

if [type] == "eximlog" {

mutate {  
add\_field =\> {  
"message\_1" =\> "%{message}"  
}  
}  
multiline {  
patterns\_dir =\> "/etc/logstash/patterns/"  
pattern =\> "%{DATE} %{TIME} %{HOSTNAME:exim\_msg\_id} (=\>|Completed)"  
negate =\> false  
what =\> "previous"  
}

grok {  
patterns\_dir =\> "/etc/logstash/patterns/"

break\_on\_match =\> false

match =\> [  
"message\_1", "%{DATE} %{TIME} %{HOSTNAME:exim\_msg\_id} %{GREEDYDATA}"  
]

```
match => ['message', '%{EXIM_ALL_RULES}']

```

}

### remove fields

```
mutate {
  remove_field => ['host', 'offset']
}

```

# Remove the really, really dirty hack to workaround bug in grok code

# which won't handle multiple matches on the same field

mutate {  
remove\_field =\> ["message\_1"]  
}

}  
}

Each message might have at least 3 lines that only common value is date,message\_id.  
The problem I stumble on is that there are other log entries unrelated to that message but they still getting included. For example in message above there is additional entry " **_2017-04-03 02:19:58 no hostname found for IP address 46.29.251.135_**" between **2017-04-03 02:19:58 1cuy9W-000F1j-JS** related rows.

Please advise.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 7, 2017, 5:25am UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/2 "2017-04-07T05:25:42Z")

</div>

(The subject of your post talks about the multiline codec but you're using the deprecated multiline filter.)

You can't use multiline for this purpose for the reason you've just discovered. Have you looked at the aggregate filter?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 7, 2017, 5:26am UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/3 "2017-04-07T05:26:48Z")

</div>

It looks like you are using the multiline filter (which has been deprecated) rather than the multiline codec, which you configure as part of each input).

Where are you getting the logs? Which input plugins are you using? Which version of Logstash are you using?

---

<div class="post-metadata">

**Author:** ![maxw2016](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@maxw2016](https://discuss.elastic.co/u/maxw2016)\
**Post date:** [April 7, 2017, 6:55pm UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/5 "2017-04-07T18:55:08Z")

</div>

I'm sorry you rigth I had same issues when I tried codec. I will update subject.......  
I didn't look into aggregate filter yet thanks for an advice.

Here is my input section that I have tried before:  
input {  
beats {  
port =\> "6001"  
codec =\> multiline {  
pattern =\> "%{DATE} %{TIME} %{EXIM\_MSGID:exim\_msg\_id} (=\>|Completed)"  
negate =\> false  
what =\> previous  
}  
}  
}

and pattern for EXIM\_MSGID  
EXIM\_MSGID [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}

---

<div class="post-metadata">

**Author:** ![maxw2016](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@maxw2016](https://discuss.elastic.co/u/maxw2016)\
**Post date:** [April 7, 2017, 7:02pm UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/6 "2017-04-07T19:02:13Z")

</div>

I'm pushing multiple log types from email server to ELK server using file beat. (file beat version 5.2.2)  
All software is running on latest version: 5.3.0  
I'm using a filter because I couldn't figure out how to apply input filter to just that one log type.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 10, 2017, 5:26am UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/7 "2017-04-10T05:26:27Z")

</div>

Do not use the multiline codec together with the beats input. Join multiline events at the source, i.e. with Filebeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2017, 5:34am UTC](https://discuss.elastic.co/t/multiline-filter-codec-question/81549/8 "2017-05-08T05:34:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.