# Multiline allow\_duplicates=\>true config eleminating duplicates

**URL:** <https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304>\
**Category:** Logstash\
**Created:** [May 30, 2016, 10:06am UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304 "2016-05-30T10:06:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Avishek](https://avatars.discourse-cdn.com/v4/letter/a/b77776/32.png) [@Avishek](https://discuss.elastic.co/u/Avishek)\
**Post date:** [May 30, 2016, 10:06am UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304/1 "2016-05-30T10:06:54Z")

</div>

Hello,

```
    I am using logstash 2.3.1. I have created a config where I have used a multiline filter as one event in my log spans accross multiple lines.
    Apart from other filters which I have in my config, my last filter is : 
     multiline {
		pattern => "Phase\s\d"
		negate => true
		what => previous
		periodic_flush => true
		allow_duplicates => true
           }

      In this filter I want to club all lines into one field at the same time I do not want to remove any duplicates. The field in question contains record counts out of each component in a program, which can be similar. However when I get the output it is dedupped. Can anyone help on this? 
     I saw a few discussions where it was raised as a bug in version 1.4.2 and later some fix was put in place. Can anyone tell me if that fix worked and is that fix incorporated in the version I am using (version -2.3.1) .

```

Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 30, 2016, 11:17pm UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304/2 "2016-05-30T23:17:48Z")

</div>

Can you format your post please, it's very hard to read with such wide lines.

---

<div class="post-metadata">

**Author:** ![Avishek](https://avatars.discourse-cdn.com/v4/letter/a/b77776/32.png) [@Avishek](https://discuss.elastic.co/u/Avishek)\
**Post date:** [May 31, 2016, 5:46am UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304/3 "2016-05-31T05:46:51Z")

</div>

Hello Mark,

```
             I have tried to get the lines shorter. Please see if this helps:

```

Original post formated:

I am using logstash 2.3.1. I have created a config where I have used a multiline filter  
as one event in my log spans accross multiple lines. Apart from other filters which  
I have in my config, my last filter is :  
multiline {  
pattern =\> "Phase\s\d"  
negate =\> true  
what =\> previous  
periodic\_flush =\> true  
allow\_duplicates =\> true  
}

In this filter I want to club all lines into one field at the same time I do not want to  
remove any duplicates. The field in question contains record counts out of each component  
in a program, which can be similar. However when I get the output it is dedupped. Can  
anyone help on this?

I saw a few discussions where it was raised as a bug in version 1.4.2 and later some fix  
was put in place. Can anyone tell me if that fix worked and is that fix incorporated in  
the version I am using (version -2.3.1).

Thanks,

---

<div class="post-metadata">

**Author:** ![Avishek](https://avatars.discourse-cdn.com/v4/letter/a/b77776/32.png) [@Avishek](https://discuss.elastic.co/u/Avishek)\
**Post date:** [June 13, 2016, 8:08am UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304/4 "2016-06-13T08:08:30Z")

</div>

Can anyone help me out with this? Looks like multiline is not functioning as expected. Does anyone have a solution to this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:53am UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304/5 "2017-07-06T04:53:25Z")

</div>

Closed on July 6, 2017, 4:53am UTC