# Multiline message problem

**URL:** <https://discuss.elastic.co/t/multiline-message-problem/226183>\
**Category:** Logstash\
**Created:** [April 2, 2020, 9:05am UTC](https://discuss.elastic.co/t/multiline-message-problem/226183 "2020-04-02T09:05:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rutger2000](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@Rutger2000](https://discuss.elastic.co/u/Rutger2000)\
**Post date:** [April 2, 2020, 9:05am UTC](https://discuss.elastic.co/t/multiline-message-problem/226183/1 "2020-04-02T09:05:50Z")

</div>

Hello,

I'm parsing multiline messages to elastic using Logstash. 99,9% of the messages are parsed correctly, but a small part is not. As far as I can see I'm not getting any errors. The messages that are not parsed correctly are available in elastic, but somehow the original timestamp is removed from the message and is given the timestamp of the moment the messages was parsed. Also none of the information in the messages is indexed. Some help would be nice.

script:

input {

file {  
path =\> ["C:/path"]  
sincedb\_path =\> "nul"  
start\_position =\> "beginning"  
mode =\> "read"  
codec =\> multiline {  
pattern =\> "^%{TIMESTAMP\_ISO8601} "  
negate =\> true  
what =\> previous  
max\_lines =\> 100000  
auto\_flush\_interval =\> 1  
}  
}  
} #end input

filter{

fingerprint {  
source =\> "message"  
target =\> "[@metadata][fingerprint]"  
method =\> "MURMUR3"  
}

grok{  
match =\> {  
"message" =\> ["%{TIMESTAMP\_ISO8601:timestamp}%{GREEDYDATA:rest}"]  
}  
}

grok{  
match =\> {  
"rest" =\> ['value1%{SPACE}=%{SPACE}%{DATA:field1}\r']  
}  
}

grok{  
match =\> {  
"rest" =\> ['value2%{SPACE}=%{SPACE}%{NUMBER:field2}']  
}  
}

grok{  
match =\> {  
"rest" =\> ['value3%{SPACE}:%{SPACE}%{NOTSPACE:field3}']  
}  
}

if [field1] == "100" {

```
grok{
  match => {
    "rest" => ['value4.%{SPACE}:%{SPACE}%{GREEDYDATA:field4}']
  }
}

```

}

```
grok{
  match => {
    "value5" => ['%{GREEDYDATA:field5}\n']
  }
}

```

dissect {  
mapping =\> {  
'field5' =\> "%{value6-\>} %{value7-\>} "  
}  
}

```
ruby {
  code=> "
    mat = event.get('value8').scan(/value9[2-3]([A-Z]+ [0-9]+)/)
    event.set('field10', mat.flatten)
            "
}

```

} # end if1

date{  
match =\> ["timestamp", "yyyy-MM-dd HH:mm:ss"]  
timezone =\> "Europe/London"  
target =\> "@timestamp"  
}

if [id\_kort] != "value11" and "value12" in [message] {  
mutate {  
add\_field =\> { "field14" =\> "%{value13}%{value14}" }  
}  
}

mutate {  
remove\_field=\> ["field4"]  
gsub=\> ["field15", " ", ""]  
strip=\> ["field2", "field3"]  
}

} # end filter

output {

elasticsearch{  
hosts =\> "localhost:9200"  
document\_id =\> "%{[@metadata][fingerprint]}"  
index =\> "name1"  
}

} # end output

logfile:

2019-12-16 08:00:00 message received; id=105  
field1: value1  
field2: value2  
field3:value3  
2019-12-16 08:00:01 message received; id =107  
field1: value5  
field2: value6  
field3:value7  
2019-12-16 08:00:03 messega received; id =110  
field1: value8  
field2: value9  
field3: value10

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 2, 2020, 1:07pm UTC](https://discuss.elastic.co/t/multiline-message-problem/226183/2 "2020-04-02T13:07:23Z")

</div>

> [@Rutger2000](#):
>
> auto\_flush\_interval =\> 1

Is it possible that messages are being split across events due to the low value for the auto flush interval?

---

<div class="post-metadata">

**Author:** ![Rutger2000](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@Rutger2000](https://discuss.elastic.co/u/Rutger2000)\
**Post date:** [April 2, 2020, 2:38pm UTC](https://discuss.elastic.co/t/multiline-message-problem/226183/3 "2020-04-02T14:38:28Z")

</div>

That's it! Thanks so much Badger, unlimited Kudo's for you!!! 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2020, 2:38pm UTC](https://discuss.elastic.co/t/multiline-message-problem/226183/4 "2020-04-30T14:38:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.