# Filebeat multiline ignores line if the newline doesn't exist

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098>\
**Category:** Beats\
**Created:** [April 1, 2016, 8:20pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098 "2016-04-01T20:20:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielmotaleite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmotaleite/32/19796_2.png) [@danielmotaleite](https://discuss.elastic.co/u/danielmotaleite)\
**Post date:** [April 1, 2016, 8:20pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098/1 "2016-04-01T20:20:54Z")

</div>

Using filebeat 1.1.2, i have a log event that have 3 lines and starts with the date, but the last line have no newline character. After 5 seconds, filebeat sends the 2 first lines of the event... after 10 minutes i get the same event and filebeat sends the last line of the previous event, then detects the date and start a new multiline. example:

2016-04-01 11:11:11 blabla\n \_ sent after 5 s  
bla bla bla\n /  
bla bla no newline - sent after 10 minutes  
(repeat the above lines each 10 minutes)

So looks like that multiline check will not look to the line is it doesn't end with a newline.  
Only after a new event is trigger it will see it (probably the "date" is really "\ndate", but is not easy to check)

Of course the log is broke, but i can not change the app... so filebeat needs one option to "ignore missing newline on timeout" so it can catch the last line without a newline on it. Or is there any workaround for this?

Thanks in advance for the help

Daniel

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 4, 2016, 11:29am UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098/2 "2016-04-04T11:29:34Z")

</div>

processing in filebeat is more like:

`line reader -> multiline -> publish`. The multiline handler by default has some timeout configured, to print the currently buffered event if no full line has been received for some time.

Look for `timeout` option in [multiline settings](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#multiline). Default is 5s. You can set this value to some very large value or set to '-1s' to disable multiline timeout.  
Drawback: event will be only generated if events newline is finally written to file. This is a common problem with loggers writing the newline character only if new log is to be appended. Problem with 'partial' lines is: it clashes with loggers buffering output without flushing-timer.

---

<div class="post-metadata">

**Author:** ![danielmotaleite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmotaleite/32/19796_2.png) [@danielmotaleite](https://discuss.elastic.co/u/danielmotaleite)\
**Post date:** [April 4, 2016, 1:37pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098/3 "2016-04-04T13:37:38Z")

</div>

Thanks for the reply

yes, i understand, but i keeping pushing the timeout and if there is no new event, the last even will be hidden for a unknown time

i would suggest adding a optional preference for _line\_reader\_timeout_ (disable by default), so after that time the _line reader_ would push to the _multiline_ the partial line it would already have. This would not be used for most people, but would allow one to fix this kind of problems on logs with small number of events

Another alternative would be adding a option _flush\_on\_text_, where when some defined text/regexp would force the _line reader_ to flush even lines without newline. Or even replace the newline with other text as line ending flag, again with a new option _line\_ending_

this seems like the only way to fix this, give the admin some control to _line reader_, either by timeout or by matching characters or newline replacement.

fixing the log format would be better, but not everyone can do it

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 4, 2016, 3:50pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098/4 "2016-04-04T15:50:03Z")

</div>

What's you application based off? I remember having seen similar issues with PHP based logging framework.

I agree, the reader should be somehow configurable to adapt. Sometimes we can not change/adapt the logger.

Feature you're asking for is partial line support. I'd recommend to add [a ticket](https://github.com/elastic/beats/issues) with partial line support based on timeouts.

Not sure about regex based matching. You've got a good use-case/sample?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:53pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-line-if-the-newline-doesnt-exist/46098/5 "2017-07-05T21:53:51Z")

</div>


