Multiline cuts off character

[manuelk (Manuel)]

Hello,

we´ve a case where Multiline seems to cut off the first character of a string within Filebeat 6.2.4.

Mulitline setting:

multiline.pattern: '^(*.JOB)|Cooldown'
multiline.negate: true
multiline.match: after

Message in File:

Filebeat output:

But this only happens when the file gets updated. When the file gets initially loaded, this does not happen.
Anyone a suggestions what could be the issue?

[andrewkroh]

Is the cut off first character included included with the previous event?

Can you try running with multiline debug enabled. If there's any flush triggered by a timeout there will be log message.

logging.level: debug
logging.selectors: [multiline]

[manuelk (Manuel)]

Hey,

thanks for your reply. I created a log and it looks like there is an timeout:

Could this cause the issue? Could also explain why it works on old files where all datapoints are already available..

[andrewkroh]

With pre-existing log data contained in the files the issue won't occur because Filebeat is able to read multiline data as fast as it can. Filebeat does not have to wait for the start of the next multiline block which is what normally triggers it to consider the previous multiline block to be complete and send it in an event.

This is probably caused by the process writing the file to not be flushing frequently enough. To account for this you can increase the multiline timeout value in Filebeat (see multiline.timeout).

[manuelk (Manuel)]

Hey,

yes this process is a real "slow mover" so we´re getting events every 5 minutes. I tried to increase timeout up to 7 minutes but still facing the same problem.

[andrewkroh]

Can you try a really long value just to see if it fixes the issue? Worst case -- the most recent event will be delayed a bit.

multiline.pattern: '^(*.JOB)|Cooldown'
multiline.negate: true
multiline.match: after
multline.timeout: 1h

Do your events have a predictable string that can be used to flush? If so you could try the multiline.flush_pattern. (original pull request for the feature)

[manuelk (Manuel)]

Unfortunately the extended timeout did not work either. There´s no predictable string for the flush_pattern - message is quite dynamic except the start pattern.

Do you have somethingin mind why it´s just affecting the first character (and only if a new event is added) ?

[andrewkroh]

@manuelk

Unfortunately the extended timeout did not work either.

Did the logs still say that the multiline timeout was reached? That's the only reason I can think of that would cause a split.

[manuelk (Manuel)]

Hey,

sorry for the late reply - Now the timeout disappeared in the error log. I can´t find any further error message but the issue with the missing char still occurs.