Filebeat multiline splitting events - which timeout to tweak?


#1

Hey, I'm using Filebeat 6.1.1 on a number of Linux hosts. I'm prospecting apt-get logs, which look like this usually

Start-Date: 2018-01-03  20:45:26
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold dist-upgrade
Upgrade: linux-image-4.9.0-4-amd64:amd64 (4.9.65-3, 4.9.65-3+deb9u1)
End-Date: 2018-01-03  20:45:38

Unfortunately in cases where "End-Date" occurs more than a few seconds after "Start-Date" something times out and the End-Date line gets punted into a separate event. Which ends up looking like this

That's bad. So I attempted to boost the multiline timeout but for some reason this didn't help matters at all. The apt-get and multiline stanza in my filebeat.yml config file currently looks like this.

- input_type: log
  paths:
    - /var/log/apt/history.log
  fields:
    type: apt
  multiline.pattern: Start-Date
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: End-Date
  timeout: 60

How should I fix this? Everything I've read online suggests that if you're willing to sacrifice latency by boosting 'the timeout' then this should work. But it doesn't. Is it the wrong timeout? I'm out of ideas :frowning:


(Adrian Serrano) #2

You have to set the timeout in the multiline configuration, like this:

multiline.timeout: 60s


#3

Thank you! This works. Sometimes it's the little things (like syntax).


(system) #4

This topic was automatically closed after 21 days. New replies are no longer allowed.