Problem with multiline filebeat docker prospector (,\n)

Elastic Stack Beats
1

Hi
I configured filebeat 6.2.1 like this:

filebeat.prospectors:
- type: docker
  document_type: eoclog4j
  paths:
  - '/var/lib/docker/containers/*/*.log'
  containers.ids:
  - '*'
  multiline:
    pattern: '([0-9]{4}-[0-9]{2}-[0-9]{2})|(\[[0-9]{2}.[0-9]{2}.[0-9]{4})'
    negate: true
    match: after
  fields_under_root: true
  fields:
    source_system: filebeat_docker
  processors:
  - add_docker_metadata: ~

Everything seems ok except that sometime multiline fails. It seems to me (I know this is strange but I cannot find other explanations) that this happens when lines ends with: ,\n
Example:

This went wrong (lines are divided):
{"log":"2018-03-01T14:34:21.032-0000 - ERROR - [server] - Error in blablblabla:
'2018-02-23 13:43:59.653165000 +0000',\n","stream":"stdout","time":"2018-03-01T14:34:21.145493486Z"}
{"log":" 'blablabla': '2018-02-23 13:43:59.465964600 +0000',\n","stream":"stdout","time":"2018-03-01T14:34:21
.14552074Z"}

This works OK (lines goes in same event):
{"log":" at blablabla\n","stream":"stdout","time":"2018-03-01T14:34:21.145679703Z"}
{"log":" at blablabla (events.js:126:13)\n","stream":"stdout","time":"2018-03-01T14:34:21.145683059Z"}
{"log":" at blablabla (events.js:214:7)\n","stream":"stdout","time":"2018-03-01T14:34:21.145686123Z"}

Any ideas?
Many thanks

2

To figure out what goes wrong with the regexp I often use the go playground linked here to test the regexp: https://www.elastic.co/guide/en/beats/filebeat/6.2/_test_your_regexp_pattern_for_multiline.html

3

Thank you ruflin.:clap::clap:
I did what you suggested and I found the problem.
I missed the ^ character in beginning of my pattern so it matched also wrong lines.
Many thanks

4

By the way... what about to put go playground tool you suggested in kibana dev tools (together grok tester) ? It could be useful.
Thanks again

5

I quite like the idea. The main issue here is that regexp != regexp. So we would need to have to go regexp executed in JS somehow. I will definitively keep the idea in mind.

We are also in the progress to make testing easier directly in Filebeat.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.