Problem with multiline filebeat docker prospector (,\n)

(Meril) #1

I configured filebeat 6.2.1 like this:

- type: docker
  document_type: eoclog4j
  - '/var/lib/docker/containers/*/*.log'
  - '*'
    pattern: '([0-9]{4}-[0-9]{2}-[0-9]{2})|(\[[0-9]{2}.[0-9]{2}.[0-9]{4})'
    negate: true
    match: after
  fields_under_root: true
    source_system: filebeat_docker
  - add_docker_metadata: ~

Everything seems ok except that sometime multiline fails. It seems to me (I know this is strange but I cannot find other explanations) that this happens when lines ends with: ,\n

This went wrong (lines are divided):
{"log":"2018-03-01T14:34:21.032-0000 - ERROR - [server] - Error in blablblabla:
'2018-02-23 13:43:59.653165000 +0000',\n","stream":"stdout","time":"2018-03-01T14:34:21.145493486Z"}
{"log":" 'blablabla': '2018-02-23 13:43:59.465964600 +0000',\n","stream":"stdout","time":"2018-03-01T14:34:21

This works OK (lines goes in same event):
{"log":" at blablabla\n","stream":"stdout","time":"2018-03-01T14:34:21.145679703Z"}
{"log":" at blablabla (events.js:126:13)\n","stream":"stdout","time":"2018-03-01T14:34:21.145683059Z"}
{"log":" at blablabla (events.js:214:7)\n","stream":"stdout","time":"2018-03-01T14:34:21.145686123Z"}

Any ideas?
Many thanks

(ruflin) #2

To figure out what goes wrong with the regexp I often use the go playground linked here to test the regexp:

(Meril) #3

Thank you ruflin.:clap::clap:
I did what you suggested and I found the problem.
I missed the ^ character in beginning of my pattern so it matched also wrong lines.
Many thanks

(Meril) #4

By the way... what about to put go playground tool you suggested in kibana dev tools (together grok tester) ? It could be useful.
Thanks again

(ruflin) #5

I quite like the idea. The main issue here is that regexp != regexp. So we would need to have to go regexp executed in JS somehow. I will definitively keep the idea in mind.

We are also in the progress to make testing easier directly in Filebeat.

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.