Java multiline with add_docker_metadata

[mimmus]

We are using this:

Enriching logs with Docker metadata using Filebeat

and now we are trying to configure multiline to fetch Java stacktrace as a single message but any attempt fails.
We are pretty sure that patterns used to identify multiline are OK.
Is it possible to use multiline with Docker json logs?

[adrisr (Adrian Serrano)]

Can you share your filebeat config?

[mimmus]

Actually, I'm getting some kind of aggregation but also a lot of single lines, I don't understand the reason

filebeat.modules:
- module: kafka

filebeat.prospectors:
- type: log
  paths:
    - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  multiline.pattern: '^MSG|WNG|ERR'
  multiline.negate: true
  multiline.match: after
  tail_files: true
  processors:
  - add_docker_metadata: ~


setup.kibana:
  host: "plogelk06.mydomain.com:5601"

output.kafka:
  hosts: ["plogelk01:9092", "plogelk02:9092", "plogelk03:9092"]
  codec.format:
    string: 'beat=%{[beat]} message=%{[log]}'
  topic: 'test_svil_filebeat'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

I can send you also some log lines but better privately as it is not easy to "anonymize".

Thanks

[adrisr (Adrian Serrano)]

Yes please, send me those logs as a private message

[mimmus]

Please accept my apologies: I had TWO filebeat containers running at same moment, with different configs!