Handling multiline log with a timestamp on each line

[ricky.kwan.ix (Ricky)]

Hi,

I'm trying to combine multiple log lines into a single line, but the issue is that each line has a timestamp (among other things) on it.

Jan 30 2020 16:52:16 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (2366144456 total) msec
Jan 30 2020 16:52:16 GMT: INFO (info): (hist.c:257)  (00: 2348746349) (01: 0002309726) (02: 0002863034) (03: 0011897069)
Jan 30 2020 16:52:16 GMT: INFO (info): (hist.c:257)  (04: 0000212507) (05: 0000095451) (06: 0000017783) (07: 0000002284)
Jan 30 2020 16:52:16 GMT: INFO (info): (hist.c:257)  (08: 0000000236) (09: 0000000011) (10: 0000000004) (11: 0000000001)
Jan 30 2020 16:52:16 GMT: INFO (info): (hist.c:266)  (12: 0000000001)

As such, the combined line ends up having the timestamp (and the other stuff) multiple times. Has anyone had a similar situation before and worked around it? I was hoping that multiline.pattern could have capturing groups or something to keep the relevant parts but that doesn't seem to be the case.

I'm running 6.4 (no new multiline-related configs in 7.X) with the following settings:

- fields_under_root: true
  paths:
    - /var/log/aerospike/aerospike.log
  type: log
  multiline.pattern: "[a-zA-Z]{3} \\d{1,2} \\d{4} \\d{2}:\\d{2}:\\d{2}(\\.\\d+)? [A-Z]{3}: [A-Z]+ \\([a-z_-]+\\): \\(hist\\.c:\\d+\\)(\\s+\\(\\d+: \\d+\\))+"
  multiline.negate: false
  multiline.match: after

[Kaiyan_Sheng]

Hi @ricky.kwan.ix, this is a very interesting problem to solve. Maybe https://www.elastic.co/guide/en/beats/filebeat/current/_examples_of_multiline_configuration.html#_application_events will help?! The start new event in your case would be histogram dump: and end event would be (12: ...)?

[ricky.kwan.ix (Ricky)]

Hi,

I tried your suggestion with the same result. And maybe it wasn't clear in my original post what I wanted, so I edited it.

Here is a truncated test result after applying the flush_pattern config.

  "message": "Jan 30 2020 22:43:04.934 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (11 total) msec\nJan 30 2020 22:43:04.934 GMT: INFO (info): (hist.c:266)  (00: 0000000011)"

The middle of the line has the timestamp, loglevel, etc that I want to remove when combined.