Filebeat multiline matching Docker log

Elastic Stack Beats
,
1

Hello,

After a bit of assistance with multiline pattern matching.

I'm using filebeat autodiscover Docker provider which is fine sending the logs to logstash, but pattern matching on timestamp doesn't seem to be working correctly.

This is filebeat config

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      config:
        - json.keys_under_root: true
        - json.add_error_key: true
        - json.message_key: execDesc
      multiline.type: pattern
      multiline.pattern: '^["][0-9]{4}[-][0-9]{2}[-][0-9]{2}[T][0-9]{2}[:][0-9]{2}[:][0-9]{2}[.][0-9]{3}[Z]'
      multiline.negate: true
      multiline.match: after

This is an example of Docker log output using docker-compose logs

tymly-alpha-api    | 2021-11-26T14:20:37.112Z statebox sendTaskHeartbeat(nfcc_brrAuditForm_1_0-0678a850-4ec4-11ec-9782-8757a1220438-452)
tymly-alpha-api    | 2021-11-26T14:20:37.119Z tymly-fastify-plugin execDesc: {
tymly-alpha-api    |   executionName: 'nfcc_brrAuditForm_1_0-0678a850-4ec4-11ec-9782-8757a1220438-452',
tymly-alpha-api    |   ctx: {
tymly-alpha-api    |     uprn: '12345678',
tymly-alpha-api    |     addressLabel: 'blah blah',
tymly-alpha-api    |     schedule: {
tymly-alpha-api    |       uprn: '12345685',
tymly-alpha-api    |       addressLabel: 'blah blah',
tymly-alpha-api    |       adopted: true,
tymly-alpha-api    |       adoptedBy: 'first.last@org.com',
tymly-alpha-api    |       adoptedDate: '2021-11-26T14:20:31.087Z',
tymly-alpha-api    |       brigade: 'Some FRS',
tymly-alpha-api    |       mhclgCode: null,
tymly-alpha-api    |       addedToSchedule: null,
tymly-alpha-api    |       importedFromSpreadsheet: null,
tymly-alpha-api    |       importLogId: null,
tymly-alpha-api    |       status: 'Initial assessment complete',
tymly-alpha-api    |       importedFromGeoplace: null,
tymly-alpha-api    |       created: '2021-04-28T13:01:57.280Z',
tymly-alpha-api    |       createdBy: 'tymly.test@org.net',
tymly-alpha-api    |       modified: '2021-11-26T14:20:31.094Z',
tymly-alpha-api    |       modifiedBy: 'first.last@org.net'
tymly-alpha-api    |     },
tymly-alpha-api    |     requiredHumanInput: {
tymly-alpha-api    |       uiName: 'nfcc_brrAuditForm',
tymly-alpha-api    |       uiType: 'form',
tymly-alpha-api    |       uiRefresh: undefined,
tymly-alpha-api    |       data: [Object]
tymly-alpha-api    |     }
tymly-alpha-api    |   },
tymly-alpha-api    |   currentStateName: 'AwaitingHumanInput',
tymly-alpha-api    |   currentResource: 'module:awaitingHumanInput',
tymly-alpha-api    |   stateMachineName: 'nfcc_brrAuditForm_1_0',
tymly-alpha-api    |   status: 'RUNNING',
tymly-alpha-api    |   executionOptions: {
tymly-alpha-api    |     instigatingClient: { appName: 'tymly-frontend-alpha', domain: '' },
tymly-alpha-api    |   },
tymly-alpha-api    |   parentExecution: null,
tymly-alpha-api    |   childCount: null,
tymly-alpha-api    |   created: 2021-11-26T14:20:37.077Z,
tymly-alpha-api    |   createdBy: 'first.last@org.net',
tymly-alpha-api    |   modified: 2021-11-26T14:20:37.108Z,
tymly-alpha-api    |   modifiedBy: 'first.last@org.net'
tymly-alpha-api    | }

So I think the regex pattern is correct for that timestamp?

But when viewing the logs in Kibana each line is showing as it's own message / document rather than being collated under the previous line with the timestamp. So even the lines which is just a brace is showing as a single message.

Interestingly when I cat the actual Docker log file each line starts with a ".

for example

{"log":"2021-11-26T13:58:23.770Z tymly-fastify-plugin execDesc: {\n","stream":"stderr","time":"2021-11-26T13:58:23.774187897Z"}
{"log":"  executionName: 'wmfs_viewProperty_1_0-eb4e3020-4ec0-11ec-9782-8757a1220438-329',\n","stream":"stderr","time":"2021-11-26T13:58:23.774228869Z"}
{"log":"  ctx: {\n","stream":"stderr","time":"2021-11-26T13:58:23.774233751Z"}
{"log":"    uprn: 100071304870,\n","stream":"stderr","time":"2021-11-26T13:58:23.774237345Z"}

I've tried updating the pattern to start with ["] to account for that but doesn't seem to help.

Thanks.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.