Hello,
After a bit of assistance with multiline pattern matching.
I'm using filebeat autodiscover Docker provider which is fine sending the logs to logstash, but pattern matching on timestamp doesn't seem to be working correctly.
This is filebeat config
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
config:
- json.keys_under_root: true
- json.add_error_key: true
- json.message_key: execDesc
multiline.type: pattern
multiline.pattern: '^["][0-9]{4}[-][0-9]{2}[-][0-9]{2}[T][0-9]{2}[:][0-9]{2}[:][0-9]{2}[.][0-9]{3}[Z]'
multiline.negate: true
multiline.match: after
This is an example of Docker log output using docker-compose logs
tymly-alpha-api | 2021-11-26T14:20:37.112Z statebox sendTaskHeartbeat(nfcc_brrAuditForm_1_0-0678a850-4ec4-11ec-9782-8757a1220438-452)
tymly-alpha-api | 2021-11-26T14:20:37.119Z tymly-fastify-plugin execDesc: {
tymly-alpha-api | executionName: 'nfcc_brrAuditForm_1_0-0678a850-4ec4-11ec-9782-8757a1220438-452',
tymly-alpha-api | ctx: {
tymly-alpha-api | uprn: '12345678',
tymly-alpha-api | addressLabel: 'blah blah',
tymly-alpha-api | schedule: {
tymly-alpha-api | uprn: '12345685',
tymly-alpha-api | addressLabel: 'blah blah',
tymly-alpha-api | adopted: true,
tymly-alpha-api | adoptedBy: 'first.last@org.com',
tymly-alpha-api | adoptedDate: '2021-11-26T14:20:31.087Z',
tymly-alpha-api | brigade: 'Some FRS',
tymly-alpha-api | mhclgCode: null,
tymly-alpha-api | addedToSchedule: null,
tymly-alpha-api | importedFromSpreadsheet: null,
tymly-alpha-api | importLogId: null,
tymly-alpha-api | status: 'Initial assessment complete',
tymly-alpha-api | importedFromGeoplace: null,
tymly-alpha-api | created: '2021-04-28T13:01:57.280Z',
tymly-alpha-api | createdBy: 'tymly.test@org.net',
tymly-alpha-api | modified: '2021-11-26T14:20:31.094Z',
tymly-alpha-api | modifiedBy: 'first.last@org.net'
tymly-alpha-api | },
tymly-alpha-api | requiredHumanInput: {
tymly-alpha-api | uiName: 'nfcc_brrAuditForm',
tymly-alpha-api | uiType: 'form',
tymly-alpha-api | uiRefresh: undefined,
tymly-alpha-api | data: [Object]
tymly-alpha-api | }
tymly-alpha-api | },
tymly-alpha-api | currentStateName: 'AwaitingHumanInput',
tymly-alpha-api | currentResource: 'module:awaitingHumanInput',
tymly-alpha-api | stateMachineName: 'nfcc_brrAuditForm_1_0',
tymly-alpha-api | status: 'RUNNING',
tymly-alpha-api | executionOptions: {
tymly-alpha-api | instigatingClient: { appName: 'tymly-frontend-alpha', domain: '' },
tymly-alpha-api | },
tymly-alpha-api | parentExecution: null,
tymly-alpha-api | childCount: null,
tymly-alpha-api | created: 2021-11-26T14:20:37.077Z,
tymly-alpha-api | createdBy: 'first.last@org.net',
tymly-alpha-api | modified: 2021-11-26T14:20:37.108Z,
tymly-alpha-api | modifiedBy: 'first.last@org.net'
tymly-alpha-api | }
So I think the regex pattern is correct for that timestamp?
But when viewing the logs in Kibana each line is showing as it's own message / document rather than being collated under the previous line with the timestamp. So even the lines which is just a brace is showing as a single message.
Interestingly when I cat the actual Docker log file each line starts with a ".
for example
{"log":"2021-11-26T13:58:23.770Z tymly-fastify-plugin execDesc: {\n","stream":"stderr","time":"2021-11-26T13:58:23.774187897Z"}
{"log":" executionName: 'wmfs_viewProperty_1_0-eb4e3020-4ec0-11ec-9782-8757a1220438-329',\n","stream":"stderr","time":"2021-11-26T13:58:23.774228869Z"}
{"log":" ctx: {\n","stream":"stderr","time":"2021-11-26T13:58:23.774233751Z"}
{"log":" uprn: 100071304870,\n","stream":"stderr","time":"2021-11-26T13:58:23.774237345Z"}
I've tried updating the pattern to start with ["] to account for that but doesn't seem to help.
Thanks.