I am trying to join two lines together from an IPlanet access log output. Below is my filebeat.yml, output from filebeats and log input. I know I'm missing something but not sure what. I tested the pattern specified in a reg-ex tester and it gives the desired results but when used for the mutiline pattern no success. In Kibana the 2 lines still come in as 2 events. Any ideas welcome.
Filebeat.yml:
filebeat.prospectors:
- type: log
paths:- /data/prd-logs/////*SystemOut.log
- /data/prd-logs/////*access
multiline.pattern: ^(["])
multiline.negate: false
multiline.match: before
output.logstash:
hosts: ["10.136.112.38:5044"]
username: "xxx"
password: "xxx"
setup.kibana:
host: "10.136.112.38:5601"
username: "xxx"
password: "xxx"
Filebeats output snipets: (The < quoted block below is an example of the 2 lines to be merged)
2018-07-20T13:33:50.230-0400 INFO instance/beat.go:468 Home path: [/opt/software/elk/filebeat-6.2.4-linux-x86_64] Config path: [/opt/software/elk/filebeat-6.2.4-linux-x86_64] Data path: [/opt/software/elk/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/software/elk/filebeat-6.2.4-linux-x86_64/logs]
2018-07-20T13:33:50.230-0400 INFO instance/beat.go:475 Beat UUID: f7e723aa-3cec-43d1-b3e2-1b7c82dded2e
2018-07-20T13:33:50.230-0400 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2018-07-20T13:33:50.231-0400 INFO pipeline/module.go:76 Beat name: alccad001.vwoa.na.vwg
2018-07-20T13:33:50.231-0400 INFO instance/beat.go:301 filebeat start running.
2018-07-20T13:33:50.231-0400 INFO registrar/registrar.go:110 Loading registrar data from /opt/software/elk/filebeat-6.2.4-linux-x86_64/data/registry
2018-07-20T13:33:50.231-0400 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-07-20T13:33:50.234-0400 INFO registrar/registrar.go:121 States Loaded from registrar: 211
2018-07-20T13:33:50.234-0400 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-20T13:33:50.234-0400 INFO crawler/crawler.go:48 Loading Prospectors: 1
2018-07-20T13:33:50.702-0400 INFO log/prospector.go:111 Configured paths: [/data/prd-logs/////SystemOut.log /data/prd-logs////*/*access]
2018-07-20T13:33:50.703-0400 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 1
2018-07-20T13:34:01.076-0400 INFO log/harvester.go:216 Harvester started for file: /data/prd-logs/aleaap001/aws/app/was9/aws_appserver1_SystemOut.log
2018-07-20T13:34:01.076-0400 INFO log/harvester.go:216 Harvester started for file: /data/prd-logs/alsqap003/gff/app/was9/gff_appserver1_SystemOut.log
2018-07-20T13:34:01.077-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-07-20T17:34:01.081Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.4"
},
"beat": {
"name": "alccad001.vwoa.na.vwg",
"hostname": "alccad001.vwoa.na.vwg",
"version": "6.2.4"
},
"source": "/data/prd-logs/aleawp002/wsa/web/wsa_was9_prd2/access",
"offset": 7508847,
"message": "10.136.124.241 - - [20/Jul/2018:13:33:31 -0400] "GET / http/1.1" 200 97",
"prospector": {
"type": "log"
}
}
2018-07-20T13:34:01.081-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-07-20T17:34:01.081Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.4"
},
"beat": {
"name": "alccad001.vwoa.na.vwg",
"hostname": "alccad001.vwoa.na.vwg",
"version": "6.2.4"
},
"offset": 7508898,
"message": "10.136.124.241 - - [20/Jul/2018:13:33:31 -0400] "",
"source": "/data/prd-logs/aleawp002/wsa/web/wsa_was9_prd2/access",
"prospector": {
"type": "log"
}
}
2018-07-20T13:34:01.081-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-07-20T17:34:01.081Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.4"
},
"prospector": {
"type": "log"
},
"beat": {
"version": "6.2.4",
"name": "alccad001.vwoa.na.vwg",
"hostname": "alccad001.vwoa.na.vwg"
},
"source": "/data/prd-logs/aleawp002/wsa/web/wsa_was9_prd2/access",
"offset": 7508908,
"message": "" 400 147"
}
2018-07-20T13:34:01.081-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-07-20T17:34:01.081Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.4"
},
"source": "/data/prd-logs/aleawp002/wsa/web/wsa_was9_prd2/access",
"offset": 7509029,
"message": "10.136.124.243 - - [20/Jul/2018:13:33:31 -0400] "POST /WSAgentRegistration/WSAgentAuthorizationService HTTP/1.1" 200 736",
"prospector": {
"type": "log"
},
"beat": {
"name": "alccad001.vwoa.na.vwg",
"hostname": "alccad001.vwoa.na.vwg",
"version": "6.2.4"
}
}
2018-07-20T13:34:01.081-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
Log file excerpts: (matches the log entries above.
10.136.124.241 - - [20/Jul/2018:13:33:31 -0400] "GET / http/1.1" 200 97
10.136.124.241 - - [20/Jul/2018:13:33:31 -0400] "^M
" 400 147
10.136.124.243 - - [20/Jul/2018:13:33:31 -0400] "POST /WSAgentRegistration/WSAgentAuthorizationService HTTP/1.1" 200 736