Filebeat nginx grok error

[jvanhoose]

We are getting a strange grok error using the nginx module in filebeat 5.5.2. The server config is:

/etc/filebeat/filebeat.yml:

filebeat:

nginx filebeat module

modules:

• module: nginx
  access:
  enabled: true
  var.pipeline: with_plugins
  var.paths:
  • /var/log/opscode/nginx/access.log
    prospector:
    tags: ["nginxaccess","chefserver"]
    error:
    enabled: true
    var.paths:
  • /var/log/opscode/nginx/error.log
    prospector:
    tags: ["nginxerror","chefserver"]
    config_dir: /etc/filebeat/conf.d
    name: est-chef-frontend-node1.campus.ksu.edu
    fields: {
    ipaddress: "10.139.40.10"
    }

output.elasticsearch:
hosts: ["https://elasticsearch.ksu.edu:5002"]
template.enabled: true
template.path: "filebeat.template.json"
template.overwrite: false
index: "filebeat-%{+yyyy.MM.dd}"
username: "filebeat"
password:

When we get the messages into elasticsearch, they are being given an error:

{
"_index": "filebeat-2017.08.24",
"_type": "doc",
"_id": "AV4WV2nlBaN2Bc9kgIf8",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2017-08-24T22:23:08.843Z",
"offset": 10752933,
"beat": {
"hostname": "est-chef-frontend-node1",
"name": "est-chef-frontend-node1",
"version": "5.5.2"
},
"input_type": "log",
"source": "/var/log/opscode/nginx/access.log",
"message": "129.130.254.176 - - [24/Aug/2017:17:23:07 -0500] "GET /organizations/omedev/data/ome_webserver_www_ksu_edu/ksre_feedtechnology HTTP/1.1" 200 "0.015" 262 "-" "Chef Client/12.6.0 (ruby-2.1.6-p336; ohai-8.8.1; x86_64-linux; +https://chef.io)" "127.0.0.1:8000" "200" "0.015" "12.6.0" "algorithm=sha1;version=1.0;" "ome-vm6.campus.ksu.edu" "2017-08-24T22:22:55Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1150",
"type": "log",
"error": "Provided Grok expressions do not match field value: [129.130.254.176 - - [24/Aug/2017:17:23:07 -0500] \"GET /organizations/omedev/data/ome_webserver_www_ksu_edu/ksre_feedtechnology HTTP/1.1\" 200 \"0.015\" 262 \"-\" \"Chef Client/12.6.0 (ruby-2.1.6-p336; ohai-8.8.1; x86_64-linux; +https://chef.io)\" \"127.0.0.1:8000\" \"200\" \"0.015\" \"12.6.0\" \"algorithm=sha1;version=1.0;\" \"ome-vm6.campus.ksu.edu\" \"2017-08-24T22:22:55Z\" \"2jmj7l5rSw0yVb/vlWAYkK/YBwk=\" 1150]",
"tags": [
"nginxaccess",
"chefserver"
]
},
"fields": {
"@timestamp": [
1503613388843
]
},
"highlight": {
"tags": [
"@kibana-highlighted-field@nginxaccess@/kibana-highlighted-field@"
]
},
"sort": [
1503613388843
]
}

As an example, here are a couple of lines from the nginx access file:

10.139.40.1 - - [24/Aug/2017:17:24:50 -0500] "GET /organizations/isodev/data/oracledbclusters/uach-d-uad-db HTTP/1.1" 200 "0.022" 237 "-" "Chef Client/12.8.1 (ruby-2.1.6-p336; ohai-8.11.1; x86_64-linux; +https://chef.io)" "127.0.0.1:8000" "200" "0.021" "12.8.1" "algorithm=sha1;version=1.1;" "hris-d-dhris-db1.campus.ksu.edu" "2017-08-24T22:24:37Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1145

10.139.40.1 - - [24/Aug/2017:17:24:50 -0500] "GET /organizations/isodev/data/oracledbclusters/ome-t-ometst-db HTTP/1.1" 200 "0.015" 249 "-" "Chef Client/12.20.3 (ruby-2.3.1-p112; ohai-8.23.0; x86_64-linux; +https://chef.io)" "127.0.0.1:8000" "200" "0.015" "12.20.3" "algorithm=sha1;version=1.1;" "iso-telc-p-db2.campus.ksu.edu" "2017-08-24T22:24:37Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1147

Our configuration is using elasticsearch 5.4.1 with filebeat 5.5.2. We were running the 5.4.1 version of filebeat, and had the same error (the hope was that an upgrade would fix the issue).

I open open to thoughts and suggestions.