I am currently working on formatting the @timestamp in access logs for the apache2 module in Filebeat. Currently getting the following error message on kibana: "Provided Grok expressions do not match field value:"
My current ingest pipeline is the following:
{
"description": "Pipeline for parsing Apache2 access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \[%{HTTPDATE:apache2.access.time}\] "(?:%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}|-)?" %{NUMBER:apache2.access.response_code} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( "%{DATA:apache2.access.referrer}")?( "%{DATA:apache2.access.agent}")?",
"%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \[%{HTTPDATE:apache2.access.time}\] "-" %{NUMBER:apache2.access.response_code} -",
"\[%{HTTPDATE:apache2.access.time}\] %{IPORHOST:apache2.access.remote_ip} %{DATA:apache2.access.ssl.protocol} %{DATA:apache2.access.ssl.cipher} "%{WORD:http.request.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}" %{NUMBER:apache2.access.body_sent.bytes}"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "apache2.access.time",
"target_field": "@timestamp",
"formats": ["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"],
"ignore_failure": false
}
}, {
"remove": {
"field": "apache2.access.time",
"ignore_failure": true
}
}, {
"rename": {
"field": "apache2.access.agent",
"target_field": "apache2.access.user_agent.original",
"ignore_failure": true
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
It was working earlier before i added the milliseconds to the format. I am trying to format the @timestamp to be something like 2019-08-20T00:02:15.000Z. Is there are problem with the time format or something else in the pipeline.
Best Regards,
Juan Vicente