Hello,
I have a weird situation where the data is correctly parsed, grok debugger doesn't return errors. But logstash still adds a grokeparse_failure tag.
Here is an exemple log (in /var/log/haproxy.log) :
Mar 1 15:40:24 example-host-01 haproxy[1944764]: 200.100.200.100:42378 [01/Mar/2023:15:40:24.589] www.example.com example/example-host-01 0/0/0/1/1 404 415 - - ---- 27/27/8/8/0 0/0 {12.34.45.56} "GET /serviceworker.js?v=1677679200 HTTP/1.1"
My logstash grok configuration :
if "haproxy" in [tags] {
grok {
#### match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{DATA:aloha_name} haproxy\[%{DATA:haproxy_process}\]: %{DATA:client_ip}:%{DATA:client_port} \[%{DATA:accept_date}\] %{DATA:frontend_name} %{DATA:backend_name}/%{DATA:server_name} %{DATA:Tq}/%{DATA:Tw}/%{DATA:Tc}/%{DATA:Tr}/%{DATA:Tt} %{DATA:http_status_code} %{DATA:bytes_read} %{DATA:request_cookie} %{DATA:response_cookie} --%{DATA:termination_state} %{DATA:actconn}/%{DATA:feconn}/%{DATA:beconn}/%{DATA:srvconn}/%{DATA:retries} %{DATA:srv_queue}/%{DATA:backend_queue} {%{DATA:remote_ip_original}} \"%{DATA:method} %{DATA:request} HTTP/%{DATA:http_version}\""}
match => { "message" => "%{DATA:all-message}$"}
remove_field => [ "message" ]
}
date {
match => [ "timestamp" , "MMM dd HH:mm:ss", "MMM d HH:mm:ss"]
target => "@timestamp"
remove_field => "timestamp"
}
}
I commented the original filter but every field is correctly provided and the same behaviour is observed
When I log the groke parse failures on logstash service, I can see this logs :
{
"@version": "1",
"input": {
"type": "log"
},
"host": {
"name": "example-host-01"
},
"agent": {
"ephemeral_id": "13f92298-1b05-49a2-a3b9-75564cba9cff",
"type": "filebeat",
"version": "8.5.2",
"id": "79d545fb-637c-45c8-88f8-0cb6a0634195",
"name": "example-host-01"
},
"event": {
"original": "Mar 1 16:37:02 example-host-01 haproxy[1944764]: 100.200.200.100:41388 [01/Mar/2023:16:37:02.493] www.example.com example/example-host-01 0/0/0/1/1 404 413 - - ---- 57/57/2/2/0 0/0 {12.34.45.56} \"GET /serviceworker.js?v=1677682800 HTTP/1.1\""
},
"ecs": {
"version": "8.0.0"
},
"log": {
"offset": 238383290,
"file": {
"path": "/var/log/haproxy.log"
}
},
"tags": "example-haproxy-_grokparsefailure",
"@timestamp": "2023-03-01T15:37:02.630Z",
"all-message": "Mar 1 16:37:02 example-host-01 haproxy[1944764]: 100.200.100.200:41388 [01/Mar/2023:16:37:02.493] www.example.com example/example-host-01 0/0/0/1/1 404 413 - - ---- 57/57/2/2/0 0/0 {12.34.45.56} \"GET /serviceworker.js?v=1677682800 HTTP/1.1\""
}
On elastic, I can see all fields well provided, so everything is working normally except that the indices have that _grokparsefailure tag added. Every log has this behaviour
Am I doing something wrong ? I noticed how the logstash is adding \ elements in front of double quotes but I'm a bit clueless here.
Thank you in advance