I have a suspicion of what's going on here and it seems more related to Beats.
First I set Logstash config to:
output {
file {
path => "/tmp/logstash.log"
codec => rubydebug {
metadata => true
}
}
}
and then tail -F -n0 /tmp/logstash.log
The output of an event is as follows:
{
"input" => {
"type" => "log"
},
"agent" => {
"name" => "ip-REDACTED.REDACTED.compute.internal",
"id" => "af433b61-1a37-4916-94f7-0e7d36f84d97",
"type" => "filebeat",
"ephemeral_id" => "0574dc2d-f48b-4e5b-955a-1539e4212adf",
"version" => "8.5.3"
},
"@timestamp" => 2023-01-12T23:57:21.928Z,
"ecs" => {
"version" => "8.0.0"
},
"log" => {
"file" => {
"path" => "/var/log/secure"
},
"offset" => 36158
},
"service" => {
"type" => "system"
},
"@metadata" => {
"beat" => "filebeat",
"pipeline" => "filebeat-8.5.3-system-auth-pipeline",
"version" => "8.5.3",
"type" => "_doc"
},
"@version" => "1",
"host" => {
"name" => "ip-REDACTED.REDACTED.compute.internal"
},
"event" => {
"original" => "{\"@timestamp\":\"2023-01-12T23:57:21.928Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.5.3\",\"pipeline\":\"filebeat-8.5.3-system-auth-pipeline\"},\"fileset\":{\"name\":\"auth\"},\"service\":{\"type\":\"system\"},\"input\":{\"type\":\"log\"},\"ecs\":{\"version\":\"8.0.0\"},\"host\":{\"name\":\"ip-REDACTED.REDACTED.compute.internal\"},\"message\":\"Jan 12 23:57:17 ip-REDACTED sudo: pam_unix(sudo-i:session): session opened for user root by ec2-user(uid=0)\",\"event\":{\"dataset\":\"system.auth\",\"timezone\":\"+00:00\",\"module\":\"system\"},\"agent\":{\"name\":\"ip-REDACTED.REDACTED.compute.internal\",\"type\":\"filebeat\",\"version\":\"8.5.3\",\"ephemeral_id\":\"0574dc2d-f48b-4e5b-955a-1539e4212adf\",\"id\":\"af433b61-1a37-4916-94f7-0e7d36f84d97\"},\"log\":{\"offset\":36158,\"file\":{\"path\":\"/var/log/secure\"}}}",
"timezone" => "+00:00",
"module" => "system",
"dataset" => "system.auth"
},
"fileset" => {
"name" => "auth"
},
"message" => "Jan 12 23:57:17 ip-REDACTED sudo: pam_unix(sudo-i:session): session opened for user root by ec2-user(uid=0)"
}
now check out the grok: it attempts to grok event.original:
However, it should be grok'ing the message field as does its cousin in the same module, system.syslog:
In other words, it would appear Elasticsearch ingest pipeline filebeat-8.5.3-system-auth-pipeline is attempting to grok an escaped JSON documented rather than just the raw message.