Grok debugger says good but logstash doesn't match

[Willl]

Hi,

Running filebeat with system module on /var/log/message. I tested the following in Kibana Grok Debugger,

Log:

Jan 2 14:42:52 dev10 sudo: pam_unix(sudo:session): session opened for user root by root(uid=0)

with Pattern:

%{SYSLOGBASE2} %{GREEDYDATA:message}

Debugger returns Structured data:

{
"program": "sudo",
"logsource": "dev10",
"message": "pam_unix(sudo:session): session opened for user root by root(uid=0)",
"timestamp": "Jan 2 14:42:52"
}

In Logstash, I have filter:

input {
beats {
port => 5044
}
}
filter {
if [ dataset ] == "system.syslog" {
grok {
match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}
}
}
output {
stdout { codec => rubydebug }
}

I kicked off Logstash and I don't get the message I expected, i.e. just the greedy data part.

{
"event" => {
"module" => "system",
"dataset" => "system.syslog",
"timezone" => "-05:00"
},
"message" => "Jan 2 14:42:52 dev10 sudo: pam_unix(sudo:session): session opened for user root by root(uid=0)",
"log" => {
"offset" => 3529252,
"file" => {
"path" => "/var/log/messages"
}
},
"fileset" => {
"name" => "syslog"
},
"input" => {
"type" => "log"
},
"ecs" => {
"version" => "1.1.0"
},
"@version" => "1",
"host" => {
"hostname" => "dev10",
"os" => {
"name" => "SLES",
"platform" => "sles",
"family" => "suse",
"version" => "12-SP4",
"kernel" => "4.12.14-94.41-default"
},
"name" => "dev10",
"architecture" => "x86_64",
"id" => "d976befa1d66e6bd04192c375dcc7785",
"containerized" => false
},
"service" => {
"type" => "system"
},
"@timestamp" => 2020-01-02T19:42:54.599Z,
"agent" => {
"hostname" => "dev10",
"ephemeral_id" => "c5e4d281-0cc8-43b2-a25a-c9831c0e9d25",
"version" => "7.5.1",
"type" => "filebeat",
"id" => "1bbe9716-9685-4270-be07-c32d0b1776c2"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}

[Willl]

@Willl

overwrite => [ "message" ]

Removed the "overwrite" and the output looks better. Now when I go to Kibana/syslog dashboard. I get shards failed.

"reason": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [host.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}

[Willl]

I sorted this out. Thanks.
I ran curl to enabled field data.