My Grok filter is not working in logstash Even though it is working in grok debugger
below is the grok string:
\((?<timestamp>%{DAY} %{MONTH} %{MONTHNUM} %{TIME} %{YEAR})\) \[(?<daemon>(.*))\] \[%{DATA:function}\] \(%{DATA:log_level}\): %{GREEDYDATA:message}
Below is the Log file output snippet
(Mon Nov 9 12:08:56 2020) [sssd[nss]] [client_recv] (0x0200): Client disconnected!
(Mon Nov 9 12:08:56 2020) [sssd[nss]] [client_close_fn] (0x2000): Terminated client [0x55ffd29d93c0][22]
Filebeat.yml
---
fields:
test_role:
- test_bastion
deployment: dev
server: bastion
logging.level: warning
logging.to_files: false
logging.to_syslog: true
fields_under_root: true
filebeat.inputs:
- paths:
- "/var/log/sssd/sssd_nss.log"
fields_under_root: true
fields:
category: secure
type: sssd
output.logstash:
hosts:
- logstash_IP:5022
logstash.conf
input {
beats {
port => 5022
}
}
filter {
if [type] == "sssd" {
grok {
match => {
"message" => "\((?<event_timestamp>%{DAY} %{MONTH} %{MONTHNUM} %{TIME} %{YEAR})\) \[(?<daemon>(.*))\] \[%{DATA:function}\] \(%{DATA:log_level}\): %{GREEDYDATA:message}"
remove_field => ["message"]
}
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "sssd"
}
}
The Output in Kibana
{
"took" : 985,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 815,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "sssd",
"_type" : "_doc",
"_id" : "jiOd5HUBzEbIbhAA0iOs",
"_score" : 1.0,
"_source" : {
"agent" : {
"version" : "7.6.0",
"type" : "filebeat",
"ephemeral_id" : "7134eef3-8c12-44ec-9950-4328398c5148",
"hostname" : "bastion",
"id" : "2adca8f1-3849-4ac9-a035-145a3e87c244"
},
"message" : "(2020-11-20 7:45:27): [nss] [sss_cmd_get_version] (0x0200): Received client version [1].",
"ecs" : {
"version" : "1.4.0"
},
"category" : "secure",
"log" : {
"file" : {
"path" : "/var/log/sssd/sssd_nss.log"
},
"offset" : 6565192
},
"@version" : "1",
"deployment" : "dev",
"host" : {
"name" : "bastion"
},
"test_role" : [
"test_bastion"
],
"type" : "sssd",
"tags" : [
"beats_input_codec_plain_applied",
"_grokparsefailure"
],
"@timestamp" : "2020-11-20T07:45:27.836Z",
"server" : "bastion"
}
}