I have one logstash configuration file which I am using for logs coming from both Windows and linux environment. I have the same filebeat configuration for both environment. The log file being shipped via filebeats is also same.
Still, the grok filters are not filtering fields on logs coming from windows, whereas they are being filtered for the same log file from linux.
I added an exact same log line in my log file, on both windows and linux environment to compare the JSON output to see why grok filter is working on linux but not on windows??
Below is my filebeat.yml from windows-
filebeat.inputs:
- type: log
enabled: true
input_type: log
fields:
tags: ["windows"]
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'C:\Program Files (x86)\ESQ SST\Logs\ngta-common.log'
#multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
#multiline.negate: true
#multiline.match: after
output.logstash:
# The Logstash hosts
hosts: ["xxx.xxx.xx.xx:5044"]
logging.level: debug
logging.to_files: true
logging.files:
path: D:\filebeat\logs
name: filebeat.log
keepfiles: 7
Below is my filebeat.yml from linux-
filebeat.inputs:
- type: log
enabled: true
input_type: log
paths:
- /home/mehak/Documents/filebeat-7.5.2-linux-x86_64/logs/ngta-common.log
fields:
tags: linux
Below is my filebeat.log from windows-
#filebeat from windows with same log line
2020-03-13T14:42:03.459-0700 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-03-13T21:42:03.458Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.5.2"
},
"agent": {
"hostname": "GlobalDemo",
"id": "980ab6dc-bb04-4b43-abf1-32348b8090a6",
"version": "7.5.2",
"type": "filebeat",
"ephemeral_id": "84e006db-c202-41ae-80b9-27a230b265a0"
},
"log": {
"offset": 2994,
"file": {
"path": "C:\\Program Files (x86)\\ESQ SST\\Logs\\ngta-common.log"
}
},
"message": "2020-09-09 09:09:09,700 [dcbh478642637-76487] DEBUG c.e.n.c.r.LogEvent - Same log message",
"input": {
"type": "log"
},
"fields": {
"tags": [
"windows"
]
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "GlobalDemo"
}
}
Below is my filebeat.log from linux-
#filebeat from linux with same log line
2020-03-13T14:43:11.016-0700 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-03-13T21:43:11.016Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.5.2"
},
"fields": {
"tags": "linux"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "mehak-VirtualBox"
},
"agent": {
"type": "filebeat",
"ephemeral_id": "ab7a7bdd-e852-430d-9574-3e42edbdcaaf",
"hostname": "mehak-VirtualBox",
"id": "26a23073-4272-498d-88ca-892985226e9f",
"version": "7.5.2"
},
"log": {
"offset": 6593,
"file": {
"path": "/home/mehak/Documents/filebeat-7.5.2-linux-x86_64/logs/ngta-common.log"
}
},
"message": "2020-09-09 09:09:09,700 [dcbh478642637-76487] DEBUG c.e.n.c.r.LogEvent - Same log message",
"input": {
"type": "log"
}
}
Below is my logstash stdout when Logstash sends to Elasticsearch for linux logs where we can see log message, javaClass, log level, time stamp fields being extracted-
#logstash from linux with same log line
[2020-03-13T14:43:12,215][DEBUG][logstash.filters.grok ][test] Event now: {:event=>#<LogStash::Event:0x297b0f99>}
{
"timestamp" => "2020-09-09 09:09:09,700",
"logmessage" => " - Same log",
"message" => "2020-09-09 09:09:09,700 [dcbh478642637-76487] DEBUG c.e.n.c.r.LogEvent - Same log message",
"input" => {
"type" => "log"
},
"agent" => {
"type" => "filebeat",
"ephemeral_id" => "ab7a7bdd-e852-430d-9574-3e42edbdcaaf",
"version" => "7.5.2",
"hostname" => "mehak-VirtualBox",
"id" => "26a23073-4272-498d-88ca-892985226e9f"
},
"log" => {
"file" => {
"path" => "/home/mehak/Documents/filebeat-7.5.2-linux-x86_64/logs/ngta-common.log"
},
"offset" => 6593
},
"fields" => {
"tags" => "linux"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"javaClass" => "c.e.n.c.r.LogEvent",
"loglevel" => "DEBUG",
"@version" => "1",
"@timestamp" => 2020-03-13T21:43:11.016Z,
"ecs" => {
"version" => "1.1.0"
},
"host" => {
"name" => "mehak-VirtualBox"
}
}
Below is my logstash stdout when Logstash sends to Elasticsearch for windows logs. Here we can see neither of the fields Such as log level, log message or javaClass are extracted like they are in Linux above -
#logstash from windows with same log line
{
"fields" => {
"tags" => [
[0] "windows"
]
},
"ecs" => {
"version" => "1.1.0"
},
"host" => {
"name" => "GlobalDemo"
},
"agent" => {
"version" => "7.5.2",
"ephemeral_id" => "84e006db-c202-41ae-80b9-27a230b265a0",
"type" => "filebeat",
"hostname" => "GlobalDemo",
"id" => "980ab6dc-bb04-4b43-abf1-32348b8090a6"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@version" => "1",
"message" => "2020-09-09 09:09:09,700 [dcbh478642637-76487] DEBUG c.e.n.c.r.LogEvent - Same log message",
"@timestamp" => 2020-03-13T21:42:03.458Z,
"input" => {
"type" => "log"
},
"log" => {
"file" => {
"path" => "C:\\Program Files (x86)\\ESQ SST\\Logs\\ngta-common.log"
},
"offset" => 2994
}
}
And lastly, my logstash config file-
#listening on this port
input {
beats {
port => 5044
}
}
filter {
if [fields][tags] == "linux" {
#mutate {
#add_field => { "host" => "%{[event_data][IpAddress]}" }
#}
grok{
match => {
"message" => [
"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{GREEDYDATA}\]%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{JAVACLASS:javaClass}%{GREEDYDATA:logmessage} "
]
add_tag => ["_grokparsefailure"]
}
}
}if [fields][tags] == "windows" {
#mutate {
#add_field => { "host" => "%{[event_data][IpAddress]}" }
#}
grok{
match => {
"message" => [
"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{GREEDYDATA}\]%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{JAVACLASS:javaClass}%{GREEDYDATA:logmessage} "
]
add_tag => ["_grokparsefailure"]
}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "%{[fields][tags]}"
}
stdout {
codec => rubydebug
}
}
Why are fields such as logmessage, log level not being extracted from windows logs whereas they are being extracted red from Linux as seen above?