Winlogbeat and filebeat config can't wokr together


(kursad) #1

Dear Friends,

I already send my application and IIS logs to elasticsearch without any problem. Currently, i'm trying to send windows eventviwer logs to elasticsearch. But interestingly when I start to logstash, logs can not go to elasticsearch properly.

When i remove my iis logstash config file winlog beat works properly. Similarly, when i remove my winlogbeat config file filebeat works properly. But when i start to logstash both filebeat and winlogbeat configs logs can't go to elasticsearch properly.

Have you got any idea how can fix this problem. my config files are below

Filebeat Config:
indent preformatted text by 4 spaces
input {
beats {
port => 5044
}
}

output {
if "_grokparsefailure" in [tags] {
stdout {
codec => rubydebug
}
}
else {
elasticsearch {
index => "iis-%{+YYYY.MM.dd}"
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
}
}
}

winlogbeat config:

indent preformatted text by 4 spaces

input {
beats {
port => 5050
}
}
output {
if "_jsonparsefailure" in [tags] {
stdout { codec => rubydebug }
}
elseif [type] == "wineventlog" {
elasticsearch {
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
manage_template => false
template => "/etc/logstash/templates/winlogbeat.template.json"
template_overwrite => true
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
}
stdout { codec => rubydebug }
} else {
stdout { codec => rubydebug }
}
}

Json config:

indent preformatted text by 4 spaces

input {
udp {
port => 5960
codec => "json"
type => "json"
}
}
output {
if "_jsonparsefailure" in [tags] {
stdout { codec => rubydebug }
} elseif [type] == "json" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
template => "/etc/logstash/templates/template.json" }
} else {
stdout { codec => rubydebug }
}
}


(Magnus Bäck) #2

All configuration files in the same Logstash pipeline are concatenated so all events from all inputs reach all filters and all outputs. If this isn't desirable you need to use conditionals. Just using different files won't do it.


(kursad) #3

Thanks for your reply magnubaeck.

I merged all my output configurations in one file. Application and winlogbeat configuration working properly, but filebeat configuration is not working properly. I can't see iis logs in elasticsearch indexs.

My new output config file is below.

output {
if "_jsonparsefailure" in [tags] {
stdout { codec => rubydebug }
} elseif [type] == "iislog" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} elseif [type] == "wineventlog" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} elseif [type] == "json" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
template => "/etc/logstash/templates/template.json"
}
} else {
stdout { codec => rubydebug }
}
}


(Magnus Bäck) #4

Are you seeing them on stdout then? Are there any error messages in the Logstash log? Why have different elasticsearch output for iislog and wineventlog events?


(kursad) #5

Yes, They are seen on stdout. a sample is below. There is no error in logstash log files.

i can't solve the problem yet. After solving it , i'm going to use one eleastic config for iislog and wineventlog.

{
Sep 27 14:06:02 btkbnsrpg01 logstash: "http_response" => 200,
Sep 27 14:06:02 btkbnsrpg01 logstash: "referer" => "",
Sep 27 14:06:02 btkbnsrpg01 logstash: "useragent" => "Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/70.0.3538.27+Safari/537.36",
Sep 27 14:06:02 btkbnsrpg01 logstash: "bytes_received" => 1309,
Sep 27 14:06:02 btkbnsrpg01 logstash: "source" => "C:\inetpub\logs\LogFiles\W3SVC46\u_ex18092708_x.log",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_os_name" => "Linux",
Sep 27 14:06:02 btkbnsrpg01 logstash: "sc_status" => "0",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_patch" => "3538",
Sep 27 14:06:02 btkbnsrpg01 logstash: "protocol" => "HTTP/1.1",
Sep 27 14:06:02 btkbnsrpg01 logstash: "http_method" => "POST",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_major" => "70",
Sep 27 14:06:02 btkbnsrpg01 logstash: "@version" => "1",
Sep 27 14:06:02 btkbnsrpg01 logstash: "beat" => {
Sep 27 14:06:02 btkbnsrpg01 logstash: "name" => "BTWEBSRPG01",
Sep 27 14:06:02 btkbnsrpg01 logstash: "hostname" => "BTWEBSRPG01",
Sep 27 14:06:02 btkbnsrpg01 logstash: "version" => "6.2.4"
Sep 27 14:06:02 btkbnsrpg01 logstash: },
Sep 27 14:06:02 btkbnsrpg01 logstash: "host" => "BTWEBSRPG01",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_os" => "Linux",
Sep 27 14:06:02 btkbnsrpg01 logstash: "offset" => 5906031,
Sep 27 14:06:02 btkbnsrpg01 logstash: "User_Client_Ip" => "176.30.230.118",
Sep 27 14:06:02 btkbnsrpg01 logstash: "User_Client_Port" => "24297",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_minor" => "0",
Sep 27 14:06:02 btkbnsrpg01 logstash: "prospector" => {
Sep 27 14:06:02 btkbnsrpg01 logstash: "type" => "log"
Sep 27 14:06:02 btkbnsrpg01 logstash: },
Sep 27 14:06:02 btkbnsrpg01 logstash: "message" => "XXXXXXXXXXXXXXXX",
Sep 27 14:06:02 btkbnsrpg01 logstash: "sub_response" => 0,
Sep 27 14:06:02 btkbnsrpg01 logstash: "bytes_sent" => 370,
Sep 27 14:06:02 btkbnsrpg01 logstash: "time_taken" => 363,
Sep 27 14:06:02 btkbnsrpg01 logstash: "tags" => [
Sep 27 14:06:02 btkbnsrpg01 logstash: [0] "beats_input_codec_plain_applied"
Sep 27 14:06:02 btkbnsrpg01 logstash: ],
Sep 27 14:06:02 btkbnsrpg01 logstash: "vhost" => "xxxxxxxx",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_name" => "Chrome",
Sep 27 14:06:02 btkbnsrpg01 logstash: "site" => "10.2.1.52",
Sep 27 14:06:02 btkbnsrpg01 logstash: "@timestamp" => 2018-09-27T08:36:18.000Z,
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_build" => "",
Sep 27 14:06:02 btkbnsrpg01 logstash: "port" => "80",
Sep 27 14:06:02 btkbnsrpg01 logstash: "service" => "W3SVC46",
Sep 27 14:06:02 btkbnsrpg01 logstash: "received_at" => "2018-09-27T11:04:53.919Z",
Sep 27 14:06:02 btkbnsrpg01 logstash: "browser_device" => "Other",
Sep 27 14:06:02 btkbnsrpg01 logstash: "page" => "/ServiceOim/CancelServiceFreezed",
Sep 27 14:06:02 btkbnsrpg01 logstash: "client_host" => "193.192.122.30",
Sep 27 14:06:02 btkbnsrpg01 logstash: "query_string" => "X-ARR-LOG-ID=f24445eb-85e3-49fd-8004-28b6832c276c",
Sep 27 14:06:02 btkbnsrpg01 logstash: "username" => "-"
Sep 27 14:06:02 btkbnsrpg01 logstash: }


(Magnus Bäck) #6

The event you just posted doesn't have "iislog" in the type field.


(kursad) #7

filebeat.yml conf is below.

  • type: log
  • document_type: iislog

1- When i change the type from log to iis. filebeat service can't start and give error. ( Error : instance/beat.go:667 Exiting: Error in initing prospector: Error creating prospector. No such prospector type exist: 'iislog )
'
2- when i change my output's if condition as below. There's still no data in elasticsearch.

output {
if "_jsonparsefailure" in [tags] {
stdout { codec => rubydebug }
} elseif [type] == "log" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} elseif [type] == "wineventlog" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} elseif [type] == "json" {
elasticsearch {
hosts => ["10.2.1.148:9200","10.2.1.149:9200","10.2.1.152:9200"]
template => "/etc/logstash/templates/template.json"
}
} else {
stdout { codec => rubydebug }
}
}


(Magnus Bäck) #8

document_type is deprecated, see https://www.elastic.co/guide/en/beats/libbeat/master/breaking-changes-6.0.html#breaking-changes-types. Use a field to set type to the desired value.


(kursad) #9

thanks magnusbaeck.. After checking filebeat 6 release breaking check , i change my output config as below. It works properly

elseif [fields][document_type] == "iislog"


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.