I have two servers configured with Filebeat to send logs to Logstash which is working properly. I would now like to view each of those logs separately under their respective indexes within Kibana Discover but I am struggling to get it working. Data for both servers is not being separated and is appearing in both indexes.
Filebeat + Logstash config below:
Server #1:
/etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/*.log
exclude_files: ['\.gz$']
processors:
- add_fields:
target: ''
fields:
server: 'server1'
output.logstash:
hosts: ["elasticserver:5400"]
ssl.certificate_authorities: ["/etc/elk-certs/elk-ssl.crt"]
ssl.certificate: "/etc/elk-certs/elk-ssl.crt"
ssl.key: "/etc/elk-certs/elk-ssl.key"
/etc/logstash/conf.d/logstash-server1.conf
input {
beats {
port => 5400
ssl => true
ssl_certificate_authorities => ["/etc/elk-certs/elk-ssl.crt"]
ssl_certificate => "/etc/elk-certs/elk-ssl.crt"
ssl_key => "/etc/elk-certs/elk-ssl.key"
ssl_verify_mode => "force_peer"
}
}
filter {
grok {
match => [ "message" , "%{IPORHOST:clientip} - %{USER:user} - \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:loadtime} %{QS:referrer} %{QS:agent} \"-\""]
overwrite => [ "message" ]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "weblogs-%{+YYYY.MM.dd}"
user => "logstash_internal"
password => "my_password"
document_type => "nginx_logs"
}
stdout { codec => rubydebug }
}
Server #2:
/etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
paths:
- /var/www/laravel/storage/logs/laravel.log
fields:
document_type: laravel_logs
output.logstash:
hosts: ["elasticserver:5444"]
ssl.certificate_authorities: ["/etc/elk-certs/elk-ssl.crt"]
ssl.certificate: "/etc/elk-certs/elk-ssl.crt"
ssl.key: "/etc/elk-certs/elk-ssl.key"
Filebeat log file sample:
2021-09-27T17:12:11.774Z INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":12740766}},"memory":{"mem":{"usage":{"bytes":4096}}}},"cpu":{"system":{"ticks":220,"time":{"ms":4}},"total":{"ticks":720,"time":{"ms":12},"value":720},"user":{"ticks":500,"time":{"ms":8}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"d069173c-0430-49ff-8054-b7ac7ed0336b","uptime":{"ms":1590092},"version":"7.15.0"},"memstats":{"gc_next":19401488,"memory_alloc":9722072,"memory_total":88706744,"rss":111407104},"runtime":{"goroutines":24}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.31,"15":0.3,"5":0.31,"norm":{"1":0.155,"15":0.15,"5":0.155}}}}}}
/etc/logstash/conf.d/logstash-server2.conf
input {
beats {
port => 5444
ssl => true
ssl_certificate_authorities => ["/etc/elk-certs/elk-ssl.crt"]
ssl_certificate => "/etc/elk-certs/elk-ssl.crt"
ssl_key => "/etc/elk-certs/elk-ssl.key"
ssl_verify_mode => "force_peer"
}
}
filter {
if [fields][document_type] == "laravel_logs" {
grok {
match => [ "message" , "\[%{TIMESTAMP_ISO8601:timestamp}\] (?<system>[\w\.]*): %{WORD:clickid} - %{GREEDYDATA:message}"]
overwrite => [ "message" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "laravel-logs-%{+YYYY.MM.dd}"
user => "logstash_internal"
password => "my_password"
document_type => "laravel_logs"
}
stdout { codec => rubydebug }
}
When in Kibana Discover and looking at my different indexes (weblogs-* and laravel-logs-*), I get data in both of those indexes but server1 data appears in both indexes when I only want server1 to index to weblogs- and server2 to index to laravel-logs-
Can someone please explain to me where I am going wrong and how to resolve this?
Many thanks,
milo
