Hi Team,
I have application running on 2 servers and application logs are getting logged on both the servers, so i want to parse logs from both servers.
filebeat is installed on two application servers,
logstash is installed on separate two servers and
elasticsearch is installed on three servers (2 of which are logstash servers also)
filebeat.yml is like below,
Application Server1 -
filebeat.inputs:
- type: log
fields_under_root: true
fields:
log_type: federate_server1
app_id: pf
multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
multiline.negate: true
multiline.match: after
paths:
- /opt/federate-0.2.0/federate/log/*
output.logstash:
hosts: ['logstash1:5044'], ['logstash2:5044']
loadbalance: true
Application Server 2 -
filebeat.inputs:
- type: log
fields_under_root: true
fields:
log_type: federate_server2
app_id: pf
multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
multiline.negate: true
multiline.match: after
paths:
- /opt/federate-0.2.0/federate/log/*
output.logstash:
hosts: ['logstash1:5044'], ['logstash2:5044']
loadbalance: true
logstash.yml -
logstash server1
input {
beats {
port => 5044
}
}
filter {
if [log_type] == "federate_server" and [app_id] == "pf"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{WORD:TraceID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "federate_server"
}
}
}
output {
if [log_type] == "federate_server" {
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
user => elastic
password => "${es_pwd}"
index => "federate"
template_name => "federate"
template_overwrite => "false"
}
}
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
user => elastic
password => "${es_pwd}"
}
}
logstash server 2
input {
beats {
port => 5044
}
}
filter {
if [log_type] == "federate_server" and [app_id] == "pf"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{WORD:TraceID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "federate_server"
}
}
}
output {
if [log_type] == "federate_server" {
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
user => elastic
password => "${es_pwd}"
index => "federate"
template_name => "federate"
template_overwrite => "false"
}
}
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
user => elastic
password => "${es_pwd}"
}
}
I know currently log_type in both filebeat.yml is not matching with log_type in both logstash.yml file.
Since both the logstash are mentioned in filebeat.yml and loadbalance is true so filebeat will send events either of the logstash servers at a time but two receive events on logstash end how can i add the other log_type in logstash.yml?. currently only one is specified.
- i.e Can I change
log_typeas below on both server'slogstash.ymlto receive events from both the application server'sfilebeat?
filter {
if [log_type] == "federate_server1" or if [log_type] == "federate_server2" and [app_id] == "pf"
output {
if [log_type] == "federate_server" or if [log_type] == "federate_server2" {
Elasticsearch {
Is the above or condition correct? if yes, what will come at below [type] => line
mutate {
replace => {
"[type]" => "federate_server"
}
}
I just want to parse logs from both the application servers which will be send by filebeat to any logstash server but this currently above config seems to be incorrect as only one log_type will matched as only one if condition is mentioned.
- Do we need to mentioned all es hosts in
outputsection (like above its mentioned3 eshosts or only one is enough and that will forward the requests to other two es nodes in cluster)
Thanks,