hey!
I am currently using logstash-6.0.0. I have a microservices architecture where I store all my logs in /var/log/mservice/ and ops infrastrucutre (kibana, elasticsearch, logstash, grafana, etc) in /var/log/ops/
I have the following script which is not picking up anything anymore
##################################
# 1) INPUTS
##################################
input {
# Microservice Logs
file {
type => "json"
tags => ["json"]
path => [
"/var/log/mservices/*.log",
"/var/log/mservices/**/*.log"
]
}
# Elasticsearch Logs
file {
type => "json"
tags => ["elasticsearch"]
start_position => "beginning"
path => [
"/var/log/ops/elasticsearch/*.log"
]
}
# Logstash Logs
file {
type => "json"
tags => ["logstash"]
start_position => "beginning"
path => [
"/var/log/ops/logstash/*.log"
]
}
}
##################################
# 2) FILTERS
##################################
filter {
# Microservice filters
if "json" in [tags] {
json {
source => "message"
}
mutate {
uppercase => ["level"]
}
}
# Elasticsearch Filters
if "elasticsearch" in [tags] {
mutate {
add_field => { "service-name" => "elasticsearch" }
}
}
# logstash Filters
if "logstash" in [tags] {
mutate {
add_field => { "service-name" => "logstash" }
}
}
}
##################################
# 3) OUTPUTS
##################################
output {
elasticsearch {
hosts => "address"
}
}
I tried to filter by type, but then I started getting the complaint:
Rejecting mapping update to [json, elasticsearch] as the final mapping would have more than 1 type multiple types. I therefore tried to filter on tags but this doesn't work
Due to reasons outside of the scope of this post, I can't use beats either.
How can I achieve what I trying to achieve?