Hi,
I am using ELK GA 6. In my logstash, I am taking inputs from 3 kafka topics and I perform different filters and outputs based on the kafka topic. Below is a small snippet of how I am doing it now;
input {
kafka{
group_id => "group_1"
topics => ["topic_1"]
add_field => { "logtype" => "logtype_1" }
.
.
.
}
kafka{
group_id => "group_2"
topics => ["topic_2"]
add_field => { "logtype" => "logtype_2" }
.
.
.
}
kafka{
group_id => "group_3"
topics => ["topic_3"]
add_field => { "logtype" => "logtype_3" }
.
.
.
}
}
filter {
if[logtype] == "logtype_1"{
grok {
# grok 1 logic
}
mutate {
# mutate 1 logic
}
}
if[logtype] == "logtype_2"{
grok {
# grok 2 logic
}
mutate {
# mutate 2 logic
}
}
if[logtype] == "logtype_3"{
grok {
# grok 3 logic
}
mutate {
# mutate 3 logic
}
}
}
output{
if[logtype] == "logtype_1"{
elasticsearch {
index => "myIndex1"
.
.
}
}
if[logtype] == "logtype_2"{
elasticsearch {
index => "myIndex2"
.
.
}
}
if[logtype] == "logtype_3"{
elasticsearch {
index => "myIndex3"
.
.
}
}
}
This works fine, but will create an extra attribute logtype. I dont want that to be inserted into my Elasticsearch. Is there any way to do this?
Thank you.