Output data in multiple indices based on fields


(Harsh Verma) #1

I have a logfile with some special logs and some general logs.I want the general logs to go in a general index and the special logs(logs which match any of my grok patterns) to go in a separate index and in the general index as well.
My pipeline conf looks like this:

input {
beats {
port => "5043"
}
}
filter {
grok {
match => [ "message", "PATTERN1",
"message", "PATTERN2"
]
}
if "_grokparsefailure" in [tags] {
mutate {
add_field => {"[@metadata][index]" => "generallogs"}
}
}
else{
mutate {
add_field => {"[@metadata][index]" => "speciallogs"}
}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{[@metadata][index]}"
}
}

The only problem with this is that special logs go only in the special log index and not in the general log index. Is there a solution ? Thanks !


#2

Make the output to the other index conditional using something like:

output {

elasticsearch {
hosts => [ "localhost:9200" ]
index => "generallogs"
}

if "_grokparsefailure" not in [tags] {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "speciallogs"
}
}

}

(Harsh Verma) #3

Makes perfect sense. Don't know why I was not able to think about this !


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.