Hi, I'm planning to convert to a new logstash config to remodel my indexing and i added an if/if else/else filter to logstash. Right now the cluster has around 85 active indices which roll daily. Almost half of it will be monthly, around 25-30 of it will be weekly and the rest will be inidexed daily. I tested the conf on a test environment and i didnt encounter much problems. My question is will i experience significant performance drop after i apply this to that much indices. As a result i'll have lots of "or" statements in the if/else filter. What downsides will that have? Will it cause the logs to experience latency or worse data loss? Logstash jvm is 4gb at the moment. The logstash filter looks like this but with the number of variables i mentioned above:
filter {
if [tag] == "x" or [tag] == "y" {
mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+YYYY.MM.dd}" } }
} else if [tag] == "z" or [tag] == "t" {
mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+YYYY.MM}" } }
} else {
mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+xxxx.ww}" } }
}
}
The indexing might change to adjust the number of given variables. For example monthly indices will be much more than the other 2 so monthly filter might just become the else condition.