Hi, I prepare this question and also find solution after few hours, so I decide to upload question and answer, maybe it will help somebody:
QUESTION:
I have Filebeat in k8s that sends logs to Logstash. I have namespace apps-special. In my ES, I create rollover index with name rollover-special-000001 and alias special. This is my Logstash pipeline filter:
filter {
if [kubernetes][namespace] in ["apps-special"] {
mutate {
add_field => { "[@metadata][es-index]" => "special" }
}
} else if [kubernetes][pod][name] {
mutate {
add_field => { "[@metadata][es-index]" => "default-%{[kubernetes][pod][name]}-%{+xxxx-ww}" }
}
}
}
Expectation:
I would expect all pods from namespace apps-special will be in one index named rollover-special-000001 (which will rollover after time/conditions). Logs from all other namespaces shoud be created as <name-of-the-pod>-<YEAR>-<WEEK>.
Result:
I have many indices with <name-of-the-pod>-<YEAR>-<WEEK> pattern. I have no idea why the if statement did not match my namespace.
My investgation:
I have other namespaces where the match works perfectly. I also check, if kubernetes metadata in messages has proper namespace. So from output:
GET /<index-name>/_search
I grep the namespace:
jq '.hits.hits[]."_source".kubernetes.namespace' | sort | uniq
"apps-special"
What i find out, my if statements with other namespaces works fine, but the list for matching indices is grater than 1.
[kubernetes][namespace] in ["namespace-1", "namespace-2", "namespace-3"]
Conclusion:
I think, I have the right logic in Logstash filter, but I have no idea why is Logstash unable to process it in right way. From my investigation, it looks like the array must be >= 2 but I am not sure. Am I missing something?
Versions:
Filebeat: v7.9.3
Logstash: v7.8.1
ANSWER:
It looks like if-in statement does not work if array has only one object (array.length = 1). When I duplicate my apps-special it starts working properly, so after my change, my filter looks like this:
filter {
if [kubernetes][namespace] in ["apps-special", "apps-special"] {
mutate {
add_field => { "[@metadata][es-index]" => "special" }
}
} else if [kubernetes][pod][name] {
mutate {
add_field => { "[@metadata][es-index]" => "default-%{[kubernetes][pod][name]}-%{+xxxx-ww}" }
}
}
}
My sources was:
- Logstash config: conditional with list not working if [field] in ["list item 1"] - Stack Overflow
if .. in []doesn't match for single-element arrays
I was not able to find it in documentation. I hope this will help somebody to save a few minutes/hours.