For each entry, i see in kibana, some have i_hostname value and some don't, even though beat.hostname is available in each entry. Same happening with idtest, i am aware that idtest is based on condition, but still for the same conditions, some times the value is extracted and show in kibana and some times it does not. Feels like for some reason, the filter is missing to apply on some entry randomly.I dont see any error in logstash log. Below is the logstash configuration. I just change the ip address to some random name. Any help would be useful. Thanks.
Preformatted text
input {
beats {
port => 5044
}
}
filter {
if "id:" in [message] {
grok {
match => { "message" => "(?(.))" }
}
mutate {add_field => {"idtest2"=> "%{idtest}"}}
}
grok {
match => { "message" => "(?(?<=fileSize:)(.))" }
}
mutate { convert => {"filesizemb" => "float"} }
mutate { add_field => {"i_hostname" => "%{[beat][hostname]}"} }
}
output {
elasticsearch {
hosts => ["hostip1:9200","hostip2:9200","hostip3:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}