Logstash filter not applying to all records


#1

For each entry, i see in kibana, some have i_hostname value and some don't, even though beat.hostname is available in each entry. Same happening with idtest, i am aware that idtest is based on condition, but still for the same conditions, some times the value is extracted and show in kibana and some times it does not. Feels like for some reason, the filter is missing to apply on some entry randomly.I dont see any error in logstash log. Below is the logstash configuration. I just change the ip address to some random name. Any help would be useful. Thanks.
Preformatted text
input {
beats {
port => 5044
}
}

filter {
if "id:" in [message] {
grok {
match => { "message" => "(?(.))" }
}
mutate {add_field => {"idtest2"=> "%{idtest}"}}
}
grok {
match => { "message" => "(?(?<=fileSize:)(.
))" }
}

mutate { convert => {"filesizemb" => "float"} }
mutate { add_field => {"i_hostname" => "%{[beat][hostname]}"} }
}

output {
elasticsearch {
hosts => ["hostip1:9200","hostip2:9200","hostip3:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}


(Magnus Bäck) #2

Please show an example event that exhibits this problem. Copy/paste from Kibana's JSON tab so we can see the raw event.


#3

This line extracted correctly the i_hostname
indent preformatted text by 4 spaces
{
"_index": "filebeat-2017.09.18",
"_type": "log",
"_id": "AV6VVgdLzp0Mera-TPrS",
"_version": 1,
"_score": null,
"_source": {
"offset": 338864,
"input_type": "log",
"source": "/var/log/ing/ing.log",
"message": "INFO\t2017-09-18 09:13:11\tFileUploader\t[pool-1-thread-3]\tid:V714377",
"type": "log",
"tags": [
"beats_input_codec_plain_applied",
"_grokparsefailure"
],
"idtest2": "INFO\t2017-09-18 09:13:11\tFileUploader\t[pool-1-thread-3]\tid:V714377",
"idtest": "INFO\t2017-09-18 09:13:11\tFileUploader\t[pool-1-thread-3]\tid:V714377",
"@timestamp": "2017-09-18T14:13:12.009Z",
"@version": "1",
"beat": {
"hostname": "manager",
"name": "manager",
"version": "5.4.1"
},
"host": "manager",
"i_hostname": "manager"
},
"fields": {
"@timestamp": [
1505743992009
]
},
"highlight": {
"source": [
"@kibana-highlighted-field@/var/log/ing/ing.log@/kibana-highlighted-field@"
]
},
"sort": [
1505743992009
]
}

Below is the not worked case:
indent preformatted text by 4 spaces

{
"_index": "filebeat-2017.09.18",
"_type": "log",
"_id": "AV6VVmjvzp0Mera-TPxV",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2017-09-18T14:13:37.011Z",
"offset": 341814,
"@version": "1",
"beat": {
"hostname": "manager",
"name": "manager",
"version": "5.4.1"
},
"input_type": "log",
"host": "manager",
"source": "/var/log/ing/ing.log",
"message": "INFO\t2017-09-18 09:13:36\tFileUploader\t[pool-1-thread-8]\tid:V714377",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
]
},
"fields": {
"@timestamp": [
1505744017011
]
},
"highlight": {
"source": [
"@kibana-highlighted-field@/var/log/ing/ing.log@/kibana-highlighted-field@"
]
},
"sort": [
1505744017011
]
}

FYI. For some reason in the first message i posted: grok match regex is modified while posting, may be forum is ripping some text i guess..

FYI. I renamed some names in this json.


(Magnus Bäck) #4

I don't know what's up here. My hunch is that the second event wasn't processed with the Logstash configuration you think you had.


#5

Thank you very much for directing in right direction to look at. In our environment filebeats is load balanced to send to 3 logstash instance. I had this configuration in only 1 logstash instance. So it only applied for the events which was processed by the logstash which had this configuration.

Sorry i wasn't initially aware of the ecosystem we had. Thanks for the quick help!

This issue is closed.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.