# Logstash filter not applying to all records **URL:** \ **Category:** Logstash\ **Created:** [September 18, 2017, 2:27pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886 "2017-09-18T14:27:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ananthasooner](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@ananthasooner](https://discuss.elastic.co/u/ananthasooner)\ **Post date:** [September 18, 2017, 2:27pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886/1 "2017-09-18T14:27:04Z")
For each entry, i see in kibana, some have i\_hostname value and some don't, even though beat.hostname is available in each entry. Same happening with idtest, i am aware that idtest is based on condition, but still for the same conditions, some times the value is extracted and show in kibana and some times it does not. Feels like for some reason, the filter is missing to apply on some entry randomly.I dont see any error in logstash log. Below is the logstash configuration. I just change the ip address to some random name. Any help would be useful. Thanks. `Preformatted text` input { beats { port =\> 5044 } } filter { if "id:" in [message] { grok { match =\> { "message" =\> "(?(._))" } } mutate {add\_field =\> {"idtest2"=\> "%{idtest}"}} } grok { match =\> { "message" =\> "(?(?\<=fileSize:)(._))" } } mutate { convert =\> {"filesizemb" =\> "float"} } mutate { add\_field =\> {"i\_hostname" =\> "%{[beat][hostname]}"} } } output { elasticsearch { hosts =\> ["hostip1:9200","hostip2:9200","hostip3:9200"] index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 18, 2017, 2:30pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886/2 "2017-09-18T14:30:03Z")
Please show an example event that exhibits this problem. Copy/paste from Kibana's JSON tab so we can see the raw event. ---
**Author:** ![ananthasooner](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@ananthasooner](https://discuss.elastic.co/u/ananthasooner)\ **Post date:** [September 18, 2017, 2:44pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886/3 "2017-09-18T14:44:20Z")
This line extracted correctly the i\_hostname indent preformatted text by 4 spaces { "\_index": "filebeat-2017.09.18", "\_type": "log", "\_id": "AV6VVgdLzp0Mera-TPrS", "\_version": 1, "\_score": null, "\_source": { "offset": 338864, "input\_type": "log", "source": "/var/log/ing/ing.log", "message": "INFO\t2017-09-18 09:13:11\tFileUploader\t[pool-1-thread-3]\tid:V714377", "type": "log", "tags": [ "beats\_input\_codec\_plain\_applied", "\_grokparsefailure" ], "idtest2": "INFO\t2017-09-18 09:13:11\tFileUploader\t[pool-1-thread-3]\tid:V714377", "idtest": "INFO\t2017-09-18 09:13:11\tFileUploader\t[pool-1-thread-3]\tid:V714377", "@timestamp": "2017-09-18T14:13:12.009Z", "@version": "1", "beat": { "hostname": "manager", "name": "manager", "version": "5.4.1" }, "host": "manager", "i\_hostname": "manager" }, "fields": { "@timestamp": [ 1505743992009 ] }, "highlight": { "source": [ "@kibana-highlighted-field@/var/log/ing/ing.log@/kibana-highlighted-field@" ] }, "sort": [ 1505743992009 ] } Below is the not worked case: indent preformatted text by 4 spaces { "\_index": "filebeat-2017.09.18", "\_type": "log", "\_id": "AV6VVmjvzp0Mera-TPxV", "\_version": 1, "\_score": null, "\_source": { "@timestamp": "2017-09-18T14:13:37.011Z", "offset": 341814, "@version": "1", "beat": { "hostname": "manager", "name": "manager", "version": "5.4.1" }, "input\_type": "log", "host": "manager", "source": "/var/log/ing/ing.log", "message": "INFO\t2017-09-18 09:13:36\tFileUploader\t[pool-1-thread-8]\tid:V714377", "type": "log", "tags": [ "beats\_input\_codec\_plain\_applied" ] }, "fields": { "@timestamp": [ 1505744017011 ] }, "highlight": { "source": [ "@kibana-highlighted-field@/var/log/ing/ing.log@/kibana-highlighted-field@" ] }, "sort": [ 1505744017011 ] } FYI. For some reason in the first message i posted: grok match regex is modified while posting, may be forum is ripping some text i guess.. FYI. I renamed some names in this json. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 18, 2017, 7:57pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886/4 "2017-09-18T19:57:34Z")
I don't know what's up here. My hunch is that the second event wasn't processed with the Logstash configuration you think you had. ---
**Author:** ![ananthasooner](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@ananthasooner](https://discuss.elastic.co/u/ananthasooner)\ **Post date:** [September 19, 2017, 3:44pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886/5 "2017-09-19T15:44:20Z")
Thank you very much for directing in right direction to look at. In our environment filebeats is load balanced to send to 3 logstash instance. I had this configuration in only 1 logstash instance. So it only applied for the events which was processed by the logstash which had this configuration. Sorry i wasn't initially aware of the ecosystem we had. Thanks for the quick help! This issue is closed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 17, 2017, 3:44pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886/6 "2017-10-17T15:44:28Z")
Closed on October 17, 2017, 3:44pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.