# Using drop to filter messages **URL:** \ **Category:** Logstash\ **Created:** [June 1, 2022, 6:19pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158 "2022-06-01T18:19:30Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [June 1, 2022, 6:19pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/1 "2022-06-01T18:19:30Z")
Hi All, I am using grok filter to parse messages coming into Logstash from filebeat. We have ELK 7.6.2 stack. I need to filter out and process "only" the following message(s) in the gc log as follows. Note that pretty much every value is a variable: ```auto [2022-06-01T16:47:10.415+0000][info][gc] GC(15217) Pause Full (Diagnostic Command) 2178M->1036M(2560M) 1134.387ms ``` Please guide on what regex should I use to make the following work. So far I am able to do this: ```auto filter { if [type] == "tv_gclog_analysis" { grok { match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}' } } if ([message] !~ "Full") { drop { } } } } ``` This does not work.. If I take out the drop part above then I do see all messages coming in and getting indexed. Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2022, 6:34pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/2 "2022-06-01T18:34:59Z")
> [@zaeemmasood](#): > > This does not work.. Indeed. If remove the drop {} and look at the event you will see that [message] is an array, because you take a field called [message] and use grok to extract a field called [message] from it, so you end up with ``` "message" => [ [0] "[2022-05-24T02:15:20.979+0000][info][gc] GC(187) Pause Full (G1 Evacuation Pause) 2559M->1698M(2560M) 724.899ms", [1] "[gc] GC(187) Pause Full (G1 Evacuation Pause)" ] ``` It strikes me as unlikely that that is useful to you. Perhaps rename the grok field to gcmessage, or else set the [overwrite](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-overwrite) option on the grok filter. In either case, why do the grok if you are going to throw away the results? Move the drop before the grok. ``` if [message] !~ "Full" { drop {} } grok { ... ``` ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [June 1, 2022, 6:49pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/3 "2022-06-01T18:49:51Z")
Thanks @Badger I tried placing the drop part before the grok and saw no message(s) coming in. This is how it looks now: ```auto filter { if [type] == "tv_gclog_analysis" { if [message] !~ "Full" { drop {} } grok { match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}' } } } } ``` I have set up rubydebug and noticed nothing coming in. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [June 1, 2022, 6:55pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/4 "2022-06-01T18:55:33Z")
> [@zaeemmasood](#): > > `if ([message] !~ "Full") ` how about you do something reverse. i.e only process log which has Full word in it ```auto if ( "Full" in [message]) { grok { } } else { drop {} } ``` ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [June 1, 2022, 7:16pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/5 "2022-06-01T19:16:41Z")
Thanks. Tried the following and still dont see messages coming in: ```auto filter { if [type] == "tv_gclog_analysis" { if "Full" in [message] { grok { match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}' } } } else { drop {} } } } ``` ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [June 2, 2022, 2:45pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/6 "2022-06-02T14:45:36Z")
try this it is possible that it is going inside if but your grok pattern is not legit and can't produce anything ```auto filter { if [type] == "tv_gclog_analysis" { if "Full" in [message] { mutate { add_field => { "zaeemmasooddddddd" => "AAAAAAAAAAAAAAAAAA" } } #grok { #match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}' } } } else { drop {} } } } ``` if this works you will see lot of AAAAAAAAAAAA on your screen. then you can fix your grok. ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [June 2, 2022, 7:41pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/7 "2022-06-02T19:41:55Z")
Thanks. With the following I see no messages coming in: ```auto filter { if [type] == "tv_gclog_analysis" { if "Full" in [message] { mutate { add_field => { "zaeemmasooddddddd" => "AAAAAAAAAAAAAAAAAA" } } #grok { #match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:int}' } #} } else { drop {} } } } ``` The rubydebug log file does not get populated. I can for sure see messages containing "Full" in the gc log as follows: ```auto [2022-06-02T19:37:05.443+0000][info][gc] GC(23145) Pause Full (Diagnostic Command) 1967M->1114M(2560M) 643.338ms ``` ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [June 2, 2022, 7:52pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/8 "2022-06-02T19:52:23Z")
you have to debug this out. because if it is not even going in loop means it is not going in if [type] == "tv\_gclog\_analysis" either? start from there one step at a time just print everything after if [type] == then if "full" in [message] ...... and so on ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [June 2, 2022, 8:50pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/9 "2022-06-02T20:50:16Z")
Thank you so much! The problem was in my setting up of the filter as it was nesting in another one. I can see only "Full" messages coming in as follows: ```auto $ grep Full 6044_rubydebug.txt [0] "[2022-06-02T20:39:43.996+0000][info][gc] GC(23418) Pause Full (Diagnostic Command) 1946M->1066M(2560M) 831.805ms", [1] "[gc] GC(23418) Pause Full (Diagnostic Command)" [0] "[2022-06-02T20:39:43.165+0000][info][gc,start] GC(23418) Pause Full (Diagnostic Command)", ``` Now the issue is I only want to see the line carrying the important stuff which is: ```auto [2022-06-02T20:45:20.349+0000][info][gc] GC(23462) Pause Full (Diagnostic Command) 1328M->1088M(2560M) 471.188ms" ``` I guess another regex would be needed? So need to filter out the following two: ```auto "[gc] GC(23462) Pause Full (Diagnostic Command)" "[2022-06-02T20:45:19.878+0000][info][gc,start] GC(23462) Pause Full (Diagnostic Command)", ``` ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [June 2, 2022, 9:52pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/10 "2022-06-02T21:52:11Z")
> [@zaeemmasood](#): > > `gc ` then you can just do this ```auto if "Diagnostic Command" in [message] { drop{} } if "gc " in [message] { #dosomething } else { drop {} } ``` i don't know if it count spaces in this or not. if this does not work you can go reverse and drop if "gc,start" in [message] { drop {} } ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [June 7, 2022, 1:24pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/11 "2022-06-07T13:24:16Z")
Thank You @elasticforme ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2022, 1:24pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158/12 "2022-07-05T13:24:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.