# New to logstash need help with config **URL:** \ **Category:** Logstash\ **Created:** [September 4, 2019, 7:06am UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987 "2019-09-04T07:06:15Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 4, 2019, 7:06am UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/1 "2019-09-04T07:06:15Z")
Hi all, I basically want to visualize response time and api url in kibana with users. i have written a simple config file that should only parse the required log line but i am getting error at just before `match` in `_grokparsefailure` if statement. ``` filter { grok { match => { "message" => "%{LOGLEVEL:severity} %{GREEDYDATA:timestamp} \[%{DATA:class}\]Req ID: %{NUMBER:req_id} URL: %{URIPATHPARAM:url}, User: %{NUMBER:user_id} - %{GREEDYDATA:username}, Resp Time: %{NUMBER:duration}"} } if "_grokparsefailure" in [tags] { match => { "message" => "%{GREEDYDATA:message}"} } date { match => ["logdate", "yyyy MM dd HH:mm:ss"] } } ``` **sample log lines** INFO 2019-08-29 09:50:20,681 [User App Mixins]Req ID: 1018 Request URL: /api/v1/userapp/booking/bookingParameters/, Method: POST INFO 2019-08-29 09:50:20,681 [User App Mixins]Req ID: 1018 Request Query Params: \ INFO 2019-08-29 09:50:20,682 [User App Mixins]Req ID: 1018 Request Data: {u'category\_id': 461, u'name': u'RBS OIBP'} INFO 2019-08-29 09:50:20,682 [User App Mixins]Req ID: 1018 Logged In User- 14351 - Sabarigiri Jayaraman, Android User, URL: /api/v1/userapp/booking/bookingParameters/ INFO 2019-08-29 09:50:20,718 [User App Mixins]Req ID: 1018 URL: /api/v1/userapp/booking/bookingParameters/, User: 14351 - Sabarigiri Jayaraman, Resp Status: 200 INFO 2019-08-29 09:50:20,718 [User App Mixins]Req ID: 1018 URL: /api/v1/userapp/booking/bookingParameters/, User: 14351 - Sabarigiri Jayaraman, Resp Time: 0.048 Any leads are appreciated. ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [September 4, 2019, 12:09pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/2 "2019-09-04T12:09:53Z")
The grok ends at the "}" prior to the if. The if has no filter plugin, "match" is a grok parameter, not a plugin. Assuming you are just missing a grok in the if, I don't think it does anything useful, "message" is already a field, I don't think you need that section at all. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 5, 2019, 5:36am UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/3 "2019-09-05T05:36:33Z")
Hi thanks for taking out time. What i wanted to do was to only print message field with logs that grok fails to parse. I tried dropping them with `if "_grokparsefailure" in [tags] { drop {} }` but it seemed to drop every event since it was not printing anything on console. As you pointed out i cannot use match in `if` but i can use `drop {}` without grok? . As i would be writing multiple matching patterns , i really need to clear my basics. Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 5, 2019, 12:05pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/4 "2019-09-05T12:05:15Z")
If you want to remove the [message] field if the grok filter successfully parses it then you can use ``` grok { match => { "message" => "some pattern" } remove_field => ["message"] } ``` The options like remove\_field that are common across filters are only applied if the filter successfully processes the event. So if there is a \_grokparsefailure the remove\_field does not get processed. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 5, 2019, 12:16pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/5 "2019-09-05T12:16:27Z")
Okay so here is a lame doubt. When my grok successfully parses a log, why would i want to remove a field? Isn't it like removing the data from parsed log that was in the message field? What i was trying was that if my log does not match a pattern put everything in message field and dont show \_grokparsefailure in tags. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 5, 2019, 12:31pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/6 "2019-09-05T12:31:14Z")
The grok filter does not modify the [message] field that it is parsing. The data will still be there. If you do not want a \_grokparsefailure tag then you can use mutate+remove\_tag to get rid of it. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 5, 2019, 1:11pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/7 "2019-09-05T13:11:44Z")
Okay i might be missing something very basic and important in here. When i apply a grok filter, the pattern that i have provided tries to match it with with the incoming log and any successful match is assigned a field and is appended into the message field as well (in the same instance). So when i use `remove_field => ["message"]` in the same instance, what exactly is it doing? Since the message field from my previous event is already overwritten by match parameter. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [September 5, 2019, 1:30pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/8 "2019-09-05T13:30:39Z")
Badger is right. lets say if you have message="who is this guy" you parse it and now you have field1=who field2=is field3=this field4=guy you just want to drop "message" then use remove\_field and your data on variable field\* says same. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 5, 2019, 1:37pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/9 "2019-09-05T13:37:18Z")
But we get a message field as well when our log parses. It depends on the pattern provided but as a sample stdout shows field1 = path = blah blah field2 = ip = blah blah field3 = message = blah and so on ... what am i missing? ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [September 5, 2019, 1:39pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/10 "2019-09-05T13:39:24Z")
I think put some example and explain in more detail. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 5, 2019, 1:43pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/11 "2019-09-05T13:43:08Z")
If you have a field in the pattern called message then the [message] field will end up as an array. ``` input { generator { count => 1 lines => ['foo bar'] } } filter { grok { match => { "message" => "%{WORD:message}" } } } output { stdout { codec => rubydebug { metadata => false } } } ``` results in ``` "message" => [ [0] "foo bar", [1] "foo" ], ``` Generally the answer to that is "don't do that". As elasticforme said, an example and more detail about your issue would help. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 5, 2019, 2:03pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/12 "2019-09-05T14:03:33Z")
``` { "timestamp" => "2019-08-29 13:00:35,949", "@version" => "1", "severity" => "INFO", "class" => "User App Mixins", "path" => "/home/rajdeep/logs/n2", "host" => "rajdeep-ThinkPad-T460s", "duration" => "0.016", "username" => "Varun Manomohan", "url" => "/api/v2/userapp/parking/", "user_id" => "5009", "message" => "INFO 2019-08-29 13:00:35,949 [User App Mixins]Req ID: 27311 URL: /api/v2/userapp/parking/, User: 5009 - Varun Manomohan, Resp Time: 0.016", "@timestamp" => 2019-09-05T13:59:22.902Z, "req_id" => "27311" } ``` See this is my output for this grok { match =\> { "message" =\> "%{LOGLEVEL:severity} %{GREEDYDATA:timestamp} [%{DATA:class}]Req ID: %{NUMBER:req\_id} URL: %{URIPATHPARAM:url}, User: %{NUMBER:user\_id} - %{GREEDYDATA:username}, Resp Status: %{NUMBER:status}|Resp Time: %{NUMBER:duration})"} } So i am getting a message field and so passing `remove_field => ["message"]` in the same grok instance should remove the data from that field? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 3, 2019, 2:03pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-with-config/197987/13 "2019-10-03T14:03:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.