New to logstash need help with config

Elastic Stack Logstash
1

Hi all,

I basically want to visualize response time and api url in kibana with users. i have written a simple config file that should only parse the required log line but i am getting error at just before match in _grokparsefailure if statement.

filter {

        grok {
                match => { "message" => "%{LOGLEVEL:severity} %{GREEDYDATA:timestamp} \[%{DATA:class}\]Req ID: %{NUMBER:req_id} URL: %{URIPATHPARAM:url}, User: %{NUMBER:user_id} -
                                         %{GREEDYDATA:username}, Resp Time: %{NUMBER:duration}"}

        }

        if "_grokparsefailure" in [tags] {
                match => { "message" => "%{GREEDYDATA:message}"}

        }

         date {
                match => [ "logdate", "yyyy MM dd HH:mm:ss" ]
        }
}

sample log lines
INFO 2019-08-29 09:50:20,681 [User App Mixins]Req ID: 1018 Request URL: /api/v1/userapp/booking/bookingParameters/, Method: POST
INFO 2019-08-29 09:50:20,681 [User App Mixins]Req ID: 1018 Request Query Params: <QueryDict: {}>
INFO 2019-08-29 09:50:20,682 [User App Mixins]Req ID: 1018 Request Data: {u'category_id': 461, u'name': u'RBS OIBP'}
INFO 2019-08-29 09:50:20,682 [User App Mixins]Req ID: 1018 Logged In User- 14351 - Sabarigiri Jayaraman, Android User, URL: /api/v1/userapp/booking/bookingParameters/
INFO 2019-08-29 09:50:20,718 [User App Mixins]Req ID: 1018 URL: /api/v1/userapp/booking/bookingParameters/, User: 14351 - Sabarigiri Jayaraman, Resp Status: 200
INFO 2019-08-29 09:50:20,718 [User App Mixins]Req ID: 1018 URL: /api/v1/userapp/booking/bookingParameters/, User: 14351 - Sabarigiri Jayaraman, Resp Time: 0.048

Any leads are appreciated.

2

The grok ends at the "}" prior to the if. The if has no filter plugin, "match" is a grok parameter, not a plugin.

Assuming you are just missing a grok in the if, I don't think it does anything useful, "message" is already a field, I don't think you need that section at all.

3

Hi thanks for taking out time. What i wanted to do was to only print message field with logs that grok fails to parse. I tried dropping them with if "_grokparsefailure" in [tags] { drop {} }

but it seemed to drop every event since it was not printing anything on console. As you pointed out i cannot use match in if but i can use drop {} without grok? . As i would be writing multiple matching patterns , i really need to clear my basics. Thanks

4

If you want to remove the [message] field if the grok filter successfully parses it then you can use

grok {
    match => { "message" => "some pattern" }
    remove_field => [ "message" ]
}

The options like remove_field that are common across filters are only applied if the filter successfully processes the event. So if there is a _grokparsefailure the remove_field does not get processed.

5

Okay so here is a lame doubt. When my grok successfully parses a log, why would i want to remove a field? Isn't it like removing the data from parsed log that was in the message field?

What i was trying was that if my log does not match a pattern put everything in message field and dont show _grokparsefailure in tags.

6

The grok filter does not modify the [message] field that it is parsing. The data will still be there. If you do not want a _grokparsefailure tag then you can use mutate+remove_tag to get rid of it.

7

Okay i might be missing something very basic and important in here. When i apply a grok filter, the pattern that i have provided tries to match it with with the incoming log and any successful match is assigned a field and is appended into the message field as well (in the same instance).
So when i use remove_field => ["message"] in the same instance, what exactly is it doing?
Since the message field from my previous event is already overwritten by match parameter.

8

Badger is right.

lets say if you have
message="who is this guy"
you parse it and now you have
field1=who
field2=is
field3=this
field4=guy
you just want to drop "message" then use remove_field and your data on variable field* says same.

9

But we get a message field as well when our log parses. It depends on the pattern provided but as a sample stdout shows

field1 = path = blah blah
field2 = ip = blah blah
field3 = message = blah
and so on ...

what am i missing?

10

I think put some example and explain in more detail.

11

If you have a field in the pattern called message then the [message] field will end up as an array.

input { generator { count => 1 lines => [ 'foo bar' ] } }
filter { grok { match => { "message" => "%{WORD:message}" } } }
output { stdout { codec => rubydebug { metadata => false } } }

results in

   "message" => [
    [0] "foo bar",
    [1] "foo"
],

Generally the answer to that is "don't do that". As elasticforme said, an example and more detail about your issue would help.

12 
{
     "timestamp" => "2019-08-29 13:00:35,949",
      "@version" => "1",
      "severity" => "INFO",
         "class" => "User App Mixins",
          "path" => "/home/rajdeep/logs/n2",
          "host" => "rajdeep-ThinkPad-T460s",
      "duration" => "0.016",
      "username" => "Varun Manomohan",
           "url" => "/api/v2/userapp/parking/",
       "user_id" => "5009",
       "message" => "INFO 2019-08-29 13:00:35,949 [User App Mixins]Req ID: 27311 URL: /api/v2/userapp/parking/, User: 5009 - Varun Manomohan, Resp Time: 0.016",
    "@timestamp" => 2019-09-05T13:59:22.902Z,
        "req_id" => "27311"
}

See this is my output for this
grok {
match => {
"message" => "%{LOGLEVEL:severity} %{GREEDYDATA:timestamp} [%{DATA:class}]Req ID: %{NUMBER:req_id} URL: %{URIPATHPARAM:url}, User: %{NUMBER:user_id} - %{GREEDYDATA:username}, Resp Status: %{NUMBER:status}|Resp Time: %{NUMBER:duration})"}
}

So i am getting a message field and so passing remove_field => ["message"] in the same grok instance should remove the data from that field?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.