New to logstash need help with config

Hi all,

I basically want to visualize response time and api url in kibana with users. i have written a simple config file that should only parse the required log line but i am getting error at just before match in _grokparsefailure if statement.

filter {

        grok {
                match => { "message" => "%{LOGLEVEL:severity} %{GREEDYDATA:timestamp} \[%{DATA:class}\]Req ID: %{NUMBER:req_id} URL: %{URIPATHPARAM:url}, User: %{NUMBER:user_id} -
                                         %{GREEDYDATA:username}, Resp Time: %{NUMBER:duration}"}


        if "_grokparsefailure" in [tags] {
                match => { "message" => "%{GREEDYDATA:message}"}


         date {
                match => [ "logdate", "yyyy MM dd HH:mm:ss" ]

sample log lines
INFO 2019-08-29 09:50:20,681 [User App Mixins]Req ID: 1018 Request URL: /api/v1/userapp/booking/bookingParameters/, Method: POST
INFO 2019-08-29 09:50:20,681 [User App Mixins]Req ID: 1018 Request Query Params: <QueryDict: {}>
INFO 2019-08-29 09:50:20,682 [User App Mixins]Req ID: 1018 Request Data: {u'category_id': 461, u'name': u'RBS OIBP'}
INFO 2019-08-29 09:50:20,682 [User App Mixins]Req ID: 1018 Logged In User- 14351 - Sabarigiri Jayaraman, Android User, URL: /api/v1/userapp/booking/bookingParameters/
INFO 2019-08-29 09:50:20,718 [User App Mixins]Req ID: 1018 URL: /api/v1/userapp/booking/bookingParameters/, User: 14351 - Sabarigiri Jayaraman, Resp Status: 200
INFO 2019-08-29 09:50:20,718 [User App Mixins]Req ID: 1018 URL: /api/v1/userapp/booking/bookingParameters/, User: 14351 - Sabarigiri Jayaraman, Resp Time: 0.048

Any leads are appreciated.

The grok ends at the "}" prior to the if. The if has no filter plugin, "match" is a grok parameter, not a plugin.

Assuming you are just missing a grok in the if, I don't think it does anything useful, "message" is already a field, I don't think you need that section at all.

Hi thanks for taking out time. What i wanted to do was to only print message field with logs that grok fails to parse. I tried dropping them with if "_grokparsefailure" in [tags] { drop {} }

but it seemed to drop every event since it was not printing anything on console. As you pointed out i cannot use match in if but i can use drop {} without grok? . As i would be writing multiple matching patterns , i really need to clear my basics. Thanks

If you want to remove the [message] field if the grok filter successfully parses it then you can use

grok {
    match => { "message" => "some pattern" }
    remove_field => [ "message" ]

The options like remove_field that are common across filters are only applied if the filter successfully processes the event. So if there is a _grokparsefailure the remove_field does not get processed.

Okay so here is a lame doubt. When my grok successfully parses a log, why would i want to remove a field? Isn't it like removing the data from parsed log that was in the message field?

What i was trying was that if my log does not match a pattern put everything in message field and dont show _grokparsefailure in tags.

The grok filter does not modify the [message] field that it is parsing. The data will still be there. If you do not want a _grokparsefailure tag then you can use mutate+remove_tag to get rid of it.

Okay i might be missing something very basic and important in here. When i apply a grok filter, the pattern that i have provided tries to match it with with the incoming log and any successful match is assigned a field and is appended into the message field as well (in the same instance).
So when i use remove_field => ["message"] in the same instance, what exactly is it doing?
Since the message field from my previous event is already overwritten by match parameter.

Badger is right.

lets say if you have
message="who is this guy"
you parse it and now you have
you just want to drop "message" then use remove_field and your data on variable field* says same.

But we get a message field as well when our log parses. It depends on the pattern provided but as a sample stdout shows

field1 = path = blah blah
field2 = ip = blah blah
field3 = message = blah
and so on ...

what am i missing?

I think put some example and explain in more detail.

If you have a field in the pattern called message then the [message] field will end up as an array.

input { generator { count => 1 lines => [ 'foo bar' ] } }
filter { grok { match => { "message" => "%{WORD:message}" } } }
output { stdout { codec => rubydebug { metadata => false } } }

results in

   "message" => [
    [0] "foo bar",
    [1] "foo"

Generally the answer to that is "don't do that". As elasticforme said, an example and more detail about your issue would help.

     "timestamp" => "2019-08-29 13:00:35,949",
      "@version" => "1",
      "severity" => "INFO",
         "class" => "User App Mixins",
          "path" => "/home/rajdeep/logs/n2",
          "host" => "rajdeep-ThinkPad-T460s",
      "duration" => "0.016",
      "username" => "Varun Manomohan",
           "url" => "/api/v2/userapp/parking/",
       "user_id" => "5009",
       "message" => "INFO 2019-08-29 13:00:35,949 [User App Mixins]Req ID: 27311 URL: /api/v2/userapp/parking/, User: 5009 - Varun Manomohan, Resp Time: 0.016",
    "@timestamp" => 2019-09-05T13:59:22.902Z,
        "req_id" => "27311"

See this is my output for this
grok {
match => {
"message" => "%{LOGLEVEL:severity} %{GREEDYDATA:timestamp} [%{DATA:class}]Req ID: %{NUMBER:req_id} URL: %{URIPATHPARAM:url}, User: %{NUMBER:user_id} - %{GREEDYDATA:username}, Resp Status: %{NUMBER:status}|Resp Time: %{NUMBER:duration})"}

So i am getting a message field and so passing remove_field => ["message"] in the same grok instance should remove the data from that field?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.