Filter with Grok and KV

Elastic Stack Logstash
1

Hi,

I'm trying to understand if I'm using the correct configuration for the following logs, or if there is a more efficient option for it.

2021-07-13T10:05:06.061Z 10.20.30.40 <110>1 2021-07-13T08:46:58Z 44.236.133.39 Keeper - 4269856080 [Keeper@Commander geo_location="Portland, Oregon, US" keeper_version_category="ADMIN" audit_event_type="login" keeper_version="Commander 15.4.85" username="test@domain.com" node_id="123456"] User test@domain.com logged in to vault

2021-07-13T10:05:06.061Z 10.20.30.40 <110>1 2021-07-13T09:16:47Z 77.52.201.194 Keeper - 4269938480 [Keeper@Commander geo_location="Vyshhorod, Kyivska oblast, UA" keeper_version_category="ADMIN" audit_event_type="enable_user" keeper_version="EMConsole 15.3.4" to_username="test2@domain.com" username="test3@domain.com" node_id="123456"] User test2@domain.comwas enabled by admin test3@domain.com

filter {
   grok {
     match => { "message" => "%{GREEDYDATA:drop1}\[Keeper\@Commander%{GREEDYDATA:text}\]%{GREEDYDATA:description} "}
     remove_field => ["drop1", "message"]
  }
   kv  {
     source => "text"
     trim_value => "\""
     }
}

Thank you

2

If you do not want to keep a field there is no need to name it and the use remove_field. You could just use

match => { "message" => "%{GREEDYDATA}\[Keeper\@Commander%{GREEDYDATA:text}\]%{GREEDYDATA:description} "}

and if a pattern is not anchored it does not need to match the entire field. So you would be better off with

match => { "message" => "\[Keeper\@Commander%{GREEDYDATA:text}\]%{GREEDYDATA:description} "}

Personally I would do that using

match => { "message" => "\[Keeper\@Commander(?<text>[^\]]*)\]%{GREEDYDATA:description} "}

in case a one the kv pairs ever contains ]

3

It's very helpful, I'll use that, thank you!

4

I’m forwarding the output as a JSON, but it seems like for some reason, the last word is dropped from the description, even though, I can see it in the message itself. Any ideas what can cause it?

 {
    "type":"keeper",
    "@version":"1",
    "keeper_version":"KeeperEnterpriseBridge 15.1.0",
    "username":"user@domain.com",
    "audit_event_type":"login",
    "geo_location":"Tel Aviv, Tel Aviv, IL",
    "port":40236,
    "@timestamp":"2021-07-15T07:08:05.795Z",
    "node_id":"123456",
    "keeper_version_category":"ADMIN",
    "host":"10.20.30.40",
    "description":" User user@domain.com logged in to",
    "message":"<110>1 2021-07-15T07:03:07Z 1.2.3.4 Keeper - 4278464624 [Keeper@Commander geo_location=\"Tel Aviv, Tel Aviv, IL\" keeper_version_category=\"ADMIN\" audit_event_type=\"login\" keeper_version=\"KeeperEnterpriseBridge 15.1.0\" username=\"user@domain.com\" node_id=\"123456\"] User user@domain.com logged in to vault"
 }
5

Hi,

The space at the end of your grok pattern indicate that the GREEDYDATA have to take each value until the last space of the field message. So it also don't take the last word after the space.

Cad

1 Like
6

Hi,

Yep it did solve it,

Thanks

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.