Need help to exlcude messages which are not having kv pair

Elastic Stack Logstash
1

HI Team,
I have split last field of my message has greedydata

Example Message:

%{ts:timestampe} %{loglevel} %{DATA} %{GREEDYDATA:Logmessage}

My "Logmessage" here at times contains some kv pair in them and i am able to extract them via kv pair extraction as below.

kv { source => "Logmessage" field_split => "," value_split => "=" }

But this kv is just applicable only for the messages which contains the correct kv. there are some other messages which doesn't meet the criteria and i want to eliminate them and want them to be treated as Logmessage only.

Problem: these unwanted messages are creating multiple fields in elasticsearch.

2

Hello @Nithani25

Do the events which are not kv fail the kv filter or they go through it?

When the kv fails, the event is tagged (see tag_on_failure).

If they're tagged, you can identify such events.

If you're able to distinguish between "correct" KV and not-KV Logmessages using a regular expression, I think the best would be to check it with the regular expression before applying the filter.

3

Sample Log messages:

    CurrentURL = [/google/abc] , UserID = nishanth

    EVENTS[1ms]("RenderingManager.getRequestContext")
    {
      [getRequestContext:  Creating RequestContext]=0
      [getRequestContext:  Setting up RequestContext]=0
      [getRequestContext:  Setting up OpenSessionsInViewHelper]=1
    }

{
  [cleanup:  doCleanupSessionsInView]=0
}

while apply below kv filter

kv { source => "Logmessage" field_split => "," value_split => "=" }

I am seeing multiple fields being created, i want only the CurrentURL and UserId to captured and corresponding fields created for them.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.