Need help to exlcude messages which are not having kv pair

HI Team,
I have split last field of my message has greedydata

Example Message:

%{ts:timestampe} %{loglevel} %{DATA} %{GREEDYDATA:Logmessage}

My "Logmessage" here at times contains some kv pair in them and i am able to extract them via kv pair extraction as below.

kv { source => "Logmessage" field_split => "," value_split => "=" }

But this kv is just applicable only for the messages which contains the correct kv. there are some other messages which doesn't meet the criteria and i want to eliminate them and want them to be treated as Logmessage only.

Problem: these unwanted messages are creating multiple fields in elasticsearch.

Hello @Nithani25

Do the events which are not kv fail the kv filter or they go through it?

When the kv fails, the event is tagged (see tag_on_failure).

If they're tagged, you can identify such events.

If you're able to distinguish between "correct" KV and not-KV Logmessages using a regular expression, I think the best would be to check it with the regular expression before applying the filter.

Sample Log messages:

    CurrentURL = [/google/abc] , UserID = nishanth

    EVENTS[1ms]("RenderingManager.getRequestContext")
    {
      [getRequestContext:  Creating RequestContext]=0
      [getRequestContext:  Setting up RequestContext]=0
      [getRequestContext:  Setting up OpenSessionsInViewHelper]=1
    }

{
  [cleanup:  doCleanupSessionsInView]=0
}

while apply below kv filter

kv { source => "Logmessage" field_split => "," value_split => "=" }

I am seeing multiple fields being created, i want only the CurrentURL and UserId to captured and corresponding fields created for them.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.