Need help to exlcude messages which are not having kv pair

HI Team,
I have split last field of my message has greedydata

Example Message:

%{ts:timestampe} %{loglevel} %{DATA} %{GREEDYDATA:Logmessage}

My "Logmessage" here at times contains some kv pair in them and i am able to extract them via kv pair extraction as below.

kv { source => "Logmessage" field_split => "," value_split => "=" }

But this kv is just applicable only for the messages which contains the correct kv. there are some other messages which doesn't meet the criteria and i want to eliminate them and want them to be treated as Logmessage only.

Problem: these unwanted messages are creating multiple fields in elasticsearch.

Hello @Nithani25

Do the events which are not kv fail the kv filter or they go through it?

When the kv fails, the event is tagged (see tag_on_failure).

If they're tagged, you can identify such events.

If you're able to distinguish between "correct" KV and not-KV Logmessages using a regular expression, I think the best would be to check it with the regular expression before applying the filter.

Sample Log messages:

    CurrentURL = [/google/abc] , UserID = nishanth

      [getRequestContext:  Creating RequestContext]=0
      [getRequestContext:  Setting up RequestContext]=0
      [getRequestContext:  Setting up OpenSessionsInViewHelper]=1

  [cleanup:  doCleanupSessionsInView]=0

while apply below kv filter

kv { source => "Logmessage" field_split => "," value_split => "=" }

I am seeing multiple fields being created, i want only the CurrentURL and UserId to captured and corresponding fields created for them.