About Ingest Processor patterns

Elastic Stack
1

My custom log lines look something like this
level=WARNING;user=john;location=xxx;

Its like "LHS=RHS;"
A processor I have to write for this is:-

"processors": [

     {
        "grok": {
          "field": "message",
          "patterns": ["(?<left1>[^=]*) (?<equal1>={1}) (?<right1>[^;]*)"]
        }
      },
      
      {
        "set":
        {
          "field":"{{{left1}}}",
          "value":"{{{right1}}}"
          
        }
        
      },
      {
        "remove":
        {
          "field":["message","left1","right1","equal1"]
        }
        
      }
      
      
      
    ]

The above code beautifully serves my purpose,
But my doubt is I am not sure how many "xxx=yyy; " patterns will be in my log line. they may be 2 or 3 or 4 or anywhere up to 10.
How to proceed with this?
Thanks for the help.

2

Seems like the kv filter is a good fit for your use case / log pattern

2 Likes
3

Thanks for the quick reply @stephenb .
Means a lot.
But I am not able to understand the exact place where I can ask my ingestion pipeline processor to use this filter plugin

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description" : "Test pipeline",
    "processors": [

    ]
  },
    "docs": [
    {
      "_source": {
        "message": "top_code = GAWEBASBIB23;"
      }
    }


  ]
}

I have tried a few different ways but all show me formatting errors.
Can someone help with where is that exactly we need to write about this filter?
Thank you.

4

Hi @katakam_chaitanya Apologies I am / was a bit confused ... Your post mentioned Logstash patterns but it look like you are using ingest processors... which are excellent as well.

So that is here

Also a little confused...

I the beginning you posted your log looked like

level=WARNING;user=john;location=xxx;

Then you show with spaces...

top_code = GAWEBASBIB23;

either will work

If So

PUT _ingest/pipeline/discuss-test
{
  "processors": [
    {
      "kv": {
        "field": "message",
        "field_split": ";",
        "value_split": "=",
        "trim_key" : " ",
        "trim_value": " "
      }
    }
  ]
}
  

POST _ingest/pipeline/discuss-test/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "level=WARNING;user=john;location=xxx;"
      }
    },
    {
    "_source": {
        "message": "level = ERROR; user = bob; location = yyy;"
      }
    }
  ]
}

Results

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "level" : "WARNING",
          "location" : "xxx",
          "message" : "level=WARNING;user=john;location=xxx;",
          "user" : "john"
        },
        "_ingest" : {
          "timestamp" : "2022-05-23T15:19:05.9806446Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "level" : "ERROR",
          "location" : "yyy",
          "message" : "level = ERROR; user = bob; location = yyy;",
          "user" : "bob"
        },
        "_ingest" : {
          "timestamp" : "2022-05-23T15:19:05.980650457Z"
        }
      }
    }
  ]
}

Hope that helps

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.