How to group similar log messages and show in bar chart

Elastic Stack Kibana
1

Hello,
I wanted to group the similar messages and display them in chart.
The message format is not unique, its slightly change every time.
below is the sample message:
"NO TRANSLATION AVAILABLE From:XYZ Code:ABC To:OSC App:Test APP"

In above message 'From', 'code' and 'App' values are changed every time, but they are duplicate in log, i wanted to those duplicate values and display in chart.

I tried creating scripted field also using 'filter aggregation' but no luck since. because they mostly match certain keyword and filter record but in my case that does not work.

Any idea how this can be achieved,?

2

Hi @balaji-khandekar-osv, welcome to our community! :wave:

It really depends on how you ingest your data but you can define an ingest pipeline with a grok processor to extract your data from your lines and from there generate any visualizations you need.

Some example execution using the Kibana Console:

# Clean up
DELETE discuss-333586
DELETE _ingest/pipeline/discuss-333586-pipeline

# Create an index
PUT discuss-333586
{
  "mappings": {
    "properties": {
      "message": { "type": "text"},
      "from": { "type": "keyword"},
      "to": { "type": "keyword"},
      "code": { "type": "keyword"},
      "app": { "type": "keyword"}
    }
  }
}

# Create a pipeline that groks the message field
PUT _ingest/pipeline/discuss-333586-pipeline
{
    "description": "A simple parsing pipeline that groks a message",
    "processors": [
          {
        "grok" : {
          "field" : "message",
          "patterns" :[ """^%{DATA:Prefx}\ From:%{DATA:from}\ Code:%{DATA:code}\ To:%{DATA:to}\ App:%{DATA:app}$"""]
        }
      }
    ]
}

# Ingest some data using the pipeline
POST discuss-333586/_bulk?pipeline=discuss-333586-pipeline
{ "index": {}}
{ "message": "Something here From:XYZ Code:ABC To:OSC App:Test APP"}
{ "index": {}}
{ "message": "Something there From:ZXY Code:CBA To:CSO App:Another Test APP"}
{ "index": {}}
{ "message": "Something here as well From:YXZ Code:BCA To:OCS App:Yet Another Test APP"}

# Check the data ingested
GET discuss-333586/_search

Where the result of the last search is:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "discuss-333586",
        "_id": "VrhjaIgBsXAnXexqHoPZ",
        "_score": 1,
        "_source": {
          "app": "Test APP",
          "Prefx": "Something here",
          "from": "XYZ",
          "code": "ABC",
          "to": "OSC",
          "message": "Something here From:XYZ Code:ABC To:OSC App:Test APP"
        }
      },
      {
        "_index": "discuss-333586",
        "_id": "V7hjaIgBsXAnXexqHoPZ",
        "_score": 1,
        "_source": {
          "app": "Another Test APP",
          "Prefx": "Something there",
          "from": "ZXY",
          "code": "CBA",
          "to": "CSO",
          "message": "Something there From:ZXY Code:CBA To:CSO App:Another Test APP"
        }
      },
      {
        "_index": "discuss-333586",
        "_id": "WLhjaIgBsXAnXexqHoPZ",
        "_score": 1,
        "_source": {
          "app": "Yet Another Test APP",
          "Prefx": "Something here as well",
          "from": "YXZ",
          "code": "BCA",
          "to": "OCS",
          "message": "Something here as well From:YXZ Code:BCA To:OCS App:Yet Another Test APP"
        }
      }
    ]
  }
}

Hope it helps!

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.