Fetch the substring of a field and draw pie-chart visualization of counts in Kibana

[anishgin (Anish G)]

Hello
I have lots of messages structured as following :

host: xxxx severity: yyy msg: Cannot find the mapping for string1.
host: xxxx severity: yyy msg: Cannot find the mapping for string3.
host: xxxx severity: yyy msg: Cannot find the mapping for string2.
host: xxxx severity: yyy msg: Cannot find the mapping for string1.

I need to extract/substring 'stringX' from the 'msg' field and count(distinct) them. I would also like to do a pie-chart visualization.

As I am a newbie, I would like to get detailed explanation in solving this.
Thanks in advance!
Anish G

[jbudz (Jon Budzenski)]

There's a few options:

1. In a limited fashion you can extract fields using scripted fields and painless. In this case I wouldn't recommend it, there's an overhead at runtime for every query and we most likely want to parse out the entire message. docs
2. Parse at index time using ingest node in elasticsearch. With ingest node, you can use the grok processor that will parse out the message into json fields. This more or less is a regular expression. docs
3. Parse with logstash grok filters. This is similar to step 2, except it happens in logstash. docs

I'd recommend 2 if you don't have a logstash install. You'll need to create a pipeline, and when indexing tell your index command to use that pipeline.

PUT _ingest/pipeline/my-pipeline-id
{
  "description" : "parse message",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["..."]
      }
    }
  ]
}