# Incorrect JSON logs parsing

**URL:** <https://discuss.elastic.co/t/incorrect-json-logs-parsing/323447>\
**Category:** Kibana\
**Created:** [January 18, 2023, 7:04pm UTC](https://discuss.elastic.co/t/incorrect-json-logs-parsing/323447 "2023-01-18T19:04:12Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ira-zaya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ira-zaya/32/115622_2.png) [@ira-zaya](https://discuss.elastic.co/u/ira-zaya)\
**Post date:** [January 18, 2023, 7:04pm UTC](https://discuss.elastic.co/t/incorrect-json-logs-parsing/323447/1 "2023-01-18T19:04:12Z")

</div>

Hi. I have json format logs, looks like this:

```auto
{
	"dt":"2023-01-18T17:41:04.8723262+00:00",
	"tz":"Etc/UTC",
	"host":"host-name",
	"containerName":"container-name",
	"level":"INFO",
	"scope":"Web API",
	"message":"exiting web api method",
	"callerMemberName":"OnResultExecuted",
	"callerFilePath":"/path",
	"callerLineNumber":94,
	"context":
		{
			"StatusCode":200,
			"ControllerName":"name",
			"ActionName":"InsertAsync"
		},
	"exception":null,
	"executionId":"1234567890abcdefghig",
	"traceId":"1234567890abcdefghig"
}

```

In Kibana it's looks like this:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/f/df6be77be88e6e4b2844129b8d102886919185ab.png)

And a "decode\_json\_fields" processor in filebeat.yml file:

```auto
filebeat.inputs:
- type: container
  paths:
  - /var/log/containers/*.log
  include_lines: ['DEBUG', 'INFO', 'ERROR', 'WARN', 'WARNING', 'FATAL', 'CRITICAL']
  #exclude_files: ['.gz$']
  multiline.pattern: ^\d
  multiline.negate: true
  multiline.match: after
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"
  - decode_json_fields:
      fields: ["level"]
      target: ""

```

So I want to get only "level" field so that later I can display it in kibana.  
Now kibana displays incorrect log level (I know that such a displaying indicates an error in the parser configuration):  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/4/d4f2ca3a1e7795591c9382ed7f9346dc95322ace.png)

Or maybe someone can advice an approach to parse all json data in many different individual fields in kibana. That would be good for me too.  
Thanks

---

<div class="post-metadata">

**Author:** ![jsanz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsanz/32/53734_2.png) [@jsanz](https://discuss.elastic.co/u/jsanz)\
**Post date:** [January 19, 2023, 3:46pm UTC](https://discuss.elastic.co/t/incorrect-json-logs-parsing/323447/2 "2023-01-19T15:46:25Z")

</div>

I'd suggest to send the `message` as it is from filebeat and configure an [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) that process your data to your requirements. So you can keep all the ingest logic close to your cluster.

Taking your input, extracting the `level` key from the message in this [simulated query](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html) is quite straight forward using the [json](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html) processor:

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "json": {
          "field": "message",
          "add_to_root": true
        }
      },
      {
        "remove": {
          "field": [
            "callerMemberName", "exception",
            "traceId", "tz",
            "callerFilePath", "message",
            "callerLineNumber", "dt",
            "executionId", "containerName",
            "scope", "host", "context"
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "message": """{
      	"dt":"2023-01-18T17:41:04.8723262+00:00",
      	"tz":"Etc/UTC",
      	"host":"host-name",
      	"containerName":"container-name",
      	"level":"INFO",
      	"scope":"Web API",
      	"message":"exiting web api method",
      	"callerMemberName":"OnResultExecuted",
      	"callerFilePath":"/path",
      	"callerLineNumber":94,
      	"context":
      		{
      			"StatusCode":200,
      			"ControllerName":"name",
      			"ActionName":"InsertAsync"
      		},
      	"exception":null,
      	"executionId":"1234567890abcdefghig",
      	"traceId":"1234567890abcdefghig"
      }"""
      }
    }
  ]
}

```

The simulated output of that pipeline is:

```auto
{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_id": "id",
        "_version": "-3",
        "_source": {
          "level": "INFO"
        },
        "_ingest": {
          "timestamp": "2023-01-19T15:42:59.808712161Z"
        }
      }
    }
  ]
}

```

Of course you can keep any other fields taking them out of the [remove](https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html) processor, and you should first define the mapping of your data, preferably with an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 16, 2023, 3:47pm UTC](https://discuss.elastic.co/t/incorrect-json-logs-parsing/323447/3 "2023-02-16T15:47:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.