# Filebeat - Parse json output

**URL:** <https://discuss.elastic.co/t/filebeat-parse-json-output/178079>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 23, 2019, 4:46pm UTC](https://discuss.elastic.co/t/filebeat-parse-json-output/178079 "2019-04-23T16:46:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![arianmotamedi](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@arianmotamedi](https://discuss.elastic.co/u/arianmotamedi)\
**Post date:** [April 23, 2019, 4:46pm UTC](https://discuss.elastic.co/t/filebeat-parse-json-output/178079/1 "2019-04-23T16:46:27Z")

</div>

I realize this has been asked numerous times but I'm having a hard time getting Filebeat 6.2.4 to send json logs to Elasticsearch (without Logstash) in a K8s environment. For example for a given json log:

> {"@timestamp":"2019-04-23T16:22:33.979906989Z","application":"my-app","correlation\_id":"55189165-dafa-4b22-a720-adfa9eafc6e6","data\_version":2,"description":"","envoy\_data":{"downstream\_local":{"address":"127.0.0.1","port":12345,"protocol":"TCP"},"downstream\_remote":{"address":"127.0.0.1","port":56004,"protocol":"TCP"},"listener":"ingress","request\_received\_time":"2019-04-04T16:22:33.979Z","request\_start\_time":"2019-04-04T16:22:33.979Z","response\_duration\_ms":0,"response\_received\_time":"2019-04-04T16:22:33.979Z","response\_start\_time":"2019-04-04T16:22:33.979Z","upstream\_cluster":"my-service","upstream\_remote":{"address":"127.0.0.1","port":8080,"protocol":"TCP"}},"level":"info","msg":"","roletype":"my-role-type","service":"foobar","type":"log"}

The entry in Kibana looks like this:

 ![kibana](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1e3733ddb60a89c9538fd3ea7dd7420302175bbf.png)

The json output is being logged under the `log` field. I need the fields to be parsed and placed in the root document, and my understanding was `json.keys_under_root: true` would do the trick, but doesn't seem to. Below is my full config:

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.config:
      prospectors:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
    logging.level: debug
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:443}']
      template.name: filebeat
      template.path: filebeat.template.json

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: log
      paths:
        - /var/lib/docker/containers/*/*.log
      json.add_error_key: true
      json.keys_under_root: true
      json.message_key: log
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            namespace: ${POD_NAMESPACE}

```

Any help in the right direction would be appreciated.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 23, 2019, 9:53pm UTC](https://discuss.elastic.co/t/filebeat-parse-json-output/178079/2 "2019-04-23T21:53:57Z")

</div>

I see you have configured your docker containers as path, but type `log`. Docker itself embeds logs in JSON, which gives you JSON in JSON.

Have you tried the [docker](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-docker.html#filebeat-input-docker) input type instead of 'log' ?

---

<div class="post-metadata">

**Author:** ![arianmotamedi](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@arianmotamedi](https://discuss.elastic.co/u/arianmotamedi)\
**Post date:** [April 29, 2019, 5:52pm UTC](https://discuss.elastic.co/t/filebeat-parse-json-output/178079/3 "2019-04-29T17:52:41Z")

</div>

Actually turns out my log file had multi-lines in it. adding

```auto
multiline.pattern: '^{'
multiline.negate: true
multiline.match: after

```

as well as

```auto
- decode_json_fields:
    fields: ["log"]
    process_array: false
    max_depth: 5
    target: ""
    overwrite_keys: true

```

to the list of processors fixed the issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 27, 2019, 5:52pm UTC](https://discuss.elastic.co/t/filebeat-parse-json-output/178079/4 "2019-05-27T17:52:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.