# Filebeat fails to process kibana json logs "failed to format message from \*json-.log "in a kubernetes enviroment

**URL:** <https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558>\
**Category:** Logs\
**Created:** [January 9, 2019, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558 "2019-01-09T15:03:22Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/1 "2019-01-09T15:03:22Z")

</div>

So ive mounted ELK stack with filebeat in a kubernetes enviroment, im parsing all the logs correctly, only problem is the kibana json-logs format that get error

> failed to format message from /var/lib/docker/containers/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba-json.log

So did a kubectl describe pods and realized that docker container was kibana. Version 6.5.2

Filebeat configuration:

> ```
> ---
> apiVersion: v1
> kind: ConfigMap
> metadata:
> name: filebeat-config
> namespace: kube-system
> labels:
> k8s-app: filebeat
> data:
> filebeat.yml: |-
> filebeat.config:
> inputs:
> path: ${path.config}/inputs.d/*.yml
> reload.enabled: false
> modules:
> path: ${path.config}/modules.d/*.yml
> reload.enabled: false
> processors:
> - add_cloud_metadata:
> - drop_fields:
> when:
> has_fields: ['kubernetes.labels.app']
> fields:
> - 'kubernetes.labels.app'
> output.elasticsearch:
> hosts: ['http://elasticsearch.whitenfv.svc.cluster.local:9200']
> ---
> apiVersion: v1
> kind: ConfigMap
> metadata:
> name: filebeat-inputs
> namespace: kube-system
> labels:
> k8s-app: filebeat
> data:
> kubernetes.yml: |-
> - type: docker
> json.keys_under_root: false
> json.add_error_key: false
> json.ignore_decoding_error: true
> containers.ids:
> - "*"
> processors:
> - add_kubernetes_metadata:
> in_cluster: true
> ---
> apiVersion: extensions/v1beta1
> kind: DaemonSet
> metadata:
> name: filebeat
> namespace: kube-system
> labels:
> k8s-app: filebeat
> spec:
> template:
> metadata:
> labels:
> k8s-app: filebeat
> spec:
> serviceAccountName: filebeat
> terminationGracePeriodSeconds: 30
> containers:
> - name: filebeat
> image: {{ filebeat_image_full }}
> args: [
> "-c", "/etc/filebeat.yml",
> "-e",
> ]
> securityContext:
> runAsUser: 0
> resources:
> limits:
> memory: 200Mi
> requests:
> cpu: 100m
> memory: 100Mi
> volumeMounts:
> - name: config
> mountPath: /etc/filebeat.yml
> readOnly: true
> subPath: filebeat.yml
> - name: inputs
> mountPath: /usr/share/filebeat/inputs.d
> readOnly: true
> - name: data
> mountPath: /usr/share/filebeat/data
> - name: varlibdockercontainers
> mountPath: /var/lib/docker/containers
> readOnly: true
> volumes:
> - name: config
> configMap:
> defaultMode: 0600
> name: filebeat-config
> - name: varlibdockercontainers
> hostPath:
> path: /var/lib/docker/containers
> - name: inputs
> configMap:
> defaultMode: 0600
> name: filebeat-inputs
> - name: data
> hostPath:
> path: /var/lib/filebeat-data
> type: DirectoryOrCreate
> ---
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRoleBinding
> metadata:
> name: filebeat
> subjects:
> - kind: ServiceAccount
> name: filebeat
> namespace: kube-system
> roleRef:
> kind: ClusterRole
> name: filebeat
> apiGroup: rbac.authorization.k8s.io
> ---
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRole
> metadata:
> name: filebeat
> labels:
> k8s-app: filebeat
> rules:
> - apiGroups: [""]
> resources:
> - namespaces
> - pods
> verbs:
> - get
> - watch
> - list
> ---
> apiVersion: v1
> kind: ServiceAccount
> metadata:
> name: filebeat
> namespace: kube-system
> labels:
> k8s-app: filebeat
> 
> ```

---

<div class="post-metadata">

**Author:** ![skh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/skh/32/38637_2.png) [@skh](https://discuss.elastic.co/u/skh)\
**Post date:** [January 9, 2019, 5:49pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/2 "2019-01-09T17:49:52Z")

</div>

Hi @paltaa,

thank you for trying out the Logs UI and reporting this problem!

We're aware that the parsing and display of log messages needs improving. The work that's currently done for that is tracked in [https://github.com/elastic/kibana/issues/26759](https://github.com/elastic/kibana/issues/26759) .

cheers,  
Sonja

---

<div class="post-metadata">

**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\
**Post date:** [January 9, 2019, 5:55pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/3 "2019-01-09T17:55:43Z")

</div>

Can you post an sample of the event document JSON from that was indexed into Elasticsearch?

Go to Discover:

1. filter by `source:"/var/lib/docker/containers/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba-json.log"`
2. click on one of the records
3. click on the JSON tab
4. copy the source.

Make sure to `xxxx` out any sensitive data.

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 6:01pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/4 "2019-01-09T18:01:21Z")

</div>

```
{
  "_index": "filebeat-6.5.2-2019.01.09",
  "_type": "doc",
  "_id": "WoPCM2gBPXD5Ivx45Ivl",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-01-09T17:57:16.136Z",
    "offset": 7033262,
    "input": {
      "type": "docker"
    },
    "host": {
      "name": "filebeat-f7hqh"
    },
    "beat": {
      "version": "6.5.2",
      "name": "filebeat-f7hqh",
      "hostname": "filebeat-f7hqh"
    },
    "meta": {
      "cloud": {
        "instance_id": "i-000000d5",
        "machine_type": "m1.large",
        "instance_name": "whitenfv-jptest-3.novalocal",
        "availability_zone": "nova",
        "provider": "openstack"
      }
    },
    "json": {
      "method": "post",
      "statusCode": 200,
      "req": {
        "method": "post",
        "headers": {
          "content-type": "application/x-ndjson",
          "accept-encoding": "gzip, deflate",
          "accept-language": "en-US,en;q=0.9,es;q=0.8,fr;q=0.7",
          "content-length": "1091",
          "accept": "application/json, text/plain, */*",
          "origin": "http://xxxx",
          "kbn-version": "6.5.2",
          "referer": "http://xxxxapp/kibana",
          "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
          "host": "198.204.227.93:30001",
          "connection": "keep-alive"
        },
        "remoteAddress": "10.233.64.0",
        "userAgent": "10.233.64.0",
        "referer": "xxxx/app/kibana",
        "url": "/elasticsearch/_msearch"
      },
      "type": "response",
      "@timestamp": "2019-01-09T17:57:14Z",
      "res": {
        "contentLength": 9,
        "statusCode": 200,
        "responseTime": 1444
      },
      "message": "POST /elasticsearch/_msearch 200 1444ms - 9.0B",
      "tags": [],
      "pid": 1
    },
    "source": "/var/lib/docker/containers/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba-json.log",
    "stream": "stdout",
    "prospector": {
      "type": "docker"
    },
    "kubernetes": {
      "replicaset": {
        "name": "kibana-865c55468"
      },
      "labels": {
        "pod-template-hash": "865c55468"
      },
      "pod": {
        "name": "kibana-865c55468-vqb78"
      },
      "node": {
        "name": "node3"
      },
      "container": {
        "name": "kibana"
      },
      "namespace": "whitenfv"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-01-09T17:57:16.136Z"
    ]
  },
  "highlight": {
    "source": [
      "@kibana-highlighted-field@/var/lib/docker/containers/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba-json.log@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1547056636136
  ]
}
```

---

<div class="post-metadata">

**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\
**Post date:** [January 9, 2019, 6:17pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/5 "2019-01-09T18:17:53Z")

</div>

This PR outlines the setting for the Infrastructure and Logging UI's: [https://github.com/elastic/kibana/pull/26579](https://github.com/elastic/kibana/pull/26579)

You can add `xpack.infra.sources.default.fields.message: ['message', '@message', 'json.message']` to `config/kibana.yml` config file. By default the UI only looks at `['message', '@message']` so you just need to tell the UI where the message field is.

Update: Corrected `config.kibana.yml` to `config/kibana.yml`

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 6:30pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/6 "2019-01-09T18:30:25Z")

</div>

In a kubernetes enviroment, where should i add it ?

---

<div class="post-metadata">

**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\
**Post date:** [January 9, 2019, 6:32pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/7 "2019-01-09T18:32:18Z")

</div>

Whoops... `config.kibana.yml` was suppose to be `config/kibana.yml`. It should go in your Kibana config file, where ever that's being read from. I'm not very familiar with how Kibana is deployed via Kubernetes.

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 6:38pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/8 "2019-01-09T18:38:04Z")

</div>

Okay for future reference, it should be done with config maps in a kubernetes enviroment:

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana
  namespace: the-project
  labels:
    app: kibana
data:
  # kibana.yml is mounted into the Kibana container
  # see https://github.com/elastic/kibana/blob/master/config/kibana.yml
  # Kubernetes Ingress is used to route kib.the-project.d4ldev.txn2.com
  kibana.yml: |-
    server.name: kib.the-project.d4ldev.txn2.com
    server.host: "0"
    elasticsearch.url: http://elasticsearch:9200
    xpack.infra.sources.default.fields.message: ['message', '@message', 'json.message']
```

---

<div class="post-metadata">

**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\
**Post date:** [January 9, 2019, 6:39pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/9 "2019-01-09T18:39:02Z")

</div>

Awesome! Did that config setting work for you?

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 6:48pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/10 "2019-01-09T18:48:46Z")

</div>

> [@paltaa](#):
>
> ```auto
> xpack.infra.sources.default.fields.message: ['message', '@message', 'json.message']
> 
> ```

Im testing, will get back to you very soon

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 7:11pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/11 "2019-01-09T19:11:38Z")

</div>

Nope, it didnt work

---

<div class="post-metadata">

**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\
**Post date:** [January 9, 2019, 7:15pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/12 "2019-01-09T19:15:46Z")

</div>

Bummer... Isn't the log message located at `json.message` relative to the `_source` attribute? I'm assuming you restarted Kibana? just double checking 😃

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 7:20pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/13 "2019-01-09T19:20:06Z")

</div>

Yes kibana restarted, anyways as the log is from kibana itself and just api logs that are not entirely important, could it be possible to leave them out, or maybe do changes in the filebeat config ?

---

<div class="post-metadata">

**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\
**Post date:** [January 9, 2019, 7:24pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/14 "2019-01-09T19:24:50Z")

</div>

It should work out of the box ☹ I was hoping that quick fix would do it but it looks like I'm gonna have to roll my sleeves up. I'll set up the same Filebeat importer on my system and get back to you in a bit. Either I will be able to get it working (as described above) or I will have to open a PR specifically for this use case. I will leave an update here either way 👍

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 7:28pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/15 "2019-01-09T19:28:07Z")

</div>

thanks alot!

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 9, 2019, 7:32pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/16 "2019-01-09T19:32:37Z")

</div>

wait a second i missed something, ill get back to you in a few minutes

---

<div class="post-metadata">

**Author:** ![paltaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paltaa/32/82692_2.png) [@paltaa](https://discuss.elastic.co/u/paltaa)\
**Post date:** [January 24, 2019, 2:21pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/17 "2019-01-24T14:21:57Z")

</div>

Problem solved, so desactivating kibana loggin fix this. Probably not the best approach but works.

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana
  namespace: whitenfv
  labels:
    app: kibana
data:
  # kibana.yml is mounted into the Kibana container
  # see https://github.com/elastic/kibana/blob/master/config/kibana.yml
  # Kubernetes Ingress is used to route kib.the-project.d4ldev.txn2.com
  kibana.yml: |-
    xpack.infra.sources.default.fields.message: ['message', '@message', 'json.message']
    server.name: kibana
    server.host: "0"
    xpack.monitoring.ui.container.elasticsearch.enabled: true
    logging.silent: true
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2019, 2:21pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-kubernetes-enviroment/163558/18 "2019-02-21T14:21:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.