Filebeat Exception - Cannot Get Text on a Start Object

[tarpanpathak]

This is my first post so hello.

I'm trying to debug a Filebeat exception error but not making much progress.

We are running Filebeat through a Helm chart and without changes to the environment are seeing the following messages across all our environments:

2018/05/30 00:22:19.159723 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:225"}}
2018/05/30 00:22:19.159734 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:301"}}
2018/05/30 00:22:19.159748 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:260"}}

The Helm chart version is: filebeat-0.1.2

Here is a link to the chart: https://github.com/kubernetes/charts/tree/master/stable/filebeat

Additionally, here are the configurations being used:

1. filebeat-chart.yml:

# Default values for filebeat.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
elasticsearch:
  host: <sensitive>
  port: 443
  username:
  password:

apiVersion: "extensions/v1beta1"

# Set the daemonset api version ^
daemonSetVersion: "extensions/v1beta1"

nameSpace: kube-system
rbac:
  create: true
  serviceAccountName: default

daemonset:
  create: true

2. filebeat-config.yml:

apiVersion: v1
data:
  filebeat.yml: "filebeat.config:\n  prospectors:\n    # Mounted `filebeat-prospectors`
    configmap:\n    path: ${path.config}/prospectors.d/*.yml\n    # Reload prospectors
    configs as they change:\n    reload.enabled: false\n  modules:\n    path: ${path.config}/modules.d/*.yml\n
    \   # Reload module configs as they change:\n    reload.enabled: false\n\nprocessors:\n
    \ - add_cloud_metadata:\n\ncloud.id: ${ELASTIC_CLOUD_ID}\ncloud.auth: ${ELASTIC_CLOUD_AUTH}\n\noutput.elasticsearch:\n
    \ hosts: ['<sensitive>']\n
    \ username: \n  password:"
kind: ConfigMap
metadata:
  creationTimestamp: 2018-04-17T00:22:56Z
  labels:
    app: filebeat
    chart: filebeat-0.1.2
    heritage: Tiller
    release: filebeat
  name: filebeat-config
  namespace: kube-system
  resourceVersion: "<sensitive>"
  selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-config
  uid: <sensitive>

3. filebeat-prospector.yaml:

apiVersion: v1
data:
  kubernetes.yml: |-
    - type: log
      paths:
        - /var/lib/docker/containers/*/*.log
      json.message_key: log
      json.keys_under_root: true
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            namespace: ${POD_NAMESPACE}
        - decode_json_fields:
            fields: ["log"]
      output.console.pretty: true
kind: ConfigMap
metadata:
  creationTimestamp: 2018-04-17T00:22:56Z
  labels:
    app: filebeat
    chart: filebeat-0.1.2
    heritage: Tiller
    release: filebeat
  name: filebeat-prospectors
  namespace: kube-system
  resourceVersion: "<sensitive>"
  selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-prospectors
  uid: <sensitive>

Any help/response is greatly appreciated.

Let me know if more details are required to help you understand this issue.

[exekias (Carlos Pérez Aradros)]

Hi @tarpanpathak,

From what I see, that version is not taking benefit from the docker prospector, so your log messages end up under the log field, instead of the expected message. I see that newer versions are correct: https://github.com/kubernetes/charts/blob/master/stable/filebeat/values.yaml#L26

Any reason to use such and old version?

Best regards

[tarpanpathak]

Hi @exekias,

Thx for pointing this out. To answer your question, no, the old/current version has been working until this issue so we have been using it.

Do you suggest upgrading to the latest (chart) version and testing/monitoring?

[tarpanpathak]

Update, I misspoke earlier, we are actually using Filebeat version 6.0.1 but still seeing these errors.

Any thoughts on why?