Filebeat Exception - Cannot Get Text on a Start Object

This is my first post so hello.

I'm trying to debug a Filebeat exception error but not making much progress.

We are running Filebeat through a Helm chart and without changes to the environment are seeing the following messages across all our environments:

2018/05/30 00:22:19.159723 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:225"}}
2018/05/30 00:22:19.159734 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:301"}}
2018/05/30 00:22:19.159748 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:260"}}

The Helm chart version is: filebeat-0.1.2

Here is a link to the chart: https://github.com/kubernetes/charts/tree/master/stable/filebeat

Additionally, here are the configurations being used:

  1. filebeat-chart.yml:
# Default values for filebeat.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
elasticsearch:
  host: <sensitive>
  port: 443
  username:
  password:

apiVersion: "extensions/v1beta1"

# Set the daemonset api version ^
daemonSetVersion: "extensions/v1beta1"

nameSpace: kube-system
rbac:
  create: true
  serviceAccountName: default

daemonset:
  create: true
  1. filebeat-config.yml:
apiVersion: v1
data:
  filebeat.yml: "filebeat.config:\n  prospectors:\n    # Mounted `filebeat-prospectors`
    configmap:\n    path: ${path.config}/prospectors.d/*.yml\n    # Reload prospectors
    configs as they change:\n    reload.enabled: false\n  modules:\n    path: ${path.config}/modules.d/*.yml\n
    \   # Reload module configs as they change:\n    reload.enabled: false\n\nprocessors:\n
    \ - add_cloud_metadata:\n\ncloud.id: ${ELASTIC_CLOUD_ID}\ncloud.auth: ${ELASTIC_CLOUD_AUTH}\n\noutput.elasticsearch:\n
    \ hosts: ['<sensitive>']\n
    \ username: \n  password:"
kind: ConfigMap
metadata:
  creationTimestamp: 2018-04-17T00:22:56Z
  labels:
    app: filebeat
    chart: filebeat-0.1.2
    heritage: Tiller
    release: filebeat
  name: filebeat-config
  namespace: kube-system
  resourceVersion: "<sensitive>"
  selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-config
  uid: <sensitive>
  1. filebeat-prospector.yaml:
apiVersion: v1
data:
  kubernetes.yml: |-
    - type: log
      paths:
        - /var/lib/docker/containers/*/*.log
      json.message_key: log
      json.keys_under_root: true
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            namespace: ${POD_NAMESPACE}
        - decode_json_fields:
            fields: ["log"]
      output.console.pretty: true
kind: ConfigMap
metadata:
  creationTimestamp: 2018-04-17T00:22:56Z
  labels:
    app: filebeat
    chart: filebeat-0.1.2
    heritage: Tiller
    release: filebeat
  name: filebeat-prospectors
  namespace: kube-system
  resourceVersion: "<sensitive>"
  selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-prospectors
  uid: <sensitive>

Any help/response is greatly appreciated.

Let me know if more details are required to help you understand this issue.

1 Like

Hi @tarpanpathak,

From what I see, that version is not taking benefit from the docker prospector, so your log messages end up under the log field, instead of the expected message. I see that newer versions are correct: https://github.com/kubernetes/charts/blob/master/stable/filebeat/values.yaml#L26

Any reason to use such and old version?

Best regards

Hi @exekias,

Thx for pointing this out. To answer your question, no, the old/current version has been working until this issue so we have been using it.

Do you suggest upgrading to the latest (chart) version and testing/monitoring?

Update, I misspoke earlier, we are actually using Filebeat version 6.0.1 but still seeing these errors.

Any thoughts on why?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.