This is my first post so hello.
I'm trying to debug a Filebeat exception error but not making much progress.
We are running Filebeat through a Helm chart and without changes to the environment are seeing the following messages across all our environments:
2018/05/30 00:22:19.159723 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:225"}}
2018/05/30 00:22:19.159734 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:301"}}
2018/05/30 00:22:19.159748 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [log]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:260"}}
The Helm chart version is: filebeat-0.1.2
Here is a link to the chart: https://github.com/kubernetes/charts/tree/master/stable/filebeat
Additionally, here are the configurations being used:
- filebeat-chart.yml:
# Default values for filebeat.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
elasticsearch:
host: <sensitive>
port: 443
username:
password:
apiVersion: "extensions/v1beta1"
# Set the daemonset api version ^
daemonSetVersion: "extensions/v1beta1"
nameSpace: kube-system
rbac:
create: true
serviceAccountName: default
daemonset:
create: true
- filebeat-config.yml:
apiVersion: v1
data:
filebeat.yml: "filebeat.config:\n prospectors:\n # Mounted `filebeat-prospectors`
configmap:\n path: ${path.config}/prospectors.d/*.yml\n # Reload prospectors
configs as they change:\n reload.enabled: false\n modules:\n path: ${path.config}/modules.d/*.yml\n
\ # Reload module configs as they change:\n reload.enabled: false\n\nprocessors:\n
\ - add_cloud_metadata:\n\ncloud.id: ${ELASTIC_CLOUD_ID}\ncloud.auth: ${ELASTIC_CLOUD_AUTH}\n\noutput.elasticsearch:\n
\ hosts: ['<sensitive>']\n
\ username: \n password:"
kind: ConfigMap
metadata:
creationTimestamp: 2018-04-17T00:22:56Z
labels:
app: filebeat
chart: filebeat-0.1.2
heritage: Tiller
release: filebeat
name: filebeat-config
namespace: kube-system
resourceVersion: "<sensitive>"
selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-config
uid: <sensitive>
- filebeat-prospector.yaml:
apiVersion: v1
data:
kubernetes.yml: |-
- type: log
paths:
- /var/lib/docker/containers/*/*.log
json.message_key: log
json.keys_under_root: true
processors:
- add_kubernetes_metadata:
in_cluster: true
namespace: ${POD_NAMESPACE}
- decode_json_fields:
fields: ["log"]
output.console.pretty: true
kind: ConfigMap
metadata:
creationTimestamp: 2018-04-17T00:22:56Z
labels:
app: filebeat
chart: filebeat-0.1.2
heritage: Tiller
release: filebeat
name: filebeat-prospectors
namespace: kube-system
resourceVersion: "<sensitive>"
selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-prospectors
uid: <sensitive>
Any help/response is greatly appreciated.
Let me know if more details are required to help you understand this issue.