I am sending filebeat output to elastic search. In filebeat logs, can see mapper parser error. trying to load dynamic template manually (through postman) but still getting error. filebeat is on K8S cluster and ES is hosted on cloud.
Here is the filbeat required config and dynamic template.
Prospectors:
- type: log
paths:
- /var/lib/docker/containers//.log
- /var/data/kubeletlogs///*.log#
json.message_key: log
json.keys_under_root: true
logging.files.keepfiles: 7
fields:
clustername: {{ .Values.clustername }}
Processors:
processors:
- add_kubernetes_metadata:
in_cluster: true
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 10
index: "app-log-6.5.4-%{+YYYY.MM.dd}"
setup.template:
name: "max-log"
pattern: "max-log-*"
enabled: true
fields: fields.yml
overwrite: true
Dynamic template:
{
"template" : "app-*",
"order" : 1,
"settings" : {
},
"mappings": {
"_default_": {
"dynamic_templates": [{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"type": "text",
"norms": false
}
}
}, {
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"location": {
"type": "geo_point"
},
"latitude": {
"type": "half_float"
},
"longitude": {
"type": "half_float"
}
}
}
}
}
}
}
- Deleting indices from ES 2)loading template through postman (PUT req) 3) starting filebeat.
Error: {"type":"mapper_parsing_exception","reason":"failed to parse field [message] of type [text] in document with id 'sU9P4mwBlXscRF4xIyDU'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:191"}}
What mapping I am doing wrong/missing?