Elasticsearch rejecting records because of parsing error

[vijayakrishna.rg (Krishna)]

Elastic version: 6.7.0
Kibana version: 6.7.0
Source to send data to elasticsearch: kubernetes, fluentd

I am having an issue while parsing message and forwarding from kubernetes cluster to elasticsearch

it's rejecting all the records because of parsing issue, almost 1 million records got rejected by elasticsearch.

added below parameter into template for index:

"index.mapping.ignore_malformed": true

ISSUE:

[2019-05-07T17:52:22,596][DEBUG][o.e.a.b.TransportShardBulkAction] [elastic1] [k8s-pi-2019.05.07][2] failed to execute bulk item (index) index {[k8s-pi-2019.05.07][fluentd][rnZsk2oBoyEjHE85xxtC], source[{"@severity":"DEBUG","@timestamp":"2019-05-07T17:52:17.537Z","@traceId":["xxx-4168-9b4c-7f17d8f6d025","PROCESSOR-CONSUME-0"],"class":"com.commons.stream.eventhubs.ProcessorComponent","file":"ProcessorComponent","line":85,"msg":{"@msg":"Consumed a message from an EventHubs queue","EventAction":{"BAD1":"EventAction(addressCreated)"},"app":"processor-update","nodeIp":"10.10.10.10","nodeName":"","podName":"processor-update-xxx-2s2qd","region":"japan"},"log":"{"@severity":"DEBUG","@timestamp":"2019-05-07T17:52:17.537Z","@traceId":["4147717c-bf50-4168-asrd-7f17d8f6d025","PROCESSOR-CONSUME-0"],"class":"com.commons.stream.eventhubs.ProcessorComponent","file":"ProcessorComponent","line":85,"msg":{"@msg":"Consumed a message from an EventHubs queue","EventAction":{"BAD1":"EventAction(addressCreated)"},"app":"processor-update","nodeIp":"10.10.10.10","nodeName":"","podName":"processor-update-86cc9687bf-xxxx","region":"japan"}}\n","stream":"stdout","docker":{"container_id":"2d9b63f5234234sdfsd35sdfsdfdf0da010da173806cb4ff2222b8754d2644f073b"},"kubernetes":{"container_name":"processor-update","namespace_name":"xx","pod_name":"processor-update-86cc9687bf-ggtgg","pod_id":"ae262158-xxx-11e9-xxx-000d3a34cdc9","labels":{"app":"processor-update","checksum/config":"6723432404290xxxxxcdd0440f5831ce07cbaf0faa42bc7c","pod-template-hash":"4277524369"},"host":"test-worker-1","master_url":"https://10.10.10.10:443/api"}}]}

org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [msg.EventAction] of type [text] in document with id 'rnZsk2oBoyEjHE85zrtC'

i have included malformed_ignore parameter while creating index, still getting an above issue.

is there a way we can include any kind of mapping in index template to ignore above issue at least [msg.EventAction] field., Please suggest me, Thanks.

[Christian_Dahlqvist]

If you look in the documentation there are limits to how this can be used when it comes to JSON objects, which is the case you seem to have here as the field is mapped as string and you send in an object.

[vijayakrishna.rg (Krishna)]

Hi @Christian_Dahlqvist, i have update index settings with "ignore_malformed" : "true", i think it supposed to ignore these mapping errors if we set it up, but still am getting above issue, any suggestion?

GET k8s-pi-2019.05.21/_settings

{
"k8s-pi-2019.05.21" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "k8s-logs",
"rollover_alias" : "k8s"
},
"mapping" : {
"total_fields" : {
"limit" : "10000"
},
"ignore_malformed" : "true"
},
"number_of_shards" : "5",
"provided_name" : "k8s-pi-2019.05.21",
"creation_date" : "1558396801813",
"number_of_replicas" : "1",
"uuid" : "7Igc6Z-cRmGlxHKzOf10Aw",
"version" : {
"created" : "6070099"
}
}
}
}
}

[Christian_Dahlqvist]

The error you showed earlier is not covered by the ignore malformed flag. This is described in the docs I linked to.