Inconsistency in Index Type Causing Document Parsing Exception

[HemdeepSaini]

In some of the indices, the field resilience4j_circuitbreaker_calls.histogram is of type double, while in others, it is of type histogram. This inconsistency is causing the following error in the APM Server logs:

{"log.level":"error","@timestamp":"2024-03-08T21:59:44.085+0530","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (document_parsing_exception): [1:480] object mapping for [resilience4j_circuitbreaker_calls] tried to parse field [resilience4j_circuitbreaker_calls] as object, but found a concrete value","service.name":"apm-server","ecs.version":"1.6.0"}

Kibana version: 8.8.1
Elasticsearch version: 8.8.0
APM Server version: 8.8.1

This inconsistency is causing document parsing exceptions in APM server, leading to operational issues. We need to address this inconsistency to ensure smooth functioning of the APM Server. Any insights or suggestions on how to resolve this issue would be greatly appreciated.

[San9 (Alex)]

Hi, I have a similar problem, but my case is more complex.
Regarding your problem: check in the data viewer for this index what type of field is used. and check the template for a given index to see how it defines the fields - dynamically or you define them yourself.
As far as I understand, when receiving logs for the first time, Elastic will determine the field type as numeric, text, or object. Then, as I understand it, when the same field comes with a new field type, a conflict arises. If you are sure that your field always has the same format, then you can specify the required text or object format in the index template. BUT THIS ONLY WORKS IF THE FIELD FORMAT IS CONSTANT and not variable. (as in my case - when the data in a field can be text in one case, and a log object in the next... then the problem remains and here you need elastic experts to come to the rescue)