Elasticsearch mapper_parsing_exception for field appearing both in text and object form

[dory]

Hello,

I'm quite new to Elasticsearch and I'm trying since a few days to fix some indexing problems we have in our indices. The cause of this errors is that we have multiple applications logging data which finally ends up in the same index and some of these applications use different formats for the same fields. For example, an application logs "message" as a simple text field and other one as a JSON object field. Because we are using dynamic mapping, depending on which field gets indexed first, the mapping for the "message" field is saved to the index mappings as either (for JSON object, for instance):

"message" : {
          "properties" : {
            "key" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }

or as (for text):

"message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }

When Elasticsearch tries then to index the field “message” in the other form, we get:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [message] of type [text] in document with id 'UX1v5H4BdDgf7Zt3AJCj'. Preview of field's value: '{key=b}'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [message] of type [text] in document with id 'UX1v5H4BdDgf7Zt3AJCj'. Preview of field's value: '{key=b}'",
    "caused_by" : {
      "type" : "illegal_state_exception",
      "reason" : "Can't get text on a START_OBJECT at 2:14"
    }
  },
  "status" : 400
}

I tried the solution with dynamic templates from this stackoverflow post , which looked to nice and logical, but did not worked for me (I'm using Elastic 7.16.0).

Then I tried to save all fields as runtime fields by setting

"mappings": {
    "dynamic": "runtime"
  }

on my index and I was not getting the exception anymore. But I have the feeling that this would be a very bad idea because the query time will increase enormously , as all fields would be calculated at query time and not saved to index.

Do you guys have any idea how to solve this problem? How can we take advantage of dynamic field mapping ("dynamic":"true") in Elasticsearch but still be able to index a field appearing both in JSON object form and text form?

Thanks in advance!

[Tomo_M (Tomohiro Mitani)]

The field type should be consistent in a single index.
If the field structure is different, you need to create multiple indices with different mappings...

@dory

The cause of this errors is that we have multiple applications logging data which finally ends up in the same index and some of these applications use different formats for the same fields.

As you said exactly, this is the cause and you need to devide the logging data to different indices.

[dory]

Hi Tomo!

Thank you for your answer! We'll try to define separate indices.

Kind regards,
Maria