Dynamic field type in template

[anna.kuranda (Anna)]

I use template and alias to manage my insert data
The issue :
I have field that have different types , so I failed on field type inconsistency
the field can be of String type or Nested JSON Object
For example :
String type
"descr":"hello word"

Object type
"descr":{
"user" : "anna",
"account":{
"id":33333,
"det":"bla"
...........
}
}

Please ,is there way to solve the issue with dynamic field type

[Christian_Dahlqvist]

Dynamic mapping just means it is assigned when the field is first encountered - the mapping still need to be consistent within an index.

[stephenb (Stephen Brown)]

Hi @anna.kuranda

This is one of the tough little issues with ingesting logs that are different.

Building on what @Christian_Dahlqvist described that in the end the mapping / data type still needs to be consistent below is a pattern that I have seen used

Create a Mapping that contains all the information etc. and then use an ingest pipeline to create the consistent mapping / document. If there is another field in the log that can help you identify the type that will make it easier

Here is a little sample code to get you thinking

DELETE discuss

PUT discuss
{
  "mappings": {
    "properties": {
      "log_type": {
        "type": "keyword"
      },
      "desc": {
        "properties": {
          "general_description": {
            "type": "text"
          },
          "other_data": {
            "type": "text"
          },
          "id": {
            "type": "keyword"
          }
        }
      }
    }
  }
}



POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "string to object",
    "version": 0,
    "processors": [
      {
        "set": {
          "if": "ctx.log_type != null && ctx.log_type == 'type1'",
          "field": "temp_desc",
          "value": "{{{desc}}}"
        }
      },
      {
        "remove": {
          "if": "ctx.log_type != null && ctx.log_type == 'type1'",
          "field": "desc"
        }
      },
      {
        "set": {
          "if": "ctx.log_type != null && ctx.log_type == 'type1'",
          "field": "desc.general_desc",
          "value": "{{{temp_desc}}}"
        }
      },
      {
        "remove": {
          "if": "ctx.log_type != null && ctx.log_type == 'type1'",
          "field": "temp_desc"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "m-index",
      "_id": "kMpUTHoBr7SFhhL5-98P",
      "_source": {
        "log_type": "type1",
        "desc": "my description"
      }
    },
    {
      "_index": "m-index",
      "_id": "kMpUTHoBr7SFhhL5-98P",
      "_source": {
        "desc": {
          "other_data": "my other data"
        }
      }
    }
  ]
}

the result of this is that it can handle both types of data

{
  "docs" : [
    {
      "doc" : {
        "_index" : "m-index",
        "_type" : "_doc",
        "_id" : "kMpUTHoBr7SFhhL5-98P",
        "_source" : {
          "log_type" : "type1",
          "desc" : {
            "general_desc" : "my description"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-11-07T20:15:15.2747358Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "m-index",
        "_type" : "_doc",
        "_id" : "kMpUTHoBr7SFhhL5-98P",
        "_source" : {
          "desc" : {
            "other_data" : "my other data"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-11-07T20:15:15.2747568Z"
        }
      }
    }
  ]
}

[anna.kuranda (Anna)]

Thank you for the answer