Mapper_parsing_exception when trying to append object to array

Elastic Stack Elasticsearch
1

Hello,

I'm having trouble to append objects resulting from an enrich processor to a field, so that I'm ending up with an array of objects. This question seems to have been asked a lot but still I couldn't make it work so far :confused:
What does work is to append the object (resulting from the enrich process) as flattened, i.e. as string, which gives me an array of stringified objects. But that's useless as I need to be able to search and aggregate that data...

The error is:

"error" : {
          "type" : "mapper_parsing_exception",
          "reason" : "object mapping for [templates] tried to parse field [null] as object, but found a concrete value"
        }

Index template:

{
  "mappings": {
    "_doc": {
      "dynamic": "true",
      "properties": {
        "templates": {
          "type": "nested",
          "dynamic": "true"
        }
      }
    }
  }
}

Pipeline:
(This pipeline processes values of an array in the ingested document as processor of a foreach processor.)

{
  "processors": [
    {
      "set": {
        "field": "tempId",
        "value": "{{{_ingest._value}}}"
      }
    },
    {
      "enrich": {
        "field": "tempId",
        "ignore_missing": true,
        "max_matches": "1",
        "policy_name": "enrich-stores",
        "target_field": "tempTp"
      }
    },
    {
      "append": {
        "field": "templates",
        "value": "{{{tempTp}}}"
      }
    },
    {
      "remove": {
        "field": [
          "tempId",
          "tempTp"
        ],
        "ignore_missing": true
      }
    }
  ]
}

Any leads would be highly appreciated, thank you!

2

Can you explain why you set the mapping to nested for the templates field? It looks as if your field looks like "templates":["some_value"] - the nested would be. wrong configuration...

Any chance you can provide a more exact reproduction if the above is not the issue?

3

Thanks for your quick reply!

The field is supposed to look as follows in a document context:

  "templates": [
    {
      "id": "a1234",
      "name": "Alain",
      "type": "italic"
    },
    {
      "id": "b297",
      "name": "Sylvia",
      "type": "bold"
    }
  ]

Each object is the result of the enrich process that resolves one or more ids against documents in another index. Does this make sense and clarifies the problem?

If I don't use an index template (mapping), then the process results in this flattened structure:

"templates": [
      "{name=Alain, type=italic, id=a1234}",
      "{name=Sylvia, type=bold, id=b297}"
    ],

What I would like to achieve is to have the above stringified objects as actual objects in the array...

4

Can you share the full output of a Simulate Ingest API with a sample document?

5

Apparently, the simulate ingest API does not consider index mappings of docs._index it is simulating, so the mapper_parsing_exception does not occur here. But you see the stringified 'templates' objects I would like to have inserted as objects...

POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "processors": [
      {
        "foreach": {
          "field": "tpA",
          "ignore_missing": true,
          "processor": {
            "pipeline": {
              "name": "react-v3-cms-tp-enrich"
            }
          }
        }
      },
      {
        "foreach": {
          "field": "tpB",
          "ignore_missing": true,
          "processor": {
            "pipeline": {
              "name": "react-v3-cms-tp-enrich"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "react-v3-cms-stores",
      "_source": {
        "tpA": [
          "Bqom9CUHSE"
        ],
        "tpB": [
          "CnLKuLGM8m"
        ],
        "name": "Werk 1",
        "identifier": "PROD_1",
        "version": 28,
        "createdAt": "2020-09-22T08:33:55.110Z",
        "updatedAt": "2021-03-05T15:54:45.154Z",
        "objectId": "k0xnrriDR5"
      }
    },
    {
      "_index": "react-v3-cms-stores",
      "_source": {
        "tpA": [],
        "tpB": [
          "TT96cE7tEh",
          "XePVWmFA3y"
        ],
        "name": "Porter",
        "identifier": "PORTER",
        "version": 69,
        "createdAt": "2021-04-13T14:37:22.540Z",
        "updatedAt": "2021-07-13T10:49:51.255Z",
        "objectId": "v6pyKuWw14"
      }
    }
  ]
}

Output:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "react-v3-cms-stores",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "identifier" : "PROD_1",
          "createdAt" : "2020-09-22T08:33:55.110Z",
          "tpA" : [
            "Bqom9CUHSE"
          ],
          "tpB" : [
            "CnLKuLGM8m"
          ],
          "templates" : [
            "{name=Simple Com, type=tpA, objectId=Bqom9CUHSE}",
            "{name=MQTT, type=tpB, objectId=CnLKuLGM8m}"
          ],
          "name" : "Werk 1",
          "version" : 28,
          "objectId" : "k0xnrriDR5",
          "updatedAt" : "2021-03-05T15:54:45.154Z"
        },
        "_ingest" : {
          "_value" : null,
          "timestamp" : "2021-07-20T09:17:03.409570925Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "react-v3-cms-stores",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "identifier" : "PORTER",
          "createdAt" : "2021-04-13T14:37:22.540Z",
          "tpA" : [ ],
          "tpB" : [
            "TT96cE7tEh",
            "XePVWmFA3y"
          ],
          "templates" : [
            "{name=Paul, type=tpB, objectId=TT96cE7tEh}",
            "{name=Tags, type=tpB, objectId=XePVWmFA3y}"
          ],
          "name" : "Porter",
          "version" : 69,
          "objectId" : "v6pyKuWw14",
          "updatedAt" : "2021-07-13T10:49:51.255Z"
        },
        "_ingest" : {
          "_value" : null,
          "timestamp" : "2021-07-20T09:17:03.409578492Z"
        }
      }
    }
  ]
}

The referenced sub pipeline react-v3-cms-tp-enrich:

{
  "description": "react-v3-cms-tp-enrich",
  "processors": [
    {
      "set": {
        "field": "tempId",
        "value": "{{{_ingest._value}}}"
      }
    },
    {
      "enrich": {
        "field": "tempId",
        "ignore_missing": true,
        "max_matches": "1",
        "policy_name": "react-v3-cms-stores",
        "target_field": "tempTp"
      }
    },
    {
      "append": {
        "field": "templates",
        "value": [
          "{{{tempTp}}}"
        ]
      }
    },
    {
      "remove": {
        "field": [
          "tempId",
          "tempTp"
        ],
        "ignore_missing": true
      }
    }
  ]
}

The referenced enrich policy react-v3-cms-stores:

{
  "match": {
    "indices": "react-v3-cms-templates",
    "match_field": "objectId",
    "enrich_fields": ["type", "name"]
  }
}

So the problem I'm having is, that as soon as I run an ingest into an index where 'templates' has a mapping to type 'nested', a mapper_parsing_error is thrown (see below). Type 'nested' is chosen in order for the template objects to be appended in the 'templates' field (array) as proper, searchable objects instead of flattened strings (as seen above).

{
  "took" : 17,
  "ingest_took" : 11,
  "errors" : true,
  "items" : [
    {
      "index" : {
        "_index" : "react-v3-cms-stores",
        "_type" : "_doc",
        "_id" : "qChFw3oB7liRmCO0HSjD",
        "status" : 400,
        "error" : {
          "type" : "mapper_parsing_exception",
          "reason" : "object mapping for [templates] tried to parse field [null] as object, but found a concrete value"
        }
      }
    },
    ...
  ]
}
6

Found a way to achieve what I needed, although technically it is not a solution to the mapper_parsing_exception – rather side-stepping it: Instead of using an append processor to append objects to an array of objects, I use the following simple script processor, which is doing the same (apparently different though)

{
  "script": {
    "lang": "painless",
    "source": "if (ctx.templates == null) ctx.templates = []; ctx.templates.add(ctx.tempTp)"
  }
}

Result for the same _simulate call from the post above:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "react-v3-cms-stores",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "identifier" : "PROD_1",
          "createdAt" : "2020-09-22T08:33:55.110Z",
          "tpA" : [
            "Bqom9CUHSE"
          ],
          "tpB" : [
            "CnLKuLGM8m"
          ],
          "templates" : [
            {
              "name" : "Simple Com",
              "type" : "tpA",
              "objectId" : "Bqom9CUHSE"
            },
            {
              "name" : "MQTT",
              "type" : "tpB",
              "objectId" : "CnLKuLGM8m"
            }
          ],
          "name" : "Werk 1",
          "version" : 28,
          "objectId" : "k0xnrriDR5",
          "updatedAt" : "2021-03-05T15:54:45.154Z"
        },
        "_ingest" : {
          "_value" : null,
          "timestamp" : "2021-07-21T09:46:11.211535671Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "react-v3-cms-stores",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "identifier" : "PORTER",
          "createdAt" : "2021-04-13T14:37:22.540Z",
          "tpA" : [ ],
          "tpB" : [
            "TT96cE7tEh",
            "XePVWmFA3y"
          ],
          "templates" : [
            {
              "name" : "Paul",
              "type" : "tpB",
              "objectId" : "TT96cE7tEh"
            },
            {
              "name" : "Tags",
              "type" : "tpB",
              "objectId" : "XePVWmFA3y"
            }
          ],
          "name" : "Porter",
          "version" : 69,
          "objectId" : "v6pyKuWw14",
          "updatedAt" : "2021-07-13T10:49:51.255Z"
        },
        "_ingest" : {
          "_value" : null,
          "timestamp" : "2021-07-21T09:46:11.211549618Z"
        }
      }
    }
  ]
}
1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.