Index Object structure

[volkerfrank]

Hi,

how can I index a document with this fields to an index?

..
                    "gitlab": {
                        "path": "/api/v4/jobs/request",
                        "method": "POST",
                        "params": [
                            {
                                "value": "[FILTERED]",
                                "key": "token"
                            },
                            {
                                "value": {
                                    "features": {
                                        "vault_secrets": "[FILTERED]",
                                        "artifacts_exclude": null
                                    },
                                    "shell": "bash",
                                    "executor": "docker",
                                    "name": "gitlab-runner",
                                    "version": "16.3.1",
                                    "config": {
                                        "gpus": ""
                                    },
                                    "platform": "linux",
                                    "revision": "f5dfa4d1",
                                    "architecture": "amd64"
                                },
                                "key": "info"
                            }
                        ],
...

When I try to index a document with this part I get the error:

(status = 400): {
    "type": "document_parsing_exception",
    "reason": "[1:5992] failed to parse field [gitlab.params.value] of type [keyword] in document with id 'E7OZhosB5SWzhhQTYN73'. Preview of field's value: '...,
    "caused_by": {
        "type": "illegal_state_exception",
        "reason": "Can't get text on a START_OBJECT at 1:5258"
    }
}, dropping event!

I think the problem is the object structure that is sometimes under gitlab.params.value.
Any idea to solve this?

[azizim (Abdelaziz IMGHARNE)]

The problem is that elastic is considering gitlab.params.value as keyword.
You might want to change to an object in your mapping or you might consider using ingest pipeline to deal with cases where the field is a string .

[volkerfrank]

OK. The problem is that the field value is sometimes a string and sometimes an object.
When I change it to object in the mapping, then the strings cannot be indexed.
Any idea how I can handle this in a pipeline?

[volkerfrank]

Thanks for the hint.
I have found the following solution. I convert the Object to String with this processor:

  {
    "foreach": {
      "field": "gitlab.params",
      "processor": {
        "convert": {
          "field": "_ingest._value.value",
          "type": "string"
        }
      },
      "description": "convert params.value to String"
    }
  }