Ingest pipeline not finding field

[emi_rose (user858708178411)]

Hi there,

When I test my ingest pipeline, which replaces a delimiter with a whitespace, with a document that has the exact field I'm trying to transform, I keep getting this error: [Field [field] not present as part of path [field.query] for Elasticsearch ingest pipeline]. I'm not understanding why the field would not be present. Using ignore_missing or ignore_failure just skips over the field completely as if it's not there.

Any help would be appreciated.
Thanks

[dadoonet (David Pilato)]

It would be easier if you share:

• Your pipeline
• A sample document

Event better, if you could share a simple _simulate call, like:

POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "set" : {
          "field" : "field2",
          "value" : "_value"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}

[emi_rose (user858708178411)]

POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "processors": [
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": "\\|x\\|",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "my_index",
      "_id": "1",
      "_version": 1,
      "_score": 0,
      "_ignored": [
        "message.keyword"
      ],
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    }
  ]
}

[stephenb (Stephen Brown)]

Pipelines do not support "flattened" objects today

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": """\|x\|""",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    },
    {
      "_source": {
        "foo": {
          "bar": {
            "foobar": "foo|x|bar"
          }
        }
      }
    }
  ]
}

#results

{
  "docs": [
    {
      "error": {
        "root_cause": [
          {
            "type": "illegal_argument_exception",
            "reason": "field [foo] not present as part of path [foo.bar.foobar]"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "field [foo] not present as part of path [foo.bar.foobar]"
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T17:55:13.057682074Z"
        }
      }
    }
  ]
}

if you actually have that syntax then you will need to dot_expander

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "dot_expander": {
          "field": "foo.bar.foobar"
        }
      },
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": """\|x\|""",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    },
    {
      "_source": {
        "foo": {
          "bar": {
            "foobar": "foo|x|bar"
          }
        }
      }
    }
  ]
}

# results

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T18:00:41.097144482Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T18:00:41.097175694Z"
        }
      }
    }
  ]
}

[emi_rose (user858708178411)]

I was wondering if that might be the case. Unfortunately, the actually document has many foo.bar* fields so I'm not sure if I'd need to expand all of them or not. Or perhaps would this automatically expand all of them?

[emi_rose (user858708178411)]

Nevermind, I tried it out and it only expands the one field. This makes the pipeline execute and the gsub processor work. Thank you very much!

[emi_rose (user858708178411)]

@stephanb For some reason, the simulation works, but the pipeline does not work upon ingestion. I've tried many different combinations of ignore_failure, ignore_missing, and override with the two processors but the foo.bar.foobar field looks untouched in the documents.

[stephenb (Stephen Brown)]

@emi_rose

You need to show us an end to end repeatable example

• Sample Documents
• Sample Pipeline
• Sample Mapping (schema) are you creating one?
• Execute to ingest / write the document and the failed result.
• Show all the commands and all the result
• otherwise we can not help.

The writing most likely fails because the document being written does not match the expected mapping (think schema)