Ingest pipeline not finding field

Hi there,

When I test my ingest pipeline, which replaces a delimiter with a whitespace, with a document that has the exact field I'm trying to transform, I keep getting this error: [Field [field] not present as part of path [field.query] for Elasticsearch ingest pipeline]. I'm not understanding why the field would not be present. Using ignore_missing or ignore_failure just skips over the field completely as if it's not there.

Any help would be appreciated.
Thanks

It would be easier if you share:

  • Your pipeline
  • A sample document

Event better, if you could share a simple _simulate call, like:

POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "set" : {
          "field" : "field2",
          "value" : "_value"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}
POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "processors": [
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": "\\|x\\|",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "my_index",
      "_id": "1",
      "_version": 1,
      "_score": 0,
      "_ignored": [
        "message.keyword"
      ],
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    }
  ]
}

Pipelines do not support "flattened" objects today

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": """\|x\|""",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    },
    {
      "_source": {
        "foo": {
          "bar": {
            "foobar": "foo|x|bar"
          }
        }
      }
    }
  ]
}

#results

{
  "docs": [
    {
      "error": {
        "root_cause": [
          {
            "type": "illegal_argument_exception",
            "reason": "field [foo] not present as part of path [foo.bar.foobar]"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "field [foo] not present as part of path [foo.bar.foobar]"
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T17:55:13.057682074Z"
        }
      }
    }
  ]
}

if you actually have that syntax then you will need to dot_expander

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "dot_expander": {
          "field": "foo.bar.foobar"
        }
      },
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": """\|x\|""",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    },
    {
      "_source": {
        "foo": {
          "bar": {
            "foobar": "foo|x|bar"
          }
        }
      }
    }
  ]
}

# results

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T18:00:41.097144482Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T18:00:41.097175694Z"
        }
      }
    }
  ]
}
1 Like

I was wondering if that might be the case. Unfortunately, the actually document has many foo.bar* fields so I'm not sure if I'd need to expand all of them or not. Or perhaps would this automatically expand all of them?

Nevermind, I tried it out and it only expands the one field. This makes the pipeline execute and the gsub processor work. Thank you very much!

1 Like

@stephanb For some reason, the simulation works, but the pipeline does not work upon ingestion. I've tried many different combinations of ignore_failure, ignore_missing, and override with the two processors but the foo.bar.foobar field looks untouched in the documents.

@emi_rose

You need to show us an end to end repeatable example

  • Sample Documents
  • Sample Pipeline
  • Sample Mapping (schema) are you creating one?
  • Execute to ingest / write the document and the failed result.
  • Show all the commands and all the result
  • otherwise we can not help.

The writing most likely fails because the document being written does not match the expected mapping (think schema)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.