# Ingest pipeline not finding field

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [September 29, 2023, 3:21pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144 "2023-09-29T15:21:46Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![emi\_rose](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emi_rose/32/83050_2.png) [@emi\_rose](https://discuss.elastic.co/u/emi_rose)\
**Post date:** [September 29, 2023, 3:21pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/1 "2023-09-29T15:21:46Z")

</div>

Hi there,

When I test my ingest pipeline, which replaces a delimiter with a whitespace, with a document that has the exact field I'm trying to transform, I keep getting this error: [Field [field] not present as part of path [field.query] for Elasticsearch ingest pipeline]. I'm not understanding why the field would not be present. Using ignore\_missing or ignore\_failure just skips over the field completely as if it's not there.

Any help would be appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 29, 2023, 4:27pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/2 "2023-09-29T16:27:47Z")

</div>

It would be easier if you share:

- Your pipeline
- A sample document

Event better, if you could share a simple [`_simulate` call](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html), like:

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "set" : {
          "field" : "field2",
          "value" : "_value"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![emi\_rose](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emi_rose/32/83050_2.png) [@emi\_rose](https://discuss.elastic.co/u/emi_rose)\
**Post date:** [September 29, 2023, 5:23pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/3 "2023-09-29T17:23:54Z")

</div>

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "processors": [
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": "\\|x\\|",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "my_index",
      "_id": "1",
      "_version": 1,
      "_score": 0,
      "_ignored": [
        "message.keyword"
      ],
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 29, 2023, 6:02pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/4 "2023-09-29T18:02:01Z")

</div>

Pipelines do not support "flattened" objects today

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": """\|x\|""",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    },
    {
      "_source": {
        "foo": {
          "bar": {
            "foobar": "foo|x|bar"
          }
        }
      }
    }
  ]
}

#results

{
  "docs": [
    {
      "error": {
        "root_cause": [
          {
            "type": "illegal_argument_exception",
            "reason": "field [foo] not present as part of path [foo.bar.foobar]"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "field [foo] not present as part of path [foo.bar.foobar]"
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T17:55:13.057682074Z"
        }
      }
    }
  ]
}

```

if you actually have that syntax then you will need to `dot_expander `

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "dot_expander": {
          "field": "foo.bar.foobar"
        }
      },
      {
        "gsub": {
          "field": "foo.bar.foobar",
          "pattern": """\|x\|""",
          "replacement": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo.bar.foobar": "foo|x|bar"
      }
    },
    {
      "_source": {
        "foo": {
          "bar": {
            "foobar": "foo|x|bar"
          }
        }
      }
    }
  ]
}

# results

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T18:00:41.097144482Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "foo": {
            "bar": {
              "foobar": "foo bar"
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T18:00:41.097175694Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![emi\_rose](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emi_rose/32/83050_2.png) [@emi\_rose](https://discuss.elastic.co/u/emi_rose)\
**Post date:** [September 29, 2023, 6:09pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/5 "2023-09-29T18:09:28Z")

</div>

I was wondering if that might be the case. Unfortunately, the actually document has many foo.bar\* fields so I'm not sure if I'd need to expand all of them or not. Or perhaps would this automatically expand all of them?

---

<div class="post-metadata">

**Author:** ![emi\_rose](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emi_rose/32/83050_2.png) [@emi\_rose](https://discuss.elastic.co/u/emi_rose)\
**Post date:** [September 29, 2023, 6:14pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/6 "2023-09-29T18:14:51Z")

</div>

Nevermind, I tried it out and it only expands the one field. This makes the pipeline execute and the gsub processor work. Thank you very much!

---

<div class="post-metadata">

**Author:** ![emi\_rose](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emi_rose/32/83050_2.png) [@emi\_rose](https://discuss.elastic.co/u/emi_rose)\
**Post date:** [September 29, 2023, 8:52pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/7 "2023-09-29T20:52:53Z")

</div>

@stephanb For some reason, the simulation works, but the pipeline does not work upon ingestion. I've tried many different combinations of ignore\_failure, ignore\_missing, and override with the two processors but the foo.bar.foobar field looks untouched in the documents.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 29, 2023, 10:08pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/8 "2023-09-29T22:08:26Z")

</div>

@emi_rose

You need to show us an end to end repeatable example

- Sample Documents
- Sample Pipeline
- Sample Mapping (schema) are you creating one?
- Execute to ingest / write the document and the failed result.
- Show all the commands and all the result
- otherwise we can not help.

The writing most likely fails because the document being written does not match the expected mapping (think schema)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2023, 10:09pm UTC](https://discuss.elastic.co/t/ingest-pipeline-not-finding-field/344144/9 "2023-10-27T22:09:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.