# Pipeline remove processor not removing all fields

**URL:** <https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879>\
**Category:** Elasticsearch\
**Created:** [August 5, 2020, 2:22pm UTC](https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879 "2020-08-05T14:22:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [August 5, 2020, 2:22pm UTC](https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879/1 "2020-08-05T14:22:24Z")

</div>

I have an ingest pipeline to remove some fields for a reindex. However, the pipeline is not removing the last field. I have changed the order and it always leaves the last field in line. I am Version: 6.4.2

```auto
{
  "reindex-prediction-clean": {
    "description": "Redindex prediction to clean fields",
    "processors": [
      {
        "remove": {
          "field": [
            "message_ovo",
            "event_from",
            "ovoid",
            "ovosev",
            "ovostate",
            "ticket_number",
            "sanity_check",
            "ovoeventtime",
            "ovo_event_time",
            "elapsed_time_sec"
          ],
          "ignore_failure": true
        }
      }
    ]
  }
}

```

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de0bff42ace971a26fd1acf17b33a857a682c34f.png)

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 5, 2020, 2:25pm UTC](https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879/2 "2020-08-05T14:25:31Z")

</div>

@chapmantrain not a permanent fix but have you tried this to see what it does?

```auto
{
  "reindex-prediction-clean": {
    "description": "Redindex prediction to clean fields",
    "processors": [
      {
        "remove": {
          "field": [
            "message_ovo",
            "event_from",
            "ovoid",
            "ovosev",
            "ovostate",
            "ticket_number",
            "sanity_check",
            "ovoeventtime",
            "ovo_event_time",
            "elapsed_time_sec",
            ""
          ],
          "ignore_failure": true
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [August 5, 2020, 3:06pm UTC](https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879/3 "2020-08-05T15:06:24Z")

</div>

Hey Aaron, thanks for the quick reply. I gave it a spin but elastic just decided it would not remove any fields. Good idea though.

Norm

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [August 5, 2020, 3:16pm UTC](https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879/4 "2020-08-05T15:16:35Z")

</div>

I think I got it. Probably should have RTFM more closely. Instead of ignore\_failure I should have had ignore\_missing: true. Looks like the pipline stopped processing when it hit a missing or null field in the parent.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 2, 2020, 3:16pm UTC](https://discuss.elastic.co/t/pipeline-remove-processor-not-removing-all-fields/243879/5 "2020-09-02T15:16:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.