# Pipeline not applied to all documents

**URL:** <https://discuss.elastic.co/t/pipeline-not-applied-to-all-documents/177230>\
**Category:** Elasticsearch\
**Created:** [April 17, 2019, 8:23am UTC](https://discuss.elastic.co/t/pipeline-not-applied-to-all-documents/177230 "2019-04-17T08:23:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marc\_Zahn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marc_zahn/32/44302_2.png) [@Marc\_Zahn](https://discuss.elastic.co/u/Marc_Zahn)\
**Post date:** [April 17, 2019, 8:23am UTC](https://discuss.elastic.co/t/pipeline-not-applied-to-all-documents/177230/1 "2019-04-17T08:23:52Z")

</div>

Hi,

I implemented a script pipeline to finally convert some fields programmatically. First I wanted to test it and it turned out that the pipeline is not applied to all documents but only to a few. This is my pipeline:

```
PUT _ingest/pipeline/survey_name
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "script" : {
        "lang": "painless",
        "on_failure": [
          {
            "set" : {
              "field" : "error",
              "value" : "field \"foo\" does not exist, cannot rename to \"bar\""
            }
          }
        ],
        "source": """
        ctx.search = '111111111111';
        """
      }
    }
  ]
}

```

The index contains these fields:

```
{
  "_routing": {
    "required": true
  },
  "properties": {
    "entity": { "type": "keyword" },
    "search": { "type": "keyword" },
    "error": {"type": "keyword"},
  ...
}

```

In the documents where the search field is not going to be set the error field is not set either. I don't see any patterns on the documents whether it works or not. Since the inserting is done by Queue-consumers the doucments are inserted closely to in parallel - Could this maybe a problem?

Can anyone help here?

EDIT: Please don't care about the namings - This is mostly copy & pasted.

---

<div class="post-metadata">

**Author:** ![Marc\_Zahn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marc_zahn/32/44302_2.png) [@Marc\_Zahn](https://discuss.elastic.co/u/Marc_Zahn)\
**Post date:** [April 17, 2019, 9:11am UTC](https://discuss.elastic.co/t/pipeline-not-applied-to-all-documents/177230/2 "2019-04-17T09:11:57Z")

</div>

EDIT: I investigated further and it seems that the pipeline is only applied on every second document.. WTH?

---

<div class="post-metadata">

**Author:** ![HenningAndersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henningandersen/32/48188_2.png) [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)\
**Post date:** [April 17, 2019, 2:14pm UTC](https://discuss.elastic.co/t/pipeline-not-applied-to-all-documents/177230/3 "2019-04-17T14:14:14Z")

</div>

Hi @Marc_Zahn,

can you provide sample indexing requests that exhibit the "every second document" behavior"?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2019, 2:14pm UTC](https://discuss.elastic.co/t/pipeline-not-applied-to-all-documents/177230/4 "2019-05-15T14:14:16Z")

</div>

Closed on May 15, 2019, 2:14pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.