# Index.final\_pipeline not working as expected

**URL:** <https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [June 16, 2021, 8:37pm UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195 "2021-06-16T20:37:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 16, 2021, 8:37pm UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/1 "2021-06-16T20:37:58Z")

</div>

Hello,

I have a filebeat that uses a module with a ingest pipeline, since I still need to make some transformations I'm trying to use another ingest pipeline with the setting `index.final_pipeline` in the index settings.

I've created the pipeline, changed the index setting, but I still do not see the field I'm adding.

I'm using the following ingest pipeline:

```auto
PUT _ingest/pipeline/pipeline-name
{
  "description": "create user.name",
  "processors": [
    {
      "rename": {
        "field": "data.win.eventdata.targetUserName",
        "target_field": "user.name"
      }
    }
  ]
}

```

This works without any problem when I simulate.

In the index settings I have:

```auto
"index.final_pipeline" : "pipeline-name",

```

Since this is a dynamic setting, it should work as soon as it was applied, but for some reason it is not working.

What am I missing? Is it possible to use the pipeline from a module and a final\_pipeline? From the documentation I'm assuming that this is supported as the final\_pipeline would run after everything.

Normally I would use logstash, but in this case I have a couple of beats sending directly to elasticsearch.

edit: I'm on version 7.12.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 21, 2021, 9:37am UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/2 "2021-06-21T09:37:03Z")

</div>

Hey Leandro,

can you provide a fully reproducible example? This snippet works for me on 7.13.1

```auto
PUT _ingest/pipeline/pipeline-name
{
  "description": "create user.name",
  "processors": [
    {
      "rename": {
        "field": "data.win.eventdata.targetUserName",
        "target_field": "user.name"
      }
    }
  ]
}

PUT test 
{
  "settings": {
    "index.final_pipeline" : "pipeline-name"
  }
}

PUT test/_doc/1
{
  "data" : { "win" : {"eventdata" : { "targetUserName":"Alex" }}}
}

GET test/_doc/1

```

--Alex

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 21, 2021, 2:33pm UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/3 "2021-06-21T14:33:58Z")

</div>

Hello Alex,

In Dev Tools everything works as expected, the problem is with the data shipped by Filebeat.

The filebeat modules uses a ingest pipeline, it is applied without any problem, but the final\_pipeline set for the same index is ignored, the fields are not renamed or create, as I also tested using the `set` processor.

Do you know how filebeat sends the request to elasticsearch when using an ingest\_pipeline?

I also tested using two pipelines to try to simulate this behavior, but it worked as expected.

For example, I created another ingest pipeline to add a new field:

```auto
PUT _ingest/pipeline/add-field-test
{
  "description": "ingest pipeline test",
  "processors": [
    {
      "set": {
        "field": "field.test",
        "value": "ingest pipeline test",
        "ignore_failure": true
      }
    }
  ]
}

```

Then I tried another ingest pipeline, this time also adding a new field.

```auto
PUT _ingest/pipeline/add-user-name
{
  "description": "create event.source",
  "processors": [
    {
      "set": {
        "field": "user.name",
        "value": "{{{data.win.eventdata.targetUserName}}}",
        "ignore_failure": false      
        
      }
    }
  ]
}

```

Adding `"index.final_pipeline" : "add-user-name"` to the test index setting and using the following request:

```auto
POST test-index/_doc/1?pipeline=add-field-test
{
  "data": {
    "win": {
      "eventdata": {
        "targetUserName": "user1"
      }
    }
  }
}

```

The output is as expected, with the fields from the pipeline `add-field-test` and the field from the `add-user-name` pipeline.

```auto
{
  "_index" : "test-index",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 547526,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "data" : {
      "win" : {
        "eventdata" : {
          "targetUserName" : "user1"
        }
      }
    },
    "field" : {
      "test" : "ingest pipeline test"
    },
    "user" : {
      "name" : "user1"
    }
  }
}

```

If the ingestion is from Filebeat, this does not work, the `index.final_pipeline` is ignored and not applied, I see no errors in Filebeat nor in Elasticsearch.

I already opened a ticket with elastic support, but didn't have time yet to run the diagnostics they asked.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 22, 2021, 7:44am UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/4 "2021-06-22T07:44:33Z")

</div>

This sounds to me, as if filebeat is setting up the index slightly differently, probably unsettting the final pipeline. Can you check the index configuration/mapping configurations?

Just a guess however 🙂

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 22, 2021, 2:05pm UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/5 "2021-06-22T14:05:04Z")

</div>

The indices are created daily and use a index template with the `index.final_pipeline` set, filebeat does not change the index template, I'm also not using ILM to create the index.

If I make a request to the `_settings` endpoint of the index I can see that the `index.final_pipeline` is there as expected.

I enabled the debug log in filebeat and saw no error or indications of why it is not working, also no errors on elasticsearch.

I will wait for an answer from the ticket I opened with the support team and will update here if I have a solution.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 1, 2021, 6:50pm UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/6 "2021-07-01T18:50:41Z")

</div>

Well, it seems to be a bug related to this [issue](https://github.com/elastic/elasticsearch/pull/71563).

I was test on a dev cluster on version 7.12.0 and it was not working, as the merge says that it was backported to version 7.12.1, which is my production cluster version, I've tried to apply the final\_pipeline in this cluster, but the behavior is the same.

I contacted the support again asking for clarifications for why it is not working or if my use case was not contemplated by this fix, will update here when I get an answer.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 19, 2021, 2:19am UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/7 "2021-07-19T02:19:46Z")

</div>

Just an update if someone has the same issue.

It is a bug, an [issue](https://github.com/elastic/elasticsearch/issues/75047) was opened and resolved in this [pull request](https://github.com/elastic/elasticsearch/pull/75314).

On my use case it wasn't working when the `index.final_pipeline` was being set dynamically, if I set it in the index template, then it will work for the next daily index created.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 16, 2021, 2:20am UTC](https://discuss.elastic.co/t/index-final-pipeline-not-working-as-expected/276195/8 "2021-08-16T02:20:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.