Index.final_pipeline not working as expected

Elastic Stack Elasticsearch
1

Hello,

I have a filebeat that uses a module with a ingest pipeline, since I still need to make some transformations I'm trying to use another ingest pipeline with the setting index.final_pipeline in the index settings.

I've created the pipeline, changed the index setting, but I still do not see the field I'm adding.

I'm using the following ingest pipeline:

PUT _ingest/pipeline/pipeline-name
{
  "description": "create user.name",
  "processors": [
    {
      "rename": {
        "field": "data.win.eventdata.targetUserName",
        "target_field": "user.name"
      }
    }
  ]
}

This works without any problem when I simulate.

In the index settings I have:

"index.final_pipeline" : "pipeline-name",

Since this is a dynamic setting, it should work as soon as it was applied, but for some reason it is not working.

What am I missing? Is it possible to use the pipeline from a module and a final_pipeline? From the documentation I'm assuming that this is supported as the final_pipeline would run after everything.

Normally I would use logstash, but in this case I have a couple of beats sending directly to elasticsearch.

edit: I'm on version 7.12.

2

Hey Leandro,

can you provide a fully reproducible example? This snippet works for me on 7.13.1

PUT _ingest/pipeline/pipeline-name
{
  "description": "create user.name",
  "processors": [
    {
      "rename": {
        "field": "data.win.eventdata.targetUserName",
        "target_field": "user.name"
      }
    }
  ]
}

PUT test 
{
  "settings": {
    "index.final_pipeline" : "pipeline-name"
  }
}

PUT test/_doc/1
{
  "data" : { "win" : {"eventdata" : { "targetUserName":"Alex" }}}
}

GET test/_doc/1

--Alex

3

Hello Alex,

In Dev Tools everything works as expected, the problem is with the data shipped by Filebeat.

The filebeat modules uses a ingest pipeline, it is applied without any problem, but the final_pipeline set for the same index is ignored, the fields are not renamed or create, as I also tested using the set processor.

Do you know how filebeat sends the request to elasticsearch when using an ingest_pipeline?

I also tested using two pipelines to try to simulate this behavior, but it worked as expected.

For example, I created another ingest pipeline to add a new field:

PUT _ingest/pipeline/add-field-test
{
  "description": "ingest pipeline test",
  "processors": [
    {
      "set": {
        "field": "field.test",
        "value": "ingest pipeline test",
        "ignore_failure": true
      }
    }
  ]
}

Then I tried another ingest pipeline, this time also adding a new field.

PUT _ingest/pipeline/add-user-name
{
  "description": "create event.source",
  "processors": [
    {
      "set": {
        "field": "user.name",
        "value": "{{{data.win.eventdata.targetUserName}}}",
        "ignore_failure": false      
        
      }
    }
  ]
}

Adding "index.final_pipeline" : "add-user-name" to the test index setting and using the following request:

POST test-index/_doc/1?pipeline=add-field-test
{
  "data": {
    "win": {
      "eventdata": {
        "targetUserName": "user1"
      }
    }
  }
}

The output is as expected, with the fields from the pipeline add-field-test and the field from the add-user-name pipeline.

{
  "_index" : "test-index",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 547526,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "data" : {
      "win" : {
        "eventdata" : {
          "targetUserName" : "user1"
        }
      }
    },
    "field" : {
      "test" : "ingest pipeline test"
    },
    "user" : {
      "name" : "user1"
    }
  }
}

If the ingestion is from Filebeat, this does not work, the index.final_pipeline is ignored and not applied, I see no errors in Filebeat nor in Elasticsearch.

I already opened a ticket with elastic support, but didn't have time yet to run the diagnostics they asked.

4

This sounds to me, as if filebeat is setting up the index slightly differently, probably unsettting the final pipeline. Can you check the index configuration/mapping configurations?

Just a guess however :slight_smile:

5

The indices are created daily and use a index template with the index.final_pipeline set, filebeat does not change the index template, I'm also not using ILM to create the index.

If I make a request to the _settings endpoint of the index I can see that the index.final_pipeline is there as expected.

I enabled the debug log in filebeat and saw no error or indications of why it is not working, also no errors on elasticsearch.

I will wait for an answer from the ticket I opened with the support team and will update here if I have a solution.

2 Likes
6

Well, it seems to be a bug related to this issue.

I was test on a dev cluster on version 7.12.0 and it was not working, as the merge says that it was backported to version 7.12.1, which is my production cluster version, I've tried to apply the final_pipeline in this cluster, but the behavior is the same.

I contacted the support again asking for clarifications for why it is not working or if my use case was not contemplated by this fix, will update here when I get an answer.

7

Just an update if someone has the same issue.

It is a bug, an issue was opened and resolved in this pull request.

On my use case it wasn't working when the index.final_pipeline was being set dynamically, if I set it in the index template, then it will work for the next daily index created.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.