With final_pipeline and a pipeline specified, the final pipeline appears to be running twice with index date math processor

Testing on 7.6.2, the following code works fine in using a final_pipeline and a pipeline for the index. Using the same code and updated index template code, it actually appears to run twice on 7.15.2 (and 7.16.0).

The following example appends a value and the value shows up 2 times rather than just once after running through the pipeline

PUT _ingest/pipeline/test-final-pipeline
{
  "processors": [
    {
      "append": {
        "field": "pipeline",
        "value": "completed"
      }
    }
  ]
}

PUT _template/test-template
{
  "index_patterns": [
    "test*"
  ],
  "settings": {
    "final_pipeline": "test-final-pipeline",
    "number_of_shards": 2
  }
}

POST /test-1/_doc
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah1"
}

PUT /_ingest/pipeline/routing-test-pipeline
{
  "description": "Time series DAY pipeline",
  "processors": [
     {
         "date_index_name": {
             "field": "time-field",
             "index_name_prefix": "test-",
             "date_rounding": "d",
             "index_name_format": "yyyy-MM-dd"
         }
     }
 ]
}

POST /test-1/_doc?pipeline=routing-test-pipeline
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah2"
}

The next example shows that trying to edit a field name, if you add ignore_missing: true, the field name is already renamed:

PUT _ingest/pipeline/test-final-pipeline
{
  "processors": [
    {
      "rename": {
        "field": "CONTENT",
        "target_field": "MALCONTENT"
      }
    }
  ]
}

PUT _template/test-template
{
  "index_patterns": [
    "test*"
  ],
  "settings": {
    "final_pipeline": "test-final-pipeline",
    "number_of_shards": 2
  }
}

POST /test-1/_doc
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah1"
}

PUT /_ingest/pipeline/routing-test-pipeline
{
  "description": "Time series DAY pipeline",
  "processors": [
     {
         "date_index_name": {
             "field": "time-field",
             "index_name_prefix": "test-",
             "date_rounding": "d",
             "index_name_format": "yyyy-MM-dd"
         }
     }
 ]
}

POST /test-1/_doc?pipeline=routing-test-pipeline
{
  "time-field": "2021-02-10T12:02:01.789Z",
  "CONTENT": "blah2"
}

Is there anything missing in these tests or could Issue #69727 or Issue #75047 be related to causing this to run twice when using the date math processor for the inde?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.