Set processor for a specific Index

[Sajal_Chakraborty]

I am trying to use the below code to create a Set Processor for Index default-*, but somehow it's not working

PUT /_ingest/pipeline/addaht
{
"description" : "sets queue wise AHT constants",
"processors": [
{
"set": {

    "if": "ctx.queueName == 'A'",

    "field": "queueAHTVal",

    "value": "10"

  }
}

]

}

The script executes successfully and I get a {"acknowledged" : true} response, but when I into Discover in Kibana, I can't see the field added.

What I am trying to achieve is that -

For Index default-*, the preprocessor queueAHTVal should be added for a document field queueName whose value should match the condition provided.

Please help.

Warm Regards,
Sajal

[Stuart_Duncan]

Hi Sajal,

The Pipeline you have specified looks good, as does the conditional. You will need to run it against your index, for example using the Reindex API:

POST _reindex

    {
      "source": {
        "index": "default"
      },
      "dest": {
        "index": "default_set",
        "pipeline": "addaht"
      }
    }

Checking this against an index with two simple (one with queueName == 'A' and one with queueName == 'B'), resulted in the following:

GET default/_search

    {
      "took" : 0,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 2,
          "relation" : "eq"
        },
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "default_set",
            "_type" : "_doc",
            "_id" : "b29B13MB1__HAGu7A4FL",
            "_score" : 1.0,
            "_source" : {
              "@timestamp" : "2099-11-15T13:12:00",
              "queueName" : "A",
              "queueAHTVal" : "10"
            }
          },
          {
            "_index" : "default_set",
            "_type" : "_doc",
            "_id" : "Em9B13MB1__HAGu7YIN3",
            "_score" : 1.0,
            "_source" : {
              "@timestamp" : "2099-11-15T13:14:00",
              "queueName" : "B"
            }
          }
        ]
      }
    }

Reindexing in this way will mean that the original index is not touched.

You can also use 'update by query' to apply this to the existing index, rather than create a new one:

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-api-ingest-pipeline

As an alternative to using the 'Set' processor, and a set of conditionals for each possible queue value, you could use the Enrich processor, and have an Elasticsearch Index that includes the mapping of queueName to the AHTVals:
https://www.elastic.co/guide/en/elasticsearch/reference/current/match-enrich-policy-type.html

If you are happy with your ingest pipeline logic, and want to apply it automatically to new events as they arrive, you can do it by specifying the pipeline to use on your indexing requests, either when you issue index commands:
https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html

PUT my-index-000001/_doc/my-id?pipeline=my_pipeline_id

Or by setting the index.default_pipeline parameter to specify your pipeline in the index settings:
https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules.html#dynamic-index-settings

Please try this and let us know how you get on,

Thanks,

Stuart.

[Sajal_Chakraborty]

Hi Stuart,

Thanks for this. I'll definitely take a look into this.

Warm Regards,
Sajal