How to make Dev tools request automatic using the Pipeline?

[Angad_Panesar1 (Angad Panesar)]

Hi, I tried to create an ingest pipeline to run every time new data is ingested but unfortunately it still does not work. Here is what I have done:

The create pipeline method:

PUT _ingest/pipeline/set_place
{
  "description": "create a field called place and sets garage to it.",
  "processors": [
    {
      "set": {
        "field": "place",
        "value": "garage",
        "if": "ctx?.ID == '3c71bf6eb508'"
      }
    }
  ]
}

The simulate pipeline method:

POST _ingest/pipeline/set_place/_simulate
{
  "docs": [
    {
      "_source": {
        "ID": "3c71bf6eb508"
      }
    }
  ]
}

The simulate pipeline method does not work even when new data is added. Is there any way that you can help me with this please?

[stephenb (Stephen Brown)]

I think there's a little confusion.

The simulate only simulates with that doc you provide.

You have to configure the pipeline to be executed when the document is ingested, this is what they are made for.

Depending on tool / method (Filebeat, Logstash, Client, POST) method are you using to ingest the data, will define how to call the pipeline on document ingest.

Tell us that, how you are ingesting your data, and we can show you how to set the pipeline to be used?

You can also just set your pipeline as the default_pipeline using this setting in your index or template See Here a bit down the page

PUT _ingest/pipeline/set_place
{
  "description": "create a field called place and sets garage to it.",
  "processors": [
    {
      "set": {
        "field": "place",
        "value": "garage",
        "if": "ctx?.ID == '3c71bf6eb508'"
      }
    }
  ]
}

# Setting the default pipeline, you could set this in your template 

PUT my-place-index/
{
  "settings": {
    "index.default_pipeline" : "set_place"
    
  }
}

# Index / ingest a document 
POST my-place-index/_doc
{
  "myfield" : "myvalue",
  "ID": "3c71bf6eb508"
}

results, the pipeline is executed when the document is ingested.

{
  "took" : 835,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-place-index",
        "_type" : "_doc",
        "_id" : "VqC_zn0BEq8CUkeUD7Wg",
        "_score" : 1.0,
        "_source" : {
          "ID" : "3c71bf6eb508",
          "place" : "garage",
          "myfield" : "myvalue"
        }
      }
    ]
  }
}

OR if you don't set it as a default pipeline you can just do this and call the pipeline explicitly

POST my-place-other-index/_doc?pipeline=set_place
{
  "myfield" : "myvalue",
  "ID": "3c71bf6eb508"
}

And the pipeline will be executed...

GET my-place-other-index/_search

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-place-other-index",
        "_type" : "_doc",
        "_id" : "WaDmzn0BEq8CUkeUoLXM",
        "_score" : 1.0,
        "_source" : {
          "ID" : "3c71bf6eb508",
          "place" : "garage",
          "myfield" : "myvalue"
        }
      }
    ]
  }
}