Add new fields using ingest pipeline script processor

[Fabio-sama (Fabio)]

Hi there!

I'd like to add an ingest pipeline to my index in order to add some further info to incoming documents.

Let's say these docs have two fields, USERNAME and CITY. I'd like to add also latitude and longitude according to the field city.

I tried with an ingest pipeline like the following:

PUT _ingest/pipeline/my_index
{
    "processors": [
      {
        "script": {
          "source": """
            if (ctx.place == params.key_rome) {
              ctx.lat = params.rome_lat;
              ctx.long = params.rome_long;
            }
            if (ctx.place == params.key_ny) {
              ctx.lat = params.ny_lat;
              ctx.long = params.ny_long;
            }
            if (ctx.place == params.key_london) {
              ctx.lat = params.london_lat;
              ctx.long = params.london_long;
            }
          """,
          "params": {
            "key_rome": "Rome",
            "rome_lat": "X",
            "rome_long": "Y",
            "key_ny": "NewYork",
            "ny_lat": "Z",
            "ny_long": "T",
            "key_london": "London",
            "london_lat": "V",
            "london_long": "W"
          }
        }
      }
    ]
}

It didn't give any error but it simply did nothing.
I also tried adding a useless line after the last 'if block' simply to copy the field "place" into another field ctx.new_place = ctx.place and it didn't work either.

What am I doing wrong?

Thank you!

[spinscale (Alexander Reelsen)]

Hey

can you share the output of

POST _ingest/pipeline/_simulate
{
  "docs": [ 
    { "_source" : { "place":"London" } }, 
    { "_source" : { "place":"Rome" } }, 
    { "_source" : { "place":"NewYork" } }, 
    { "_source" : { "place":"Berlin" } } 
    ],
  "pipeline": {
    "processors": [
      {
        "script": {
          "source": """
            if (ctx.place == params.key_rome) {
              ctx.lat = params.rome_lat;
              ctx.long = params.rome_long;
            }
            if (ctx.place == params.key_ny) {
              ctx.lat = params.ny_lat;
              ctx.long = params.ny_long;
            }
            if (ctx.place == params.key_london) {
              ctx.lat = params.london_lat;
              ctx.long = params.london_long;
            }
""",
          "params": {
            "key_rome": "Rome",
            "rome_lat": "X",
            "rome_long": "Y",
            "key_ny": "NewYork",
            "ny_lat": "Z",
            "ny_long": "T",
            "key_london": "London",
            "london_lat": "V",
            "london_long": "W"
          }
        }
      }
    ]
  }
}

This example works for me and enriches three out of those four cities. Have you specified the pipeline when indexing a document? Please provide an exact reproduction and the elasticsearch version if it still does not work as expected.

--Alex

[Fabio-sama (Fabio)]

Ok I guess I found my problem. I only manage to edit stored data and not incoming ones, since I basically apply the ingest pipeline to my index via an update_by_query call.

How can I set my ingest pipeline as the default one for that specific index?

Thanks!

[spinscale (Alexander Reelsen)]

See index.default_pipeline in https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html

[Fabio-sama (Fabio)]

Great!

What I needed was to do:

PUT /my_index/_settings
{
    "index" : {
        "default_pipeline" : "my_pipe"
    }
}

Thank you so much for the tip!