Remove a Field using a wildcard on reindex

[hogbinj (James Hogbin)]

I've done a foolish thing and created a large number of fields in the format

 "fingerprint.attachment_5C6C2D4E000092B707F41EF1" : "Attachment 5C6C2D4E000092B707F41EF1.gif is an IMAGE and therfore NOT processed"

I used an underscore rather than a period

I'm trying to reindex to remove these fields using this pipeline but it's not working. Can anybody point out where I'm going wrong.

PUT _ingest/pipeline/remove-fingerprint-attachment-field
{
  "description" : "Remove Fingerprint Attachment Field pipeline",
  "processors" : [
     {
     "foreach" : {
        "field" : "fingerprint",
         "processor" : {
            "script": {
              "lang": "painless",
              "source": "if (ctx._ingest._value.contains(params.field_part)) {ctx._ingest_value.remove}",
              "params": {
                "params":
                  {"field_part":"attachment_"}
              }
            }
          }
        }
    }  
  ]
}

The error is

"cause": {
  "type": "exception",
  "reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [fingerprint] not present as part of path [fingerprint]",
  "caused_by": {
    "type": "illegal_argument_exception",
    "reason": "java.lang.IllegalArgumentException: field [fingerprint] not present as part of path [fingerprint]",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "field [fingerprint] not present as part of path [fingerprint]"
    }
  },
  "header": {
    "processor_type": "foreach"
  }
}

[jpcarey (Jared Carey)]

@hogbinj

"fingerprint.attachment_5C6C2D4E000092B707F41EF1"

foreach would iterate on an array, which the source data does not appear to match that.

You could likely do this with just a reindex script rather than needing to use an ingest pipeline. Below is a quick sample using the ingest simulate with an ingest script processor to add the values to a list (if there are multiple) and remove.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
Iterator it = ctx.entrySet().iterator();
def tmp_list = new ArrayList();
while (it.hasNext()) {
  def k = it.next();
  if (k.getKey().contains("fingerprint.attachment_")) {
    tmp_list.add(k.getValue());
    it.remove();
  }
}
ctx.put("fingerprint.errors", tmp_list);
"""
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "fingerprint.attachment_5C6C2D4E000092B707F41EF1": "Attachment 5C6C2D4E000092B707F41EF1.gif is an IMAGE and therfore NOT processed",
        "fingerprint.attachment_5C6C2D4E000092B707F41EF2": "Attachment 5C6C2D4E000092B707F41EF2.gif is an IMAGE and therfore NOT processed",
        "test_key": 1,
        "test_object": {
          "test_key2": "key 2"
        },
        "test_array": [
          "value1",
          "value2"
        ]
      }
    }
  ]
}

[hogbinj (James Hogbin)]

THANK YOU VERY MUCH!!

how do I find out where to script painless like that. I spent ages on the ES website but missed the Iterator part.

Is there an idiots guide anywhere?

James

[jpcarey (Jared Carey)]

Personally, just look for java examples on the web, and pay close attention to the slight differences in syntax and the whitelisted java classes for painless.