Ingest question - attachment processor plugin and dynamic fields

Elastic Stack Elasticsearch
1

I have a question regarding the ingest attachment plugin (I would say it is question in general about ingest processor at all).

Let's assume that I have some kind of index:

PUT /myindex
{
  "mappings": {
    "mytype": {
      "properties": {
          "#id" : {"type": "long"}, 
          "myfield": {
              "type": "nested",
              "properties": {
                "name":    { "type": "text"  },
                "revision": {
                    "properties": {
                       "#data": { "type": "text" }
                    } 
                } 
              }
        }
      }
    }
  }
}

I am creating the indexes dynamically by application and the myfield could have different name defined by user in document definition. The internal structure of myfield will be always the same and the file data will be stored within revision field on the #data property (or it could be different unique name). The myfield and revision field will be arrays (by nested or by objects - I think it doesn't matter).

Example PUT:

PUT myindex/mytype/1?pipeline=attachment
{
    "#id" : 4,
    "myfield": [
        {
            "name": "the first one",
            "revision": [
                {
                    "#data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="        
                }
            ]
        }
   ]
}

Now I need to configure the ingest attachment processor plugin to detect by some pattern or some request dynamic parameters the name of myfield. I could have a couple of that kind of fields within my index with different name but the structure of that fil will be always the same.

PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information from arrays",
  "processors" : [
    {
      "foreach": {
        "field": "*.revision", // does ingest support wilcard pattern or something like that (?)
        "processor": {
          "attachment": {
            "target_field": "_ingest._value.#attachment",
            "field": "_ingest._value.#data"
          }
        }
      }
    }
  ]
}

Additional question: how could I define the field (path) for nested fields for ingest processor? I was trying as usual the dot but for pipeline configuration I received exception (revision is not integer):

> PUT _ingest/pipeline/attachment
> {
>   "description" : "Extract attachment information from arrays",
>   "processors" : [
>     {
>       "foreach": {
>         "field": "myfield.revision",
>         "processor": {
>           "attachment": {
>             "target_field": "_ingest._value.attachment",
>             "field": "_ingest._value.data"
>           }
>         }
>       }
>     }
>   ]
> }

Thanks for help.

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.