Array of objects datatype exists

Elastic Stack Elasticsearch
1

I have a field called alarm_images where I store a list of objects with three fields download_date,
index and url. I wanted to lighten my mapping since I was using lot of memory for lot of fields and I wanted to disable doc_values and index for all values since I only have to retrieve the list with the rest of the document every once in a while. My problem is that I have to check if alarm_images object exists and using a mapping like the following I always get that alarm_images does not exist while it does.

        "alarm_images": {
           "type": "nested",
           "properties": {
              "download_date": {
                 "type": "date",
                 "index": false,
                 "doc_values": false
              },
              "index": {
                 "type": "integer",
                 "index": false,
                 "doc_values": false
              },
              "url": {
                 "type": "keyword",
                 "index": false,
                 "doc_values": false
              }
           }

is there any way to use nested object without index and doc_values and still preserve the capability to check if the field exists?

2

I believe that one of the easiest way is to add a boolean value.

You can compute it may be at index time using an ingest script processor.

3

I am new to ingest node, I've only used it with the basic pipeline plugins. How can I add a field based on a condition?
I tried using script pipeline but I got stuck adding the new field to the document.
Should I use another pipeline?

This is what I got so far.

POST _ingest/pipeline/_simulate
{
   "pipeline": {
      "description": "checks if the image field is populated",
      "processors": [
         {
            "script": {
               "lang": "painless",
               "inline": "if(ctx._source?.image != null){ctx._source.has_image = true} else {ctx._source.has_image = false}"
            }
         }
      ]
   },
   "docs": [
      {
         "_index": "sensor-2017-06-30",
         "_type": "status",
         "_id": "id",
         "_source": {
            "image": {
               "url": "image.jpg",
               "download_type": "last",
               "download_date": "2017-06-30T06:33:56.092611+00:00"
            }
         }
      }
   ]
}
4

Is it a typo?

ctx._source?.image != null

Should it be

ctx._source.image != null
5

I solved my issue! :slight_smile:
It was not the typo you suggested.
With the first one

ctx._source?.image != null

I get always false in the condition while with the second

ctx._source.image != null

I get a NullPointerException,
This is the solution I came with, actually I don't know why it works since I though I had to add _source after ctx. Thank you for your quick responsiveness. Have a nice day.

POST _ingest/pipeline/_simulate
{
   "pipeline": {
      "description": "checks if image and alarm_images field is populated",
      "processors": [
         {
            "script": {
               "lang": "painless",
               "inline": "if(ctx.image != null){ctx.has_image = true} else {ctx.has_image = false}"
            }
         },
         {
            "script": {
               "lang": "painless",
               "inline": "if(ctx.alarm_images != null){ctx.has_alarm_image = true} else {ctx.has_alarm_image = false}"
            }
         }
      ]
   },
   "docs": [
      {
         "_index": "sensor-2017-06-30",
         "_type": "status",
         "_id": "id",
         "_source": {
            "image": {
               "url": "image.jpg",
               "download_type": "last",
               "download_date": "2017-06-30T06:33:56.092611+00:00"
            }
         }
      },
      {
         "_index": "sensor-2017-06-30",
         "_type": "status",
         "_id": "id",
         "_source": {
            "message": "painless is not painless"
         }
      },
      {
         "_index": "sensor-2017-06-30",
         "_type": "status",
         "_id": "id",
         "_source": {
            "alarm_images": [
               {
                  "url": "alarm_image.jpg",
                  "index": 0,
                  "download_date": "2017-06-30T06:34:07.744397+00:00"
               },
               {
                  "url": "alarm_image.jpg",
                  "index": 1,
                  "download_date": "2017-06-30T06:34:08.851667+00:00"
               }
            ]
         }
      }
   ]
}
1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.