Unable to display Array of Polygons in Kibana Maps

Kibana is unable to display an Array of geo_shape "polygon" data; however, I am able to display a single Polygon geo_shape. I would expect Kibana to parse the Array of results and display each of the polygon objects from within an Array.

Using Elastic Stack 7.16.3 and the DevTools Console, I created an Elasticsearch index with the following:

    PUT dat
    {
      "mappings": {
        "properties": {
          "polygons": {
            "properties": {
                "count": {
                  "type": "integer"
                },
                "side": {
                  "type": "text"
                },
                "location" : {
                    "type": "geo_shape"
                }
            }
          }
        }
      }
    }

Elasticsearch does not require us to specify whether we are inserting an object or an array of objects; it will handle either. So, I'm able to insert, for example, this Array of Polygon objects and have them recognized as a "geo_shape".

    POST /dat/_doc/1
    {
      "polygons": [
        {
          "side": "L",
          "count": 4,
          "location": {
            "type": "polygon",
            "coordinates": [
            [[-10.0, 5.0], [-10.0, 10.0], [-5.0, 10.0], [-5.0, 5.0], [-10.0, 5.0]]
            ]
          }
        },
        { 
          "side": "R",
          "count": 4,
          "location": {
           "type": "polygon",
            "coordinates": [
              [[0.0, 20.0], [0.0, 30.0], [-10.0, 30.0], [-10.0, 20.0], [0.0, 20.0]]
            ]
         }
       }
      ]
    }

And I can index a second document into Elasticsearch; this time a single Polygon object (not an array).

    POST /dat/_doc/2
    {
      "polygons":
        {
          "side": "L",
          "count": 4,
          "location": {
            "type": "polygon",
            "coordinates": [
            [[-10.0, 5.0], [-10.0, 10.0], [-5.0, 10.0], [-5.0, 5.0], [-10.0, 5.0]]
            ]
          }
        }
    }

I then created an Index Pattern for dat* in Kibana and try to display the indexed data.

I'm then able to simulate the query that Kibana uses (which I gathered directly from Kibana's Inspector) to show that it is able to retrieve the data stored in Elasticsearch:

    GET dat*/_search
    {
      "docvalue_fields": [],
      "size": 10000,
      "track_total_hits": 10001,
      "_source": [
        "polygons.location"
      ],
      "script_fields": {},
      "stored_fields": [
        "polygons.location"
      ],
      "runtime_mappings": {},
      "query": {
        "bool": {
          "must": [],
          "filter": [
            {
              "bool": {
                "must": [
                  {
                    "exists": {
                      "field": "polygons.location"
                    }
                  },
                  {
                    "geo_bounding_box": {
                      "polygons.location": {
                        "top_left": [
                          -180,
                          85.05113
                        ],
                        "bottom_right": [
                          180,
                          -66.51326
                        ]
                      }
                    }
                  }
                ]
              }
            }
          ],
          "should": [],
          "must_not": []
        }
      }
    }

And get the following results:

    {
      "took" : 695,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 2,
          "relation" : "eq"
        },
        "max_score" : 0.0,
        "hits" : [
          {
            "_index" : "dat",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 0.0,
            "_source" : {
              "polygons" : [
                {
                  "location" : {
                    "coordinates" : [
                      [
                        [
                          -10.0,
                          5.0
                        ],
                        [
                          -10.0,
                          10.0
                        ],
                        [
                          -5.0,
                          10.0
                        ],
                        [
                          -5.0,
                          5.0
                        ],
                        [
                          -10.0,
                          5.0
                        ]
                      ]
                    ],
                    "type" : "polygon"
                  }
                },
                {
                  "location" : {
                    "coordinates" : [
                      [
                        [
                          0.0,
                          20.0
                        ],
                        [
                          0.0,
                          30.0
                        ],
                        [
                          -10.0,
                          30.0
                        ],
                        [
                          -10.0,
                          20.0
                        ],
                        [
                          0.0,
                          20.0
                        ]
                      ]
                    ],
                    "type" : "polygon"
                  }
                }
              ]
            }
          },
          {
            "_index" : "dat",
            "_type" : "_doc",
            "_id" : "2",
            "_score" : 0.0,
            "_source" : {
              "polygons" : {
                "location" : {
                  "coordinates" : [
                    [
                      [
                        -10.0,
                        5.0
                      ],
                      [
                        -10.0,
                        10.0
                      ],
                      [
                        -5.0,
                        10.0
                      ],
                      [
                        -5.0,
                        5.0
                      ],
                      [
                        -10.0,
                        5.0
                      ]
                    ]
                  ],
                  "type" : "polygon"
                }
              }
            }
          }
        ]
      }
    }

When this data is displayed in Kibana, only the single Polygon Object (a simple square) is displayed, but the two Polygon objects in the Polygon Array are NOT displayed.

I would have expected all three polygon/squares to have been displayed in Kibana.

I also verified that the geo_shape field is of the correct type.

GET dat/_field_caps?fields=polygons.location

    {
      "indices" : [
        "dat"
      ],
      "fields" : {
        "polygons" : {
          "object" : {
            "type" : "object",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false
          }
        },
        "polygons.location" : {
          "geo_shape" : {
            "type" : "geo_shape",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : true
          }
        }
      }
    }

I think an alternative would be for me to flatten the Polygon Array and index individual documents for each polygon. But, I really don't want to do that because the polygon data I have should essentially be one "event" because they are all related to a single instance in time.

Another alternative I considered was to use "multipolygon" geo_shape. But, each Polygon object has additional meta-data attached to it (e.g. count, side) and this information would be lost with "multipolygon". In a separate exercise, I was able to display multiple polygons in Kibana, but doing so loses that additional metadata that is associated with each polygon area because multipolygon doesn't support having additional metadata with each of the coordinate arrays.

Hi @scottfred Welcome to the community and thanks for trying the Maps Features.

I updated your Thread Title and the labels lets see if anyone from the map groups answers.

I reproduced your results , I also see in the inspect the map request is bringing back the document with the array of shapes (see below). It appears that arrays of shapes being displayed is not currently supported.

I think you are going to need to take this approach and just add the data that correlates these shapes to a single event (event ID, timestamp etc)

The query is bringing back the array but it is not displaying... I changed your data a bit so they were not on top of each other. The square that shows is the single doc 2

1 Like

@stephenb Thanks for taking the time to confirm my results.

I also wondered if this is related to this Kibana > Maps GitHub bug (which has been fixed/merged) back in 2018 version 6.x: [Maps] elasticsearch document source does not support arrays · Issue #27698 · elastic/kibana · GitHub

Interesting ... I pinged one of the folks that worked on that.. see if anything comes back...

The fix for elasticsearch document source does not support arrays was limited to only supporting an array of Geometries, and not arrays of objects where the geometry is a child key of the object. Arrays of nested objects are not supported. I would recommend splitting each object in your array into its own document.

The below works because its an array of Geometries

{
      "polygons": [
        {
            "type": "polygon",
            "coordinates": [
            [[-10.0, 5.0], [-10.0, 10.0], [-5.0, 10.0], [-5.0, 5.0], [-10.0, 5.0]]
            ]
        },
        {
           "type": "polygon",
            "coordinates": [
              [[0.0, 20.0], [0.0, 30.0], [-10.0, 30.0], [-10.0, 20.0], [0.0, 20.0]]
            ]
       }
      ]
    }

The below does not work because its an array of objects where the geometry is a child key in the object

{
      "polygons": [
        {
          "side": "L",
          "count": 4,
          "location": {
            "type": "polygon",
            "coordinates": [
            [[-10.0, 5.0], [-10.0, 10.0], [-5.0, 10.0], [-5.0, 5.0], [-10.0, 5.0]]
            ]
          }
        },
        { 
          "side": "R",
          "count": 4,
          "location": {
           "type": "polygon",
            "coordinates": [
              [[0.0, 20.0], [0.0, 30.0], [-10.0, 30.0], [-10.0, 20.0], [0.0, 20.0]]
            ]
         }
       }
      ]
    }
1 Like

Thanks for the additional information @Nathan_Reese . Can you tell me if I can add additional key/value pairs the Geometries, like this?

{
      "polygons": [
        {
            "type": "polygon",
            "coordinates": [
            [[-10.0, 5.0], [-10.0, 10.0], [-5.0, 10.0], [-5.0, 5.0], [-10.0, 5.0]]
            ],
            "side": "L",
            "count": "4"
        },
        {
           "type": "polygon",
            "coordinates": [
              [[0.0, 20.0], [0.0, 30.0], [-10.0, 30.0], [-10.0, 20.0], [0.0, 20.0]]
            ],
            "side": "R",
            "count": "3"
       }
      ]
    }

Can you tell me if I can add additional key/value pairs the Geometries, like this?

That is not allowed

Can you tell me where this "Geometries" object is defined?

Hmmm... @Nathan_Reese maybe I wasn't clear about what I wanted, or maybe this works in a way you didn't expect it to work, but I was able to build a "polygons" array with a couple Objects in the array with "type" => "polygon" and "coordinates" => AND have the additional key/value fields without any problems.

Here's the resulting Polygon object (with additional key/value pairs) in it:

{
           "tags" => [
        [0] "multiline"
    ],
           "host" => "fbe690cefded",
       "polygons" => [
        [0] {
                "ecl-alt" => "d",
            "coordinates" => [
                [0] [
                    [0] [
                        [0] 0.0,
                        [1] 0.0
                    ],
                    [1] [
                        [0] 0.0,
                        [1] 10.0
                    ],
                    [2] [
                        [0] -10.0,
                        [1] 10.0
                    ],
                    [3] [
                        [0] -10.0,
                        [1] 0.0
                    ],
                    [4] [
                        [0] 0.0,
                        [1] 0.0
                    ]
                ]
            ],
                    "bat" => "c",
                   "type" => "polygon",
                   "side" => "R",
                  "count" => 4
        },
        [1] {
                "ecl-alt" => "d",
            "coordinates" => [
                [0] [
                    [0] [
                        [0] 0.0,
                        [1] 20.0
                    ],
                    [1] [
                        [0] 0.0,
                        [1] 30.0
                    ],
                    [2] [
                        [0] -10.0,
                        [1] 30.0
                    ],
                    [3] [
                        [0] -10.0,
                        [1] 20.0
                    ],
                    [4] [
                        [0] 0.0,
                        [1] 20.0
                    ]
                ]
            ],
                    "bat" => "c",
                   "type" => "polygon",
                   "side" => "R",
                  "count" => 4
        }
    ],
           "path" => "/data/test.dat",
   "@version" => "1",
   "@timestamp" => 2022-02-28T19:12:26.648Z
}

Here's the search query from Kibana > Maps Inspector:

{
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "epoch_millis"
    }
  ],
  "size": 10000,
  "track_total_hits": 10001,
  "_source": [
    "polygons"
  ],
  "script_fields": {},
  "stored_fields": [
    "@timestamp",
    "polygons"
  ],
  "runtime_mappings": {},
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "polygons"
                }
              },
              {
                "geo_bounding_box": {
                  "polygons": {
                    "top_left": [
                      -180,
                      85.05113
                    ],
                    "bottom_right": [
                      180,
                      -66.51326
                    ]
                  }
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2022-02-28T18:58:16.540Z",
              "lte": "2022-02-28T19:13:16.540Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Here are the results from the query:

{
  "rawResponse": {
    "took": 43,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 1,
        "relation": "eq"
      },
      "max_score": 0,
      "hits": [
        {
          "_index": "dat",
          "_type": "_doc",
          "_id": "zMK_QX8BnIT2RMugjILD",
          "_score": 0,
          "_source": {
            "polygons": [
              {
                "side": "R",
                "bat": "c",
                "ecl-alt": "d",
                "coordinates": [
                  [
                    [
                      0,
                      0
                    ],
                    [
                      0,
                      10
                    ],
                    [
                      -10,
                      10
                    ],
                    [
                      -10,
                      0
                    ],
                    [
                      0,
                      0
                    ]
                  ]
                ],
                "count": 4,
                "type": "polygon"
              },
              {
                "side": "R",
                "bat": "c",
                "ecl-alt": "d",
                "coordinates": [
                  [
                    [
                      0,
                      20
                    ],
                    [
                      0,
                      30
                    ],
                    [
                      -10,
                      30
                    ],
                    [
                      -10,
                      20
                    ],
                    [
                      0,
                      20
                    ]
                  ]
                ],
                "count": 4,
                "type": "polygon"
              }
            ]
          },
          "fields": {
            "@timestamp": [
              "1646075546648"
            ]
          }
        }
      ]
    }
  },
  "isPartial": false,
  "isRunning": false,
  "total": 1,
  "loaded": 1,
  "isRestored": false
}

Both squares are plotted without any issues which is what I wanted... the Elasticsearch database contains the additional metadata and Kibana Maps is able to plot an array of the Polygons.

Can you seen any issues with why this would cause issues? Geo Queries? GeoSpatial queries for Intersections with other shapes?

You will see issues when styling by value. Also, you may see problems with tooltips.

Okay @Nathan_Reese Thanks for the additional information. Those things (styling by value and tooltips) aren't important to us, so we may keep going down this road rather than increasing the amount of data we need to save to store the metadata related to each of these polygons separately. Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.