[Vega Map] How to get geoshapes from elasticsearch index coordinates

[lehti]

Hi!

I need to make a map using Vega-visualization. The coordinates come from an Elasticsearch index. I can load the index into Vega just fine. The issue is that I have no idea how to get Vega to understand the coordinates as anything, especially as geoshapes. My index is of the form:

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 23,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my_index",
        "_type" : "_doc",
        "_id" : "IjNdJnwBrZ33gI2pCUby",
        "_score" : 1.0,
        "_source" : {
          "coordinates" : {
            "type" : "polygon",
            "coordinates" : [
              [
                [
                  25.05822022737424,
                  60.27598832155146
                ],
	        …
                [
                  25.05822022737424,
                  60.27598832155146
                ]
              ]
            ]
          },
          "name" : "area1"
        }
      },
      {
        "_index" : "my_index",
        "_type" : "_doc",
        …

How to get the coordinates from the index and plot them as geoshapes? Thanks in advance!

[Nathan_Reese]

What version of Kibana are you using? Have you tried displaying the geo_shapes with Maps?

[lehti]

I'm using 7.15.0, and the geoshapes work fine in Kibana's own maps.

[Nathan_Reese]

Looking at the docs for vega geoshape transform, it looks like it is not possible to use the native response from Elasticsearch. Elasticsearch does not return geojson, but rather returns custom json with embedded geojson geometries. Vega geoshape transform requires geojson.

What is missing is a way to wrap _source.coordinates from a geometry like below.

{
            "type" : "polygon",
            "coordinates" : [
              [
                [
                  25.05822022737424,
                  60.27598832155146
                ],
	        …
                [
                  25.05822022737424,
                  60.27598832155146
                ]
              ]
            ]
          }

into a feature like the below.

{
  "type": "Feature",
  "geometry": {
            "type" : "polygon",
            "coordinates" : [
              [
                [
                  25.05822022737424,
                  60.27598832155146
                ],
	        …
                [
                  25.05822022737424,
                  60.27598832155146
                ]
              ]
            ]
          }
       }
   }

If you can figure out a way to modify the results in vega then you should be able to use geoshape transform.

Below is the vega configuration that I am using. I created an index of world countries from https://vector.maps.elastic.co/files/world_countries_v7.geo.json

{
  $schema: https://vega.github.io/schema/vega/v5.json
  config: {
    kibana: {type: "map", latitude: 30, longitude: -120, zoom: 3}
  }
  data: [
    {
      name: world_countries
      url: {
        index: world_countries_v7
        %context%: true
        body: {
          size: 250
        }
      }
      format: {property: "hits.hits"}
    }
  ]
  scales: []
  marks: [
    {
      type: shape
      from: {data: "world_countries"}
      encode: {
        enter: {
          "strokeWidth": {"value": 0.5},
          "stroke": {"value": "#bbb"},
          "fill": {"value": "#000"},
          "zindex": {"value": 0}
        }
      },
      transform: [
        { 
          type: "geoshape", 
          projection: "projection",
          field: "_source.coordinates"
        }
      ]
    }
  ]
}

[lehti]

Thanks for the tip! Do you know if it is possible to wrap Elasticsearch results in another schema (in this case, ""type": "Feature","geometry": ")? If not, should this be done by string concat + parsing to json?

[lehti]

A bit of an update: I've now managed to get the coordinates into the desired form. However, the shapes just don't appear. My current code:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.16.8.json",
  "data": {
    "url": {"index": "my_index"},
    "format": {property: "hits.hits", "type": "json"},
  },
  "transform": [{
    "calculate": "merge({'type': 'Feature'}, {'geometry': datum._source.coordinates})",
    "as": "coordinates"
  },
  {
    "type": "geojson",
    "field": "coordinates"
  }],
  mark: "geoshape",
  encoding: {shape: {field: "coordinates"}}
}

So, what am I missing here?