Geo-point fields in nested mappings not accessible in Maps

Elastic Stack Kibana
#1

My index has a nested definition that includes a property of type geo-point. I can see it in Discover but it is not accessible from Documents layer in Maps. Is this the expected behavior or does my mapping need to change? I am using ES 7.6 currently.

example from mapping:

{
    "activityLocation": {
        "type": "nested",
        "properties": {
            "coordinates": {"type": "geo-point"},
            "name": {"type": "keyword"}
        }
    }
}

corresponding data in record:

{
    "activityLocation": {
        "coordinates": {"lat": 34.76544, "lon": -45.8888"},
        "name": "A Place"
    }
}

If I have the fields mapped separated - not as part of nested definition, then I can access the geo-point field in Maps. This is fine for an object that has a single entry - I can flatten.

Problem is when I have a field that is a list of objects, which is why I am treating as nested object so that I can retain relationships within each object.

{ 
    "activityLocation.coordinates": {"type": "geo-point"},
    "activityLocation.name": {"type": keyword"}
}

I know also, that for the "list of objects", I can populate a separate field to capture all the geo-points - just trying to avoid that, if possible. It is an accessible property in Maps.

{
    "activityLocationGeo.coordinates": {"type": "geo_point"},
    "activityLocation": {
        "type": "nested",
        "properties": {
            "coordinates": {"type": "geo-point"},
            "name": {"type": "keyword"}
        }
    }
}

Any other suggestions? Thank you.

#2

Limited nested fields support was added to Kibana in 7.6 with the following features

  • Index Patterns will detect Nested Fields correctly
  • You will be able to watch Nested Fields in Discover
  • Filtering on Nested Fields via the filter bar works
  • KQL allows searching for nested fields (see the [KQL documentation]

Maps does not support nested geo_point fields. There is an open issue, https://github.com/elastic/kibana/issues/35135. Would you mind commenting on the issue with your use case?

2 Likes
#3

I will comment as suggested. Thank you for clarifying for me.

#4

Can I get some support for writing an ingest pipeline that will walk the array of nested objects an pull out the "coordinates" field into a separate field in the index that Maps can use?
I can provide details.

I tried using an ingest pipeline with a "foreach" to walk the list and then an "append" within the "foreach" to copy each "coordinates" object to a new field (a list of "coordinates") for the index BUT "coordinates" is getting converted to a string and lat/long are no longer recognized as a "geo_point". think I I understand why that is happening in the append processor.

Can you suggest a better way to access the "coordinates" objects from the nested list?

Mapping:

{ 
    "mappings": {
        "properties": {
            "nationsTravelled": {
                "type": "nested",
                "properties":  {
                    "coordinates": {"type": "geo_point"},
                    "name": {"type": "keyword"}
                }
            },
           "nationsTravelledCoordsList": {"type":"geo_point"}
       }
}

Ingest pipeline piece:

{
     "foreach": {
         "field": "nationsTravelled",
         "processor": {
             "append": {
                 "field": "nationsTravelledCoordList",
                  "value": ["{{_ingest._value.coordinates}}"]
             }
         }
     }
}
#5

Hi @bgiordano,

I'd suggest you post your last question as a new topic, this time in the Elastic Stack -> Elasticsearch forum using the stack-ingest-management tag so you reach the correct audience for your issue.

#6

Nevermind, I think I got it. Instead of a foreach processor I used a painless script. You have below the full set of instructions from the dev console but the main part is the script itself that simply loops over the nested field and adds the coordinates to a new array that is later assigned to the nationsTravelledCoordsList field.

Mind that this probably would benefit from some type checking.

       int records = ctx.nationsTravelled.length;
       List result = []; 
       for (int i = 0; i < records; i++){
         result.add(ctx.nationsTravelled[i].coordinates)
       }
       ctx.nationsTravelledCoordsList = result ;

Let us know if this was helpful.

Script on DevTools 
# Remove the index if it exists
DELETE test_index_nested

# Create the index with the nested field and the outer list to receive the data
PUT test_index_nested
{ 
    "mappings": {
        "properties": {
            "nationsTravelled": {
                "type": "nested",
                "properties":  {
                    "coordinates": {"type": "geo_point"},
                    "name": {"type": "keyword"}
                }
            },
           "nationsTravelledCoordsList": {"type":"geo_point"}
       }
    }
}

# Create a pipeline with a script processor
PUT _ingest/pipeline/nested_test
{
  "processors": [
    {
     "script": {
       "source": """
       int records = ctx.nationsTravelled.length;
       List result = []; 
       for (int i = 0; i < records; i++){
         result.add(ctx.nationsTravelled[i].coordinates)
       }
       ctx.nationsTravelledCoordsList = result ;
       """
     }
    }
  ]
}

# Test the pipeline, mind that in string notation the order is lat/lon (!)
POST _ingest/pipeline/nested_test/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "nationsTravelled": [
          {
            "coordinates": "40.797177,-73.960648",
            "name": "New York"
          },
          {
            "coordinates": "19.425154,-99.113159",
            "name": "Mexico City"
          },
          {
            "coordinates": "-33.449777,-70.648956",
            "name": "Santiago de Chile"
          },
          {
            "coordinates": "-33.961586,18.462524",
            "name": "Cape Town"
          }
        ]
      }
    }
  ]
}

# Insert the document
POST test_index_nested/_doc?pipeline=nested_test
{
  "nationsTravelled": [
          {
            "coordinates": "40.797177,-73.960648",
            "name": "New York"
          },
          {
            "coordinates": "19.425154,-99.113159",
            "name": "Mexico City"
          },
          {
            "coordinates": "-33.449777,-70.648956",
            "name": "Santiago de Chile"
          },
          {
            "coordinates": "-33.961586,18.462524",
            "name": "Cape Town"
          }
        ]
}

# Check the result
GET test_index_nested/_search