Geo_point mapping python and Streamsets fails with Elasticsearch

Elastic Stack Elasticsearch
1

I have this mapping in elasticsearch

"mappings": {
      "properties": {
            "fromCoordinates": {"type": "geo_point"},
            "toCoordinates": {"type": "geo_point"},
            "seenCoordinates": {"type": "geo_point"},
        }
    }

With the kibana's console, there is no problem with all possible combinations of geo_ip fields supported by elasticsearch, i.e:

(lat, lon)

PUT /anindex/_doc/1
{
   "fromCoordinates": {
     "lat": 36.857200622558594    
     "lon": 117.21600341796875,

  },
  "toCoordinates": {
    "lat": 22.639299392700195    
    "lon": 113.81099700927734,

  },
  "seenCoordinates": {
     "lat": 36.91663    
     "lon": 117.216,
   }
}

(lon,lat)

PUT /anindex/_doc/2
{
 "fromCoordinates": [36.857200622558594, 117.21600341796875], 
 "toCoordinates": [22.639299392700195, 113.81099700927734], 
 "seenCoordinates": [36.91663, 117.216] 
}

But a I tried inserting, into elasticsearch, the data through python, and I always have this error:

RequestError(400, 'illegal_argument_exception', 'mapper [fromCoordinates] of different type, current_type [geo_point], merged_type [ObjectMapper]')

In python, I construct the json from a dictionary, and this is the result when I printed:

fromCoordinates = {}
fromCoordinates['lat'] = fromLat  
fromCoordinates['lon'] = fromLon 

dataDictionary.update({'fromCoordinates': fromCoordinates , 'toCoordinates': toCoordinates, 'seenCoordinates': seenCoordinates})

print(json.dumps(dataDictionary).encode('utf-8'))

{"fromCoordinates": {"lat": 43.9962005615, "lon": 125.684997559}, 
"toCoordinates": {"lat": 40.080101013183594, "lon": 116.58499908447266}, 
"seenCoordinates": {"lat": 33.62672, "lon": 109.37243}}

and load with this

data = json.dumps(dataDictionary).encode('utf-8')
es.create(index='anindex', doc_type='document', id=0, body=data)

The array version has the same problems:

fromCoordinates = [fromLon, fromLat]

This is the json created and printed in python:

{"fromCoordinates": [113.81099700927734, 22.639299392700195], 
  "toCoordinates": [106.8010025024414, 26.53849983215332], 
   "seenCoordinates": [107.46743, 26.34169]}

In this case I have this response

RequestError: RequestError(400, 'mapper_parsing_exception', 'geo_point expected')

The same error occurs if I try with StreamSets to elasticsearch, having the both types of json shown before:

mapper [fromCoordinates] of different type, current_type [geo_point], merged_type [ObjectMapper]

This is the mapping loaded in elastic:

GET /anindex/_mapping
{
  "anindex" : {
    "mappings" : {
      "properties" : {
        "fromCoordinates" : {
          "type" : "geo_point"
        },
        "toCoordinates" : {
          "type" : "geo_point"
        },        
        "seenCoordinates" : {
          "type" : "geo_point"
        }
       }
      }
    }
 }

Any ideas?

UPDATE:
With curl, postman, kibana's console there is no problem with that. It clearly seems that the JSON nested fields are treated as an object and not "pure" json (plain) when I send them from python or from StreamsSets. I also tested with python and StreamSets without nested fields, and everything is ok. I also tried, without creating the mapping first and leaving elastic to guess and create by itself, and elastic mapped the coordinates as numbers or strings.
I also tried building the json "by hand" and send it, and get the same error.

2

SOLUTION:

After the example given in stackoverflow I realized that the doc_type parameter in
'es.create(index='anindex', doc_type='document', id=0, body=data)'
is causing the error, I removed it, and it worked..... But I still wondering why in StreamSets has the same error... but I`ll continue with python.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.