Convert latitude and longitude fields into geo_point type,

Elastic Stack Elasticsearch
1

Hello,
I have the below fields in my index.

index pattern : s3-*
service.action.networkConnectionAction.remoteIpDetails.geoLocation.lat - type is number
service.action.networkConnectionAction.remoteIpDetails.geoLocation.lon - type is number

service.action.portProbeAction.portProbeDetails.remoteIpDetails.geoLocation.lat - type is number
service.action.portProbeAction.portProbeDetails.remoteIpDetails.geoLocation.lon - type is number
I am trying to plot a coordinated map using the above fields. However Kibana requires the fields to be of geo_point type. I tried using a template as below ( ES 7.1 )

PUT _template/gdt
{
"template": "s3-*",
"settings": {},
"mappings": {
"properties": {
"service.action.portProbeAction.portProbeDetails.remoteIpDetails.geoLocation": {
"type": "geo_point"
},
"service.action.networkConnectionAction.remoteIpDetails.geoLocation": {
"type": "geo_point"
}
}
}
}

However no luck in getting it to work. Any advise?

2

Anyone any inputs please?

3

Welcome!

Could you please check what the actual mapping is for your index?

4

Hello, The current mapping is as below
"geoLocation" : {
"properties" : {
"lat" : {
"type" : "float"
},
"lon" : {
"type" : "float"
}
}
},

5

So it's seems that your index has not been created using the template.

6

Basically, I am using requests.post to create and load the json file to ES. If I execute the template API provided in my question, it creates the mapping as per the template. However, when the process is ran, it can't load the data into ES and fails with a 400 error. It seems to be failing because the json data contains many other fields and not just the "lon" and "lat" fields. Basically, I was using the fields as per the content of the json data in ES.

This is the python code snippet
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
resp = requests.post(es_Url, auth=awsauth, headers=headers, json=docData)

In the above, the docData is basically the son load of the file content
docData = json.loads(line)

7

I don't have a clear understanding of what you are doing and how.

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

8

Apologies for the confusion. Summarizing my problem as below

  1. I am using a python script to create index and load the son data into ES. Specifically using the python module "requests.post".
  2. The index is created just fine. The fields and mappings are created just fine as per the contents of the son data.
  3. The mapping is created as below. Notice the fields "Lon" and "lat" are of the float type in the mapping for geoLocation.
  4. I wanted to create a visualisation using coordinated map using the above fields mentioned. However ES expects the type to be geo_point. Now, my question is - how do I convert it to a geo_point type?
    "service" : {
    "properties" : {
    "action" : {
    "properties" : {
    "actionType" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    },
    "dnsRequestAction" : {
    "properties" : {
    "blocked" : {
    "type" : "boolean"
    },
    "domain" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    },
    "protocol" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    }
    }
    },
    "networkConnectionAction" : {
    "properties" : {
    "blocked" : {
    "type" : "boolean"
    },
    "connectionDirection" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    },
    "localPortDetails" : {
    "properties" : {
    "port" : {
    "type" : "long"
    },
    "portName" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    }
    }
    },
    "protocol" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    },
    "remoteIpDetails" : {
    "properties" : {
    "city" : {
    "properties" : {
    "cityName" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    }
    }
    },
    "country" : {
    "properties" : {
    "countryName" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    }
    }
    },
    "geoLocation" : {
    "properties" : {
    "lat" : {
    "type" : "float"
    },
    "lon" : {
    "type" : "float"
    }
    }
    },
9

You need to create the proper mapping when you create your index then index the data.
You can't change it after.
Here you need to reindex your data.

10

Thanks. I believe the issue is with the python module. If I create the mappings beforehand and then try to write the data, it just fails to do so.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.