Can't change index field mapping to geo_ip

[jonasv]

Hi,

I'm trying to get Kibana tile map get working, basically I'm after changing geoip.location field type to geo_ip . This is my index file:

{
"fedora26-2017.02.09": {
"aliases": {},
"mappings": {
"nginx-access": {
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"@version": {
"type": "string"
},
"beat": {
"properties": {
"hostname": {
"type": "string"
},
"name": {
"type": "string"
}
}
},
"browserdevice": {
"type": "string"
},
"browsermajor": {
"type": "string"
},
"browserminor": {
"type": "string"
},
"browsername": {
"type": "string"
},
"browseros": {
"type": "string"
},
"browseros_name": {
"type": "string"
},
"browserpatch": {
"type": "string"
},
"count": {
"type": "long"
},
"geoip": {
"properties": {
"area_code": {
"type": "long"
},
"city_name": {
"type": "string"
},
"continent_code": {
"type": "string"
},
"country_code2": {
"type": "string"
},
"country_code3": {
"type": "string"
},
"country_name": {
"type": "string"
},
"dma_code": {
"type": "long"
},
"ip": {
"type": "string"
},
"latitude": {
"type": "double"
},
"location": {
"type": "double"
},
"longitude": {
"type": "double"
},
"postal_code": {
"type": "string"
},
"real_region_name": {
"type": "string"
},
"region_name": {
"type": "string"
},
"timezone": {
"type": "string"
}
}
},
"input_type": {
"type": "string"
},
"offset": {
"type": "long"
},
"our_server": {
"type": "string"
},
"referer": {
"type": "string"
},
"request": {
"type": "string"
},
"request_arrival_time": {
"type": "string"
},
"request_body_bytes_sent": {
"type": "string"
},
"request_ip": {
"type": "string"
},
"request_length": {
"type": "string"
},
"request_processing_time": {
"type": "string"
},
"request_response_code": {
"type": "string"
},
"source": {
"type": "string"
},
"tags": {
"type": "string"
},
"type": {
"type": "string"
},
"upstream_response_time": {
"type": "string"
},
"user_agent": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1486636747483",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "97bE3qj5QWqsckpbf0TnqA",
"version": {
"created": "2040499"
}
}
},
"warmers": {}
}
}

And this is how I'm changing it (via app sense):

PUT fedora26-2017.02.09
{
"mappings": {
"nginx-access": {
"properties": {
"geoip": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
}
}

but getting this instead:
{
"error": {
"root_cause": [
{
"type": "index_already_exists_exception",
"reason": "already exists",
"index": "fedora26-2017.02.09"
}
],
"type": "index_already_exists_exception",
"reason": "already exists",
"index": "fedora26-2017.02.09"
},
"status": 400
}

So I'm not sure how to change field type here ? What exactly needs to be done ?

Thank you

[wenpos (Fanfan)]

this may give some suggestions:
https://www.elastic.co/guide/en/elasticsearch/reference/5.2/geo-point.html

[jonasv]

I've read that page already, there is nothing related to my problem.

[wenpos (Fanfan)]

The last EXCEPTION indicate an incorrect url to rebuild the existed index. how would you index on the mappings?

[Matthew_Bullock]

You are already defining your geoip in the first index creation you need to define the geo_point within this.

"location": {
"type": "double"

[jonasv]

Wenpos:

I'm not sure if I understand your question, I'm very new to elk, just trying to make tile map to work in kibana. So if you could please be more specific then I could try to answer..

Matthew:

I've tried to change from geo_ip to double, but still getting this:
{
"error": {
"root_cause": [
{
"type": "index_already_exists_exception",
"reason": "already exists",
"index": "fedora26-2017.02.09"
}
],
"type": "index_already_exists_exception",
"reason": "already exists",
"index": "fedora26-2017.02.09"
},
"status": 400
}

[kravigupta (Ravi Kumar Gupta)]

Try this -

PUT fedora26-2017.02.09/_mappings/nginx-access
{
"properties": {
"geoip": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}

You should be calling _mapping endpoint. Calling index name only will make it an index api call.

HTH

[jonasv]

Thanks Kravigupta.

I get this message now when I send your query:

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "mapper [geoip.location] of different type, current_type [double], merged_type [geo_point]"
}
],
"type": "illegal_argument_exception",
"reason": "mapper [geoip.location] of different type, current_type [double], merged_type [geo_point]"
},
"status": 400
}

[kravigupta (Ravi Kumar Gupta)]

Create an scripted field using Kiban > Index Patters > Fedora* > Scripted fields.

name it geoip.location
set value as
[geoip.latitude,geoip.longitude]

If this does not give you any error, go for it and then do what I suggested earlier.

[jonasv]

Same problem.

But why exactly I'm getting that error ? I'm just trying to change field type to geo_point but I can't find a way to do that

[jonasv]

I'm trying to put nginx-access mapping like this using app sense:

PUT fedora26-2017.02.09/_mappings/nginx-access
{
"fedora26-2017.02.09": {
"mappings": {
"nginx-access": {
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"@version": {
"type": "string"
},
"beat": {
"properties": {
"hostname": {
"type": "string"
},
"name": {
"type": "string"
}
}
},
"browserdevice": {
"type": "string"
},
"browsermajor": {
"type": "string"
},
"browserminor": {
"type": "string"
},
"browsername": {
"type": "string"
},
"browseros": {
"type": "string"
},
"browseros_name": {
"type": "string"
},
"browserpatch": {
"type": "string"
},
"count": {
"type": "long"
},
"geoip": {
"properties": {
"city_name": {
"type": "string"
},
"continent_code": {
"type": "string"
},
"country_code2": {
"type": "string"
},
"country_code3": {
"type": "string"
},
"country_name": {
"type": "string"
},
"ip": {
"type": "string"
},
"latitude": {
"type": "double"
},
"location": {
"type": "double"
},
"longitude": {
"type": "double"
},
"real_region_name": {
"type": "string"
},
"region_name": {
"type": "string"
},
"timezone": {
"type": "string"
}
}
},
"input_type": {
"type": "string"
},
"offset": {
"type": "long"
},
"our_server": {
"type": "string"
},
"referer": {
"type": "string"
},
"request": {
"type": "string"
},
"request_arrival_time": {
"type": "string"
},
"request_body_bytes_sent": {
"type": "string"
},
"request_ip": {
"type": "string"
},
"request_length": {
"type": "string"
},
"request_processing_time": {
"type": "string"
},
"request_response_code": {
"type": "string"
},
"source": {
"type": "string"
},
"tags": {
"type": "string"
},
"type": {
"type": "string"
},
"upstream_response_time": {
"type": "string"
},
"user_agent": {
"type": "string"
}
}
}
}
}
}

and in return I'm getting this:

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [fedora26-2017.02.09 : {mappings={nginx-access={properties={@timestamp={type=date, format=strict_date_optional_time||epoch_millis}, @version={type=string}, beat={properties={hostname={type=string}, name={type=string}}}, browserdevice={type=string}, browsermajor={type=string}, browserminor={type=string}, browsername={type=string}, browseros={type=string}, browseros_name={type=string}, browserpatch={type=string}, count={type=long}, geoip={properties={city_name={type=string}, continent_code={type=string}, country_code2={type=string}, country_code3={type=string}, country_name={type=string}, ip={type=string}, latitude={type=double}, location={type=double}, longitude={type=double}, real_region_name={type=string}, region_name={type=string}, timezone={type=string}}}, input_type={type=string}, offset={type=long}, our_server={type=string}, referer={type=string}, request={type=string}, request_arrival_time={type=string}, request_body_bytes_sent={type=string}, request_ip={type=string}, request_length={type=string}, request_processing_time={type=string}, request_response_code={type=string}, source={type=string}, tags={type=string}, type={type=string}, upstream_response_time={type=string}, user_agent={type=string}}}}}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [fedora26-2017.02.09 : {mappings={nginx-access={properties={@timestamp={type=date, format=strict_date_optional_time||epoch_millis}, @version={type=string}, beat={properties={hostname={type=string}, name={type=string}}}, browserdevice={type=string}, browsermajor={type=string}, browserminor={type=string}, browsername={type=string}, browseros={type=string}, browseros_name={type=string}, browserpatch={type=string}, count={type=long}, geoip={properties={city_name={type=string}, continent_code={type=string}, country_code2={type=string}, country_code3={type=string}, country_name={type=string}, ip={type=string}, latitude={type=double}, location={type=double}, longitude={type=double}, real_region_name={type=string}, region_name={type=string}, timezone={type=string}}}, input_type={type=string}, offset={type=long}, our_server={type=string}, referer={type=string}, request={type=string}, request_arrival_time={type=string}, request_body_bytes_sent={type=string}, request_ip={type=string}, request_length={type=string}, request_processing_time={type=string}, request_response_code={type=string}, source={type=string}, tags={type=string}, type={type=string}, upstream_response_time={type=string}, user_agent={type=string}}}}}]"
},
"status": 400
}

Which is nothing else but old mapping data. So in other words , I can get mapping data extract, but if I'm trying to send it back to ELK - it fails, which makes no sense to me.

[jonasv]

Guys, I've cracked the issue, please mark it as closed.

Thank you for those who helped.

[kravigupta (Ravi Kumar Gupta)]

Jonasv, it would help for others if you could post the solution.

[jonasv]

Sure.

If you have a look what I was doing here:

PUT fedora26-2017.02.09/_mappings/nginx-access
{
"fedora26-2017.02.09": {
"mappings": {
"nginx-access": {
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
...

So I was sending that payload to fedora26-2017.02.09/_mappings/nginx-access , but correct way is to delete
{
"fedora26-2017.02.09": {
"mappings": {
"nginx-access": {

and then send to fedora26-2017.02.09/_mappings/nginx-access

Also, it took me a while to find a simple sentence somewhere that once the data is in - you can't change existing index or mapping. So in real world - you create index structure and mapping (for example through kibana appsense) and then you load the data. I wish these words were somewhere in manual at the beginning, but unfortunately it is one big mess in my eyes. It is good and valuable if you need to read about some function (for example about grok or mutate - it is perfect), but if you are very new to ELK and such kind of search systems - there is no value in manual, better read tutorials and look for answers outside the scope of this website, anyway, it is just my opinion and ELK technology is not the easiest thing you can learn.