How to convert geoip's ingest plugin data into "geo_point" type?

[John16 (John)]

Hello,

I am using "geoip" ingest plugin to obtain coordinates for a given IP (excerpt from my pipeline):
"geoip" : { "field" : "ip", "target_field" : "geoip", "properties": [ "location" ] },

It results in the following fields added into my data:
"geoip": { "location": { "lon": -122.33, "lat": 47.608 } },

But Kinaba does not recognize that "geoip" field as of type "geo_point". Is there any way to convert that geoip["location"] into another variable of geo_point type?
Or even better to produce the result in geo_point type straight away?

Thanks!

[spinscale (Alexander Reelsen)]

Hey,

you need to make sure, that geoip.location is a geo point in your mapping in order to make this work. This cannot be done with the processor, but must be done before.

--Alex

[John16 (John)]

Okay, I added "geoip.location": { "type": "geo_point" }, to my mapping.
And
"geoip" : { "field" : "ip", "target_field" : "geoip", "properties": [ "location" ] }

to my pipeline.

Now I receive the following error when trying to import my log:
2017/02/07 15:27:28.019061 client.go:432: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"[geoip.location] is defined as an object in mapping [test_log] but this name is already used for a field in other types"}

[John16 (John)]

I also tried to create a mapping of type geo_point:
"g": { "type": "geo_point" },

and then "rename" geoip.location in pipeline into that field in hope ES will cast it to proper type:

"geoip" : { "field" : "ip", "target_field" : "geoip", "properties": [ "location" ] }, "rename":{ "field": "geoip.location", "target_field": "g" }

but I get the same error:
2017/02/07 19:00:14.946821 client.go:432: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"[g] is defined as an object in mapping [speller_log] but this name is already used for a field in other types"}

So I am really confused: I do not understand how to assign the result of ingest processor "geoip" to a field of type geo_point to be used in Kibana...

[talevy (Tal Levy)]

The reason you are receiving these exceptions is because you have already indexed documents with those fields in your index. Elasticsearch took the liberty of auto-guessing the mapping to use for that field (and it did not choose geo_point). You may need to reindex into a new index and declare the geo_point mapping on the field at index-creation before you begin indexing documents.

[John16 (John)]

@talevy

The reason you are receiving these exceptions is because you have already indexed documents with those fields in your index.

Well, no. During experiments I always delete existing index(es) first.
I double-checked. I drop all indexes and then load single log line.

I get this error:
2017/02/09 07:05:16.521191 client.go:432: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"[geoip.location] is defined as an object in mapping [test_log] but this name is already used for a field in other types"}

in my mapping:
"geoip.location": { "type": "geo_point" }

in my pipeline:
"geoip" : { "field" : "ip", "target_field" : "geoip", "properties": [ "location" ] }

my log line:
2017-01-29T00:00:06 189 200 194.0.68.126 GET / -

[Matthew_Bullock]

Hi John I had the same issue I found

deleting the index via curl
deleting the pipeline via curl
deleting the physical index in kibana in management

Add the geopoint at the same time as creating the index. So i use waf as my index you will use the test_log

curl -XPUT "http://localhost:9200/waf?pretty" -d'
{
"mappings": {
"logs": {
"properties": {
"occurred_at": {
"type": "date",
"format": "date_time||date_time_no_millis"
},
"geoip.location": {
"type": "geo_point"
}
}
}
}
}'

Then create the pipeline:

curl -XPUT "http://localhost:9200/_ingest/pipeline/geoip-info" -d'
{
"description": "Add geoip info",
"processors": [
{
"geoip": {
"field": "ip",
"properties": ["location"],
"ignore_failure": true
}
}
]
}'

Then import your data.

[John16 (John)]

@Matthew_Bullock

1. deleting the index via curl
2. deleting the pipeline via curl
3. deleting the physical index in kibana in management

1. I do curl -XDELETE 'localhost:9200/test-idx?pretty'
2. I do not delete old pipeline, but rather modify the existing:
   curl -XPUT 'localhost:9200/_ingest/pipeline/test-pipeline?pretty' -H 'Content -Type: application/json' -d' .....
   I think explicit delete is not necessary, at least my changes are applied correctly.
3. What do you mean "delete physical index"? Doesn't 1) do that?

[John16 (John)]

So far I found the only solution that works for me:

{ "geoip" : { "field" : "ip", "target_field" : "geoip_tmp", "properties": [ "location" ] } }, { "set": { "field": "geoip", "value": "{{geoip_tmp.location.lat}}, {{geoip_tmp.location.lon}}" } }, { "remove" : { "field": "geoip_tmp" } }