Not Able to Create Geopoint data

Hi I'm trying to make some map based on some ip address which are private, so I cannot use geoip filter here. Currently I'm implementing some naive method to assign the (lat, lon) coordinates using some if else conditions based on the pattern of sub nets. The problem is after I try to convert the original data to geo point type data, I still get this error:

index pattern does not contain any of the following field types: geo_point

My data flow is filebeat -> logstash -> ES and my logstash config file looks like this:
input {
beats {
port => "5044"
}
}
filter {
json {
source => "message"
}
date {
match => ["timestamp","MMM dd, yyyy hh:mm:ss a"]
target => "@timestamp"
}
ruby {
code => 'event.set("ip2", event.get("ipAddressGuest").split(".")[1])
event.set("ip3", event.get("ipAddressGuest").split(".")[2])'
}
mutate {
convert => { "ip2" => "integer"
"ip3" => "integer"}
}
# More if else statements in the future
if [ip2] == 241 {
ruby {
code => 'event.set("lat", 43.5)
event.set("lon", -80.2)'
}
mutate {
rename => {
'lon' => "[location][lon]"
'lat' => "[location][lat]"
}
}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => 'ip_name-%{+YYYY.MM.dd}'
}
}

I look at some of the previous discussions regarding this and I think I might I need to create a template where i assign geopoint data type to location. So I make my own template file like this and add that to the output part of my logstash config file but it still won't work.

{
"template" : "logstash-",
"version" : 50001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "norms" : false},
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword" }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date", "include_in_all": false },
"@version": { "type": "keyword", "include_in_all": false },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
},
"location" : {"type": "geo_point"}
}
}
}
}
Can anybody help me with this? Really appreciate it.

Ran through the same exact problem yesterday and the template didn't work.
I would always get type number on Kibana.

Using the template settings and looking at the LS logs i was able to see that my template is being applied to elastic search from LS, but once in Kibana the field is not type "geo_point"

Im thinking and just realized this today, but haven't tested it yet that your template must have this at the end to work, just before you close the last bracket in your template "template": "yourIndexPattern-*"

You may need the manage template setting enabled as well.

In my case I was using geoip filter and wanted to keep using the logstash default template, after giving up on new templates and realizing that the default LS template will take care of my geoip mapping.

Hi thanks for the reply. Sorry I don't quite get what do you mean. Can you share more details?

Im thinking and just realized this today, but haven't tested it yet that your template must have this at the end to work, just before you close the last bracket in your template "template": "yourIndexPattern-*"

So what should be added to where? Are you talking about the template.json file?

Also what do you mean by "manage template setting enabled". I do target my logstash conf file at the template I am using. Do I miss something else here?

Hi and sorry for the confusion.

All I was trying to say is that i had the same issue you are having here.
I used a template and I can see in the logstash logs its being applied but when I look in Kibana I still dont see the right "geo_point mapping" to the fields I have specified in my template, simply was not working.

So what I have realized in one of the templates JSON files i.e. filebeat, the last line in the template file is the name pattern for your index. Simply in your template add this "template": "ip_name-*" just before you close the last bracket. I think that makes sure it gets applied to all indices following that pattern/name.

Again that's what I was missing im my template and haven't had the chance to test it yet and see if that would change anything an get me the right mapping. As for template management I came across a post that suggested that its should be enabled but maybe for an older version.

So I change the very first line
"template" : "logstash-",
in my template file into
"template": "ip_name-
".
but it still does not work. Anyway thank you.

OK, so why my local template still does not work remains unknown. I am very sure that my logstash is pointed to that template file.
Anyway I use the Dev Tools tab on Kibana and create my customize template there (I just copy paste the content of my local template file). Now I get the geo map working.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.