Getting Kibana's geo hash with geo_point type to work

Elastic Stack Kibana
1

I posted an issue on Github, forgetting all about the discussion forums here.

issue outlined on github

logstash 2.2.0
kibana 4.4.0
elasticsearch 2.2.x

#copy / paste from github#

I have the following mapping applied to an index, pulled via _mapping API endpoint:

"cse-2016.02.09" : {
"mappings" : {
"event" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "strict_date_optional_time||epoch_millis"
},
"host" : {
"type" : "string"
},
"src_ip" : {
"type" : "string",
"index" : "not_analyzed"
},
"src_ip_geo" : {
"type" : "nested",
"properties" : {
"area_code" : {
"type" : "long"
},
"city_name" : {
"type" : "string"
},
"continent_code" : {
"type" : "string"
},
"coordinates" : {
"type" : "geo_point"
},
"country_code2" : {
"type" : "string"
},
"country_code3" : {
"type" : "string"
},
"country_name" : {
"type" : "string"
},
"dma_code" : {
"type" : "long"
},
"ip" : {
"type" : "string"
},
"latitude" : {
"type" : "double"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "double"
},
"postal_code" : {
"type" : "string"
},
"real_region_name" : {
"type" : "string"
},
"region_name" : {
"type" : "string"
},
"timezone" : {
"type" : "string"
}
}
},
"src_ip_network" : {
"properties" : {
"asn" : {
"type" : "string"
},
"number" : {
"type" : "string"
}
}
},
"src_latitude" : {
"type" : "string",
"index" : "not_analyzed"
},
"src_longitude" : {
"type" : "string",
"index" : "not_analyzed"
},
}
},
"default" : { }
}
}
My config for the index is:

input {
file {
path => "/var/log/community/events.log"
start_position => "end"
}
}

filter {
json {
source => "message"
}

geoip {
source => "src_ip"
target => "src_ip_geo"
database => "/opt/GeoLiteCity.dat"
add_field => [ "[src_ip_geo][coordinates]", "%{[src_ip_geo][longitude]}" ]
add_field => [ "[src_ip_geo][coordinates]", "%{[src_ip_geo][latitude]}" ]
}
mutate {
convert => [ "[src_ip_geo][coordinates]", "float"]
}

geoip {
source => "dst_ip"
target => "dst_ip_geo"
database => "/opt/GeoLiteCity.dat"
add_field => [ "[dst_ip_geo][coordinates]", "%{[dst_ip_geo][longitude]}" ]
add_field => [ "[dst_ip_geo][coordinates]", "%{[dst_ip_geo][latitude]}" ]
}

mutate {
convert => [ "[dst_ip_geo][coordinates]", "float"]
}

geoip {
source => "src_ip"
target => "src_ip_network"
database => "/opt/GeoIPASNum.dat"
}
}

output {
elasticsearch {
hosts => ["10.0.100.36:9200"]
index => "cse-%{+YYYY.MM.dd}"
document_type => "event"
template_name => "community_sensor_event"
template => "/etc/logstash/cse-template.json"
template_overwrite => true
manage_template => true
}
}
my mappings template consists of the fields from _mappings api for the relevant fields I am trying to put on the kibana tile map are:

"src_ip_geo": {
"type": "nested",
"properties": {
"coordinates": {
"type": "geo_point"
},
"location": {
"type": "geo_point"
}
}
},
I have not been able to dig out why kibana is erroring, any ideas?

`
pulling the JSON from the log lines, shows:

"src_ip_geo": {
  "ip": "203.xxx.xxx.xxx",
  "country_code2": "LK",
  "country_code3": "LKA",
  "country_name": "Sri Lanka",
  "continent_code": "AS",
  "latitude": 7,
  "longitude": 81,
  "timezone": "Asia/Colombo",
  "location": [
    81,
    7
  ],
  "coordinates": [
    81,
    7
  ]
},

`

I have not found anything useful in any Kibana logs, so I am not sure how to proceed.

2

You have src_ip_geo mapped as;

"src_ip_geo" : {
"type" : "nested",
"properties" : {
"area_code" : {
"type" : "long"
},

Which is not a geo point.

3

Mark,

I have the type on src_ip_geo set to "nested" because it has additional fields below it. I could be very wrong in my understanding of how this should be done, but that is what I was able to gather from reading the documentation as well as a bit of testing. I have the specific fields I want to be geo_points specified within that object:

 "properties": {
        "src_ip_geo": {
          "type": "nested",
          "properties": {
            "coordinates": {
              "type": "geo_point"
            },
            "location": {
              "type": "geo_point"
            }
          }
        },

Does the nested replace what I am attempting to do? Going to reach out on twitter/IRC and I'll update this thread with any progress.

4

That looks fine then.

KB sees the field as a geopoint?