Geo_point

Elastic Stack Logstash
1

I am trying to use the mapping portion of Kibana to map all the points that I have in my database for a given time period. I just started ELK so I don't know a lot. I have been struggling to insert geo_point type of data into elastic 2.3 from logstash. My Kibana 4.5 doesn't seem to be able to recognize the points as geo points and when I try to select objects, nothing shows up in the tile maps. Here is my logstash configs. I am using the standard mappings that came with the application. HELP! Any suggestions would be appreciated. I believe I need to create a type of geo_point.

file: simple-out.conf

input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:oracle:thin:@//192.168.229.129:1521/XE"
# The user we wish to execute our statement as
jdbc_user => "dev"
jdbc_password => "dev"
jdbc_validate_connection => true
# The path to our downloaded jdbc driver
jdbc_driver_library => "/home/siteadm/lib/ojdbc6-11.2.0.1.0.jar"
# The name of the driver class for oracle
jdbc_driver_class => "Java::oracle.jdbc.OracleDriver"
#schedule => "0 * * * * "
# our query
statement => "SELECT r.id as request_id, r.tfpsysid as tfpsysid, r.central_tfpsys_id as central_tfpsys_id, r.tfpsys_location_title as financial_institution, r.tfpsys_location_address_line1 as fi_address, r.tfpsys_location_city as fi_city, r.tfpsys_location_state_code as fi_state, r.tfpsys_location_zip as fi_zip, t.latitude as latitude, t.longitude as longitude from requests r inner join tfpsys_locations t on r.tfpsysid=t.id where r.updated_date > sysdate-10"
}
}
filter {
if [latitude] and [longitude] {
mutate {
add_field => [ "[location]", "%{longitude}" ]
add_field => [ "[location]", "%{latitude}" ]
}
}
mutate {
convert => [ "[location]", "float" ]
}

}

output {
elasticsearch {
hosts => localhost
}
stdout { codec => dots }

}

2

If you inspect the mapping of an index in Elasticsearch I think you'll find something similar to this:

        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "float" },
            "longitude" : { "type" : "float" }
          }
        }

This tells us that the geoip.location field ([geoip][location] in Logstash notation) is a geo_point field. I don't think you'll find anything similar about the location field that you're currently using. So, adjust what your messages look like or change the index template so that the location field is also mapped as geo_point.

3

I am trying to do that and I guess I just don't understand enough how to map it. I was trying to use the filter. Can you point me in the right direction? How would my filter change?

4

I suppose

mutate {
  add_field => [ "[geoip][location]", "%{longitude}" ]
  add_field => [ "[geoip][location]", "%{latitude}" ]
}

would work. Or, again, if you have no need for the geoip parent field you might want to change your mappings instead.

5

Thank you! Just had to add the conversion to float and it works!

mutate {
add_field => ["[geoip][location]","%{longitude}"]
add_field => ["[geoip][location]","%{latitude}"]
}

    mutate {
            convert => [ "[geoip][location]", "float" ]
    }
6

To anyone that needs help. I also posted a template to elasticsearch

curl
-XPUT localhost:9200/template/atemplate_index -d '
{
"template": "atemplate-*",

"settings": {

"index.refresh_interval":

"60s"

},

"mappings": {

"_default_": {

  "_all": {"enabled":

true, "omit_norms": true},

  "dynamic_templates": [ {

    "message_field": {

      "match":

"message",

      "match_mapping_type":

"string",

      "mapping": {

        "type":

"string", "index": "analyzed",
"omit_norms": true

      }

    }

  }, {

    "string_fields": {

      "match": "*",

      "match_mapping_type":

"string",

      "mapping": {

        "type":

"string", "index": "analyzed",
"omit_norms": true,

        "fields": {

          "raw":

{"type": "string", "index":
"not_analyzed", "ignore_above": 256}

        }

      }

    }

  } ],

  "properties": {

    "@version": {

"type": "string", "index":
"not_analyzed" },

    "name": {"type":

"string"},

    "lonlat": { "type":

"geo_point" }
}
}
}
}
'