Need some help with Geo Enrichment - SOLVED

Elastic Stack Logstash
9

Really need some help warkolm, banging my head on this one :frowning:

Here is what I've been able to do thus far:

  1. Add in a filter mutate to split airport by the ',' so we now have an array from the string containing the latitude and longitude
  2. Add in a filter mutate to add the fields called latitude and longitude defined as airport[0] and airport[1]
  3. Then add in a filter mutate to convert them from string which is what natively is happening into a float
  4. In filter mutate name the longitude and latitude as: [location][lon] & [location][lat]
  5. In my output section I define manage_template as true and pass in the template location and template_override = true

When I run I get the following error: "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [default]: Mapping definition for [location] has unsupported parameters: [dynamic : true]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"Mapping definition for [location] has unsupported parameters: [dynamic : true]"}}}}, :level=>:warn}

Here is the config
input {
jdbc {
jdbc_connection_string => "jdbc:sqlserver://host:1433;Database=db"
jdbc_user => "user"
jdbc_password => "password"
jdbc_driver_library => "/Users/wtaylor/Downloads/sqljdbc_4.0/enu/sqljdbc4.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
statement => "SELECT top 5 * from SALES where PURCHASE_DATE_ET = '2016-02-22' and STANDARD_AMOUNT > 0"
}
}

filter {
translate {
field => "destination_airport_code"
dictionary_path => "/Users/wtaylor/Downloads/logstash-2.2.2/bin/geocord.yaml"
fallback => "unknown"
destination=> "airport"
}
}

filter {
mutate {
split => {"airport" => ","}
}
}

filter {
mutate {
add_field => ["latitude","%{[airport[0]}"]
add_field => ["longitude","%{[airport[1]}"]
}
}

filter {
mutate {
convert => { "longitude" => "float" }
convert => { "latitude" => "float" }
}
}

filter{
mutate {
rename => {
"longitude" => "[location][lon]"
"latitude" => "[location][lat]"
}
}
}

output {
stdout { codec => json_lines }

elasticsearch {
index => "bre"
document_type => "purchase"
manage_template => true
template => "/Users/wtaylor/Downloads/logstash-2.2.2/bin/template.json"
template_overwrite=>"true"
}
}

My Template Json looks like:
{
"template" : "bre",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"properties" : {
"@timestamp": { "type": "date", "doc_values" : true },
"@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"location" : {
"type" : "geo_point",
"dynamic": true,
"doc_values" : true,
"lat_lon": true
},
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"ip": { "type": "ip", "doc_values" : true },
"latitude" : { "type" : "float", "doc_values" : true },
"longitude" : { "type" : "float", "doc_values" : true }
}
}

  }
}

}
}

1 Like