Getting GeoIP field for NGINX Logs

[jaysbeekay (Jonathan)]

Hi,

I am having trouble trying to get the location from NGINX logs to be shown as a GeoIP (instead of a float) within Kibana to visualise on a dashboard

I am currently filebeats to ship logs from my NGINX server to Logstash for processing. My Logstash configuration file looks like this:

   input {
        beats {
            host => "0.0.0.0"
            port => 5044
        }
    }

    filter {
     grok {
       match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
       overwrite => [ "message" ]
     }
     mutate {
       convert => ["response", "integer"]
       convert => ["bytes", "integer"]
       convert => ["responsetime", "float"]
     }
     geoip {
       source => "clientip"
       target => "geoip"
       add_tag => [ "nginx-geoip" ]
     }
     date {
       match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
       remove_field => [ "timestamp" ]
     }
     useragent {
       source => "agent"
     }
    }

    output {
     elasticsearch {
       hosts => ["localhost:9200"]
       index => "weblog"
       document_type => "my_type"
     }
     stdout { codec => rubydebug }
    }

I have tried creating a custom index using PUT _template/weblog and setting "location" : {"type" : "geo_point"},(along with all other field mappings) and receive a message stating:

    #! Deprecation: Deprecated field [template] used, replaced by [index_patterns]
            {
              "acknowledged" : true
            }

I then deleted the old index and when it is recreated though the receipt of NGINX logs from Logstash, I view the mapping, location is changed back to a float.

Any help appreciated. Thanks

[fadjar340 (Fadjar Tandabawana)]

Create custom mapping for your needs

{
"geoip" : {
  "dynamic": true,
  "properties" : {
    "ip": { "type": "ip" },
    "location" : { "type" : "geo_point" },
    "latitude" : { "type" : "half_float" },
    "longitude" : { "type" : "half_float" }
 }
}

Put into output

output {
   elasticsearch {
     hosts => ["http://elastic01:9200"]
     template => "/file/path/geo-template.json"
     template_overwrite => true
     manage_template => true
   }
#    stdout { codec => rubydebug }
}