Logstash geoip filter targeting a geo_point field

Elastic Stack Kibana
1

Kibana, Elasticsearch and Logstash 6.2.2.

Based on IP addresses in our log data we want to view the location of site visitors on a map in Kibana.

Our mapping is setup with a field ip of type ip and another field location of type geo_point and Logstash is processing the IP as follows:

geoip {
    source => "ip"
	target => "location"
}

The Logstash output is:

{                                                        
    ...   
     "location" => {                                     
           "region_name" => "Texas",                     
              "location" => {                            
            "lat" => 32.7791,                            
            "lon" => -96.8028                            
        },                                               
             "longitude" => -96.8028,                    
           "region_code" => "TX",                        
         "country_code3" => "US",                        
           "postal_code" => "75202",                     
         "country_code2" => "US",                        
              "timezone" => "America/Chicago",           
          "country_name" => "United States",             
              "latitude" => 32.7791,                     
                    "ip" => "63.143.42.246",             
        "continent_code" => "NA",                        
              "dma_code" => 623,                         
             "city_name" => "Dallas"                     
    },                                                   
      ...

However ES is rejecting the indexing and Logstash reports:

[2018-03-16T19:31:21,296][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"access-app02-test_2018-03-16T17:47:11.000Z_2113", :_index=>"access-dev", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x2e5312a4], :response=>{"index"=>{"_index"=>"access-dev1", "_type"=>"doc", "_id"=>"access-app02-test_2018-03-16T17:47:11.000Z_2113", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"field must be either [lat], [lon] or [geohash]"}}}}}

Not sure what that means: geohash is not a field type AFAIK and [lat], [lon] are subfields of [location].

"mappings": {
"doc": {
"properties": {
"bytesSent": {
"type": "long"
},
"ip": {
"type": "ip"
},
"location": {
"type": "geo_point"
},
...

I might add, at this juncture, that the geoip filter docs are woefully lacking in proper examples. Indeed, their examples (add_field, remove_field) look like a copy-paste error from the mutate plugin.

2

What does the Logstash config look like?

You are right on the second part, these are generic functions across most plugins and the docs replicate that. The docs are open source though, and we welcome PRs :slight_smile:

3

Which Logstash Config do you mean? Our logstash.yml is empty and I've posted the geoip filter part of our pipeline. Here's the full pipeline:

input {
  beats {
    port => 5045
    host => "0.0.0.0"
  }
}

filter {
	grok { 
		match => { "message" => "\[%{HTTPDATE:timestamp}\] %{IPORHOST:ip1} (?:%{IPORHOST:ip2}|-) %{POSINT:status} \"%{WORD:method} %{NOTSPACE:url} HTTP/%{NUMBER:httpversion}\" %{NUMBER:bytesSent} %{NUMBER:timeTaken} %{NOTSPACE:sessionId} \"%{DATA:userAgent}\"" }
	}
	date {
		match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
	}
	if [ip2] {
		mutate { add_field => { "ip" => "%{ip2}" } }
	} else {
		mutate { add_field => { "ip" => "%{ip1}" } }
	}
	# Lookup geo location from IP
	geoip {
		source => "ip"
		target => "location"
	}
	mutate {
		update => {
			"source" => "%{[fields][source]}"
		}
		rename => {
			"@timestamp" => "timestamp"
		}
		add_field => {
			"server" => "%{[fields][server]}"
			"logfile" => "%{[fields][logfile]}"
			"[@metadata][index]" => "access"
			"[@metadata][system]" => "%{[fields][system]}"
			"[@metadata][id]" => "%{logfile}_%{timestamp}_%{offset}"
		}
		remove_field => ["fields", "offset", "@version", "@timestamp", "prospector", "host", "beat", "tags", "message", "ip1", "ip2", "httpversion"]
	}
}

output {
	stdout {codec => rubydebug { metadata => false } }
	elasticsearch { hosts => ["blah-test"] index => "%{[@metadata][index]}-%{[@metadata][system]}" document_id => "%{[@metadata][id]}" }
}
4

Log example (masked):

[16/Mar/2018:18:42:54 +0100] 100.9.122.202 216.0.97.94 200 "GET /path/somelink HTTP/1.1" 0 15 0E1986C9E11A382597665B3B42822E68 "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
[16/Mar/2018:18:45:18 +0100] 100.9.122.202 100.9.0.8 200 "GET /path/otherlink1 HTTP/1.1" 0 0 0E1986C9E11A382597665B3B42822E69 "Mozilla/5.0 (Windows NT; Windows NT 6.3; de-CH) WindowsPowerShell/5.1.14409.1012"
[16/Mar/2018:18:47:11 +0100] 100.9.122.202 63.1003.42.246 200 "GET /path/otherlink2 HTTP/1.1" 0 16 4A57FC26280CD4231BD83284A2C53EDB "Mozilla/5.0+(compatible; test)"
[16/Mar/2018:18:42:54 +0100] 100.9.122.202 216.0.97.94 200 "GET /path/somelink HTTP/1.1" 0 15 0E1986C9E11A382597665B3B42822E68 "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
[16/Mar/2018:18:45:18 +0100] 100.9.122.202 100.0.121.8 200 "GET /path/otherlink1 HTTP/1.1" 0 0 0E1986C9E11A382597665B3B42822E69 "Mozilla/5.0 (Windows NT; Windows NT 6.3; de-CH) WindowsPowerShell/5.1.14409.1012"
[16/Mar/2018:18:47:11 +0100] 100.9.122.202 63.0.42.246 200 "GET /path/otherlink2 HTTP/1.1" 0 16 4A57FC26280CD4231BD83284A2C53EDB "Mozilla/5.0+(compatible; test)"
[16/Mar/2018:18:47:11 +0100] 100.9.122.202 63.0.42.246 200 "GET /path/otherlink2 HTTP/1.1" 0 16 4A57FC26280CD4231BD83284A2C53EDB "Mozilla/5.0+(compatible; test)"
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.