where can i change for type to ip??
"clientip" : {
"type" : "string",
"norms" : {
"enabled" : false
},
thanks so much
Hi warkolm, i have news:
i finally created my own "index pattern", i have my template, and when logstash starting catch my template, i see that with the kopf plugin but my problem continues
My geoip and client mappings:
"geoip": {
"dynamic": "true",
"properties": {
"location": {
"doc_values": true,
"type": "geo_point"
},
"longitude": {
"doc_values": true,
"type": "float"
},
"latitude": {
"doc_values": true,
"type": "float"
},
"ip": {
"doc_values": true,
"type": "ip"
}
}
},
for client ip:
"clientip": {
"doc_values": true,
"type": "ip"
},
I regenerated thel index and appears the new fields:
this is my apache configuration for logstash
filter {
if [type] == "apache_error" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
add_field => [ "received_from", "%{host}" ]
}
}
}
filter {
if [type] == "apache_access" {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
filter {
if [type] == "apache_access_ssl" {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
and the output for logstash
output {
elasticsearch {
host => "marioneto01"
cluster => "oknels"
protocol => "http"
index => "okn-%{+YYYY.MM.dd}"
template => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.5-java/lib/logstash/outputs/elasticsearch/okn-template.json"
template_name => "okn-*"
manage_template => "true"
#template_overwrite => "true"
}
stdout { codec => rubydebug }
}
ideas?? thanks
What kind of IP addresses do you have in your log ? Geoip filter is not supposed to try to populate if the source is a private IP address. It only works on public address.
Can you try with ip adress 8.8.8.8 and see if it works ?
Moreover, the filter create a geoip object with latitude, longitude, etc... so all you need in your filter conf is
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
}
and you don't need the mutate filter
I resolved all my problems with the new versions 2.x
thanx so much
Go back the problems with the 2.2.1 versión 
Anyone? Please help me!!
Hi, I would like to create GeoIP visualisation in my Kibana. However, I can't even seem to find the geo_point feld. What should I do? Im a noob here so pardon me if there are anyt im lacking of and I'm trying my hardest to get it done for my project. I am using Windows 2012 R2 on VMware workstation to do this.
Elasticsearch: https://download.elastic.co/elastic...p/elasticsearch/2.3.1/elasticsearch-2.3.1.zip
Logstash: https://download.elastic.co/logstash/logstash/logstash-2.3.1.zip
Kibana: https://download.elastic.co/kibana/kibana/kibana-4.5.0-windows.zip
Shippers:
Filebeat: https://download.elastic.co/beats/filebeat/filebeat-1.2.1-windows.zip
Packetbeat: https://download.elastic.co/beats/packetbeat/packetbeat-1.2.1-windows.zip
Topbeat: https://download.elastic.co/beats/topbeat/topbeat-1.2.1-windows.zip
Winlogbeat: https://download.elastic.co/beats/winlogbeat/winlogbeat-1.2.1-windows.zip
This is a very old thread. Please create a new one for you question.
2 Likes
i've the same issue, i suppose that is for the mapping index
If i have understand your problem is just the map doesn't appear in kibana ?
You must to go in your map visualisation, select the "WMS compliant map server", and edit the options like this :
- WMS URL : http://ows-tile.terrestris.de/osm-basemap/service?
- WMS Layers: OSM-WMS
- WMS Version: 1.1.1
- WMS format: image/jpeg
To have good colors you must unselect "Desaturate map tiles"
1 Like
Great Clement so many thanks, now i see the map but my problems are in the geoip filter.
Can you show me your filter configuration please ?
1 Like
Sure
filter {
if [type] == "apache_access" {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
}
}
}
filter {
if [type] == "apache_access_ssl" {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
}
}
}
filter {
if [type] == "apache_error" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
add_field => [ "received_from", "%{host}" ]
}
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
}
}
filter {
if [type] == "apache_error_ssl" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
add_field => [ "received_from", "%{host}" ]
}
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
}
}
Try with this configuration:
filter {
if "apache_access" in [type] {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
merge => { "[tags]" => "[fields][tags]" }
remove_field => "[fields][tags]"
}
}
}
filter {
if "apache_access_ssl" in [type] {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
merge => { "[tags]" => "[fields][tags]" }
remove_field => "[fields][tags]"
}
}
}
filter {
if "apache_error" in [type] {
grok {
pattern => "%{COMBINEDAPACHELOG}"
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][lati" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
merge => { "[tags]" => "[fields][tags]" }
remove_field => "[fields][tags]"
}
}
}
filter {
if "apache_error_ssl" in [type] {
grok {
pattern => "%{COMBINEDAPACHELOG}"
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/usr/share/GeoIP/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{eoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geip][coordinates]", "float"]
merge => { "[tags]" => "[fields][tags]" }
remove_field => "[fields][tags]"
}
}
}
I have an another problem, i can see the field, geoip.location however I see no map when i apply the visualisation.
Sample Log file
84.141.88.166 - - [29/Aug/2016:10:56:47 +0200] store.abc.de "GET /searchAsYouType.action?term=afs35 HTTP/1.1" 200 176 "V8P4zwoyM@sAACuuVy0AAAII" "https://store.abc.de/b2bOrderHistoryTab.action" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0" 391575 1B047AA8C7BD127FB6103AE2F7C462B8
Further more all the field gets values clearly but I could not get the MAP!
My Logstash ConfiguratioN:
# Analyze geo location if [client_ip] { geoip { source => "client_ip" target => "client_geoip" #database => "/jsm/logstash/GeoLiteCity.dat" #add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] #add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] } if ("_grokparsefailure" in [tags]) { mutate { add_tag => [ "geoip_parsefailure" ] remove_tag => [ "_grokparsefailure" ] } } }
I have posted some founding here: https://autofei.wordpress.com/2017/01/11/visualize-geo-location-of-log-using-elasticsearch-logstash-kibana/
But I don't quite get is why the Kibana hard code the "Geohash -> geoip.location" in the UI. Is there a way to change it based on your template? Thanks!
Hi there
would you kindly do the same config for bind dns logging as i see you did it here for apahe