Kibana/ELK geo problem

[xon (xt)]

I want to add geoloc info on my apache logs.

I am new in this and I have the following setup (apache, redis, logstash, elastic, kibana):

filebeat:

filebeat.prospectors:

• input_type: log
  paths:
  • /var/log/apache2/*.log
    output.redis:
    hosts: ["192.168.0.21"]
    port: "6379"
    password: "test123"
    key: "filebeat"
    db: 0
    timeout: 5

redis:

requirepass test123
bind 192.168.0.21

logstash:

input {
redis {
host => "192.168.0.21"
password => "test123"
type => "redis-input"
data_type => "list"
key => "filebeat"
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}

output {
elasticsearch {
hosts => "192.168.0.25:9200"
manage_template => false
index => "filebeat-%{+YYYY.MM.dd}"
document_type => "apache"
}
}

The above is fine however it doesnt have the geo stuff!

I was following a guide to add geoloc details on my apache logs however I was getting an error about the geo-point.

The changes that I did in the logstash.conf were:

filter {
geoip {
source => "clientip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}"]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"]
}

mutate {
  convert => [ "[geoip][coordinates]", "float"]
}

}

From that point and on everything broke, nothing is getting indexed anymore and even when I delete the current indices the process doesnt start. I can see Apache talking to Redis however Logstash is not talking to Redis and obviously I cant see anything in Kibana. I went over this --> Geo_Point Field Exists, But Kibana Can't Find It but I am not sure what I have to do ?

I am using version 5...

Any help/thoughts?

[jordansissel]

It sounds like you have two problems now ... I'll talk about both.

I was following a guide
The changes that I did in the logstash.conf were:

The guide you're following (I don't know which one), is out of date.

The geoip plugin provides a location field by default and is a geojson-structured field.

What version of Logstash is this? Are you certain you have a clientip field? Can you show a sample document from your filebeat indexes?

---

Second problem:

From that point and on everything broke, nothing is getting indexed anymore and even when I delete the current indices the process doesnt start

What do you mean 'process doesn't start' ? What program? What indicates that it is not starting?

[xon (xt)]

Hi @jordansissel thanks for your reply.

So the guide I was following is the one from digital ocean (can be found here --> https://www.digitalocean.com/community/tutorials/how-to-map-user-location-with-geoip-and-elk-elasticsearch-logstash-and-kibana )

Regarding the geo stuff I can see the fields with (?) next in Kibana meaning that these arent indexed (?). Either way the error message that I was getting back was: geopoint was not there. I went over this ( Geo_point - no compatible types ) and also this ( Geo_Point Field Exists, But Kibana Can't Find It ) but I need a bit more light to understand what I have to do... I can definitely see 'clientip' field since I have already configured some Kibana dashboards with that...

I am using logstash version 5.0.1

For the second problem nevermind I managed to delete the indices and do everything from scratch so it works now. Still I dont have the geolocation though Is there another guide that I can follow?

Thanks

[jordansissel]

The question mark, in my experience, means that Kibana doesn't have knowledge of that field. You can usually go to the Settings area of Kibana for your index pattern and click the refresh button to resolve that problem.

[xon (xt)]

Yep I have done this. Didnt fix it