Getting geoip to work

Elastic Stack Logstash
1

Hello, all.

I want to geoip e.g. apache access log. My logstash-forwarder config looks like this:

{
"network": {
"servers": [ "192.168.0.213:5044" ],
"ssl key": "/etc/ssl/private/logstash-forwarder.key",
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 150
},
"files": [
{
"paths": [ "/var/log/messages" ],
"fields": { "type": "syslog" }
},
{
"paths": [ "/var/log/secure" ],
"fields": { "type": "syslog" }
},
{
"paths": [ "/var/log/httpd/access_log" ],
"fields": { "type": "apache-access" }
}
]
}

I created a logstash conf file (12-apache.conf) that looks like this:

input {
file {
type => "apache-access"
path => "/var/log/httpd/*"
start_position => "beginning"
}
}

filter {
grok {
# Here you find more on the default patterns logstash ships with:
# https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
locale => "en"
}

geoip {
source => "line"
database => "/etc/logstash/GeoLiteCity.dat"
target => "geoip"
add_tag => "geoip"
}
}

output {
elasticsearch { hosts => ["192.168.0.213:9200"] }
stdout { codec => rubydebug }
}

Note that geoip source = [line], because I find that the ip address of the host accessing the apache server is contained in it, as in the following logstash capture:

tline 192.168.0.134 - - [21/Jan/2016:09:08:35 -0500] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"

None of the other fields presented in logstash have anything to do with geoip. It seems to me that I'd have to create a regex to pull the ip address out of the line field, but I simply don't have the skill.

Can anyone help me get geoip working?

With thanks,

Diggy

2

What line field? I don't see anything in your config that would create such a field. Please show us an example event from the stdout { codec => rubydebug } output.

The grok pattern you're using should extract a field with the client IP address (the first column in an Apache combined log file) and that's the field you'll want to pass to the geoip filter.

3

Thanks for your response!

I'm very new to ELK and, so, have probably made some configuration mistakes. I've posted what I see when I do a Kibana serach for http (apache) access on a test CentOS 6 server here: http://snag.gy/CSwKJ.jpg. Note the field "line". I'm not even sure how or why it's there, but it does contain the ip address of the device that access the httpd server.

Your continued help is appreciated.

4

Oh, okay. I didn't remember that logstash-forwarder (which is deprecated—you should switch to Filebeat) placed each line in the line field. That's okay, but it means you need to tell the grok filter to parse that field instead of the message field that you don't have. When that's done you can point the geoip filter to the field containing the IP address.

5

Magnus,

Sure enough, changing the grok filter to parse "line" (instead of "message") now presents a field called "clientip", just as I've seen in other posts about geoip on the Net. Stupid me for not seeing that myself, but ... . I then changed the geoip "source" to be "clientip" and, voila, geoip now works. Thank you so much!!!

I will consider filebeats, but wanted to make my initial setup work first.

Diggy