Apache Log and _geoip_lookup_failure error

Elastic Stack Logstash
1

I am using Apache behind HAProxy. The sample log file is as follows

194.1.143.146, 10.80.4.30 - - [13/Dec/2017:15:08:12 +0530] "GET /Portal/themes/js/language/showjs.js HTTP/1.1" 200 7106 "https://myserver.com/search/?searchElement=soccer&language=en" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0"

The first IP is Client IP and second IP is HAproxy IP. I have noticed that because of presence of comma charater after ClientIP,
the error _geoip_lookup_failure is appearing. The X-Forwarded-For option in HAProxy provides comma separated IP addresses to Apache Log file. I want to filter this (,) character before index creation.
How can I filter this Comma charter so that geoip can get value after index creation?

2

What does your current Logstash configuration look like?

3

Here is content of "/usr/share/logstash/logstash.conf
" file

input {
beats {
port => 5443
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
add_tag => "geo_point"
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout { codec => rubydebug }
}

and apache-filter.conf is as follows

filter {
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
add_tag => "geo_point"
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}

Please note when I manually remove Comma(,) charater between two IPs, it works properly.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.