Magnus, I changed the sintax of Logstash filter plugin and got the following result:
filter {
if [type] == "apache-error" {
grok {
match => ["message", "\[%{WORD:dayname} %{WORD:month} %{DATA:day} %{DATA:hour}:%{DATA:minute}:%{DATA:second} %{YEAR:year}\] \[%{NOTSPACE:loglevel}\] \[pid \d+:tid \d+\] \[client %{IP:clientip}:\d+\] %{GREEDYDATA:message}"]
overwrite => [ "message" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
add_field => {
"time_stamp" => "%{day}/%{month}/%{year}:%{hour}:%{minute}:%{second}"
"clientip" => "%{clientip}"
}
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "apache-geoip" ]
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
date {
match => ["time_stamp", "dd/MMM/YYYY:HH:mm:ss"]
remove_field => [ "time_stamp","day","dayname","month","hour","minute","second","year"]
}
}
}
The match sintax allowed extract the client IP, however the clientip field show the doble IP. See the Kibana output.
@timestamp November 2nd 2016, 21:50:52.762
t@version 1
t_id AVgnsFg1HAYfP2pKRUOB
t_index filebeat-2016.11.03
#_score 1
t_type apache-error
tbeat.hostname reverse-apache
tbeat.name reverse-apache
?clientip 66.249.88.60, 66.249.88.60
#count 1
?day 02
?dayname Wed
?fields -
?geoip.area_code 650
?geoip.city_name Mountain View
?geoip.continent_code NA
?geoip.coordinates -122.0574, 37.41919999999999
?geoip.country_code2 US
?geoip.country_code3 USA
?geoip.country_name United States
?geoip.dma_code 807
?geoip.ip 66.249.88.60
?geoip.latitude 37.41919999999999
geoip.location -122.0574, 37.41919999999999
?geoip.longitude -122.0574
?geoip.postal_code 94043
?geoip.real_region_name California
?geoip.region_name CA
?geoip.timezone America/Los_Angeles
thost reverse-apache
?hour 21
tinput_type log
?loglevel :error
tmessage [client 66.249.88.60] ModSecurity: Access denied with code 401 (phase 4). Pattern match "^5\\\\d{2}$" at RESPONSE_STATUS. [file "/usr/local/modsecurity/crs/activated_rules/modsecurity_crs_50_outbound.conf"] [line "53"] [id "970901"] [rev "2"] [msg "The application is not available"] [data "Matched Data: 502 found within RESPONSE_STATUS: 502"] [severity "ERROR"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "WASCTC/WASC-13"] [tag "OWASP_TOP_10/A6"] [tag "PCI/6.5.6"] [hostname "www.xxx.com"] [uri "/sigh/consult/"] [unique_id "WBqJrQpC--kAAA0JcCsAAAGD"], referer: http://www.xxx.com/
?minute 50
?month Nov
#offset 198,039,484
?second 49.856841
tsource /var/log/apache2/xxx.com_error.log
ttags beats_input_codec_plain_applied, apache-geoip, _dateparsefailure
?time_stamp 02/Nov/2016:21:50:49.856841
ttype apache-error
?year 2016
Do you have any suggestion to fix duplicity?
Thanks.