Hello all. I am fairly new to ELK and I am trying to troubleshoot a problem I am facing with geoip.
I followed this guide to install everything: https://www.elastic.co/guide/en/elastic-stack-get-started/7.4/get-started-elastic-stack.html
Then after I understood a little bit better how things are working, I disabled metricbeat and installed filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html
I am able to parse 2 different kinds of logs from Apache using grok.
After that I tried to add the geoip fitler, however it is not working.
I searched and I saw that(after following the guide for filebeat) I have default index pattern called "Filebeat-*", the indices are created using the format "Filebeat-version-YYYY.MM.dd", and the geoip mapping is set to String.
here is my logstash configuration:
input {
beats {
port => 5044
}
}
filter {
if "WARN" in [message] { drop{ } }
if " - - " in [message] {
grok { match => { "message" => "%{IP:client} %{DATA:em} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} %{DATA:http}\" %{NUMBER:response}" } }
geoip { source => "client" }
}else {
grok { match => { "message" => "%{IP:client} \[%{HTTPDATE:timestamp}\] %{WORD:method} %{URIPATHPARAM:request} %{DATA:http} %{NUMBER:response} %{NUMBER:duration}" } }
geoip { source => "client" }
}
if [system][process] {
if [system][process][cmdline] {
grok {
match => {
"[system][process][cmdline]" => "^%{PATH:[system][process][cmdline_path]}"
}
remove_field => "[system][process][cmdline]"
}
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
I tried deleting the default index pattern, deleting the latest index of Filebeat(ending in 10.30) but still I can not seem to be able to solve it.
I am 100% sure that I have failed somewhere along the way, but can you provide me with some information, so future indices can have the correct geoip mapping?
Thank you very much