Hello everyone
I'm new to the ELK world and need some help with a basic Apache HTTP filebeat -> logstash -> Elasticsearch <- Kibana filter/pipeline for learning purposes.
Version:
8.3.3
Problem:
A dummy apache access output that I want visualize in Kibana incl. geoip.
57.85.164.98 - - [20/Sep/2017:15:31:04 +0200] "GET /favicon.ico HTTP/1.1" 200 7581 "https://codingexplained.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
The '@timestamp' field does not match the timestamp field inside the apache access log. However, I can't filter the 'timestamp' (the Apache one) in Kibana nor does the default Dashboard work.
I assume I have to map the 'timestamp' somehow with the '@timestamp' ?
I read the manual x times and it won't work. Also I noticed that the example config here is outdated regarding ECS ?
Here is my pipeline/filter:
input {
beats {
port => 5044
host => "0.0.0.0"
}
}
filter {
if [event][dataset] != "apache.access" {
drop { }
}
grok {
match => { "[event][original]" => '%{HTTPD_COMBINEDLOG}' }
}
mutate {
remove_field => ["event.orginal", "log", "input", "service", "host", "ecs", "@version"]
}
grok {
"match" => {
"[source][address]" => "^(%{IP:[source][ip]}|%{HOSTNAME:[source][domain]})"
}
}
date {
match => [ "[apache][access][time]", "dd/MMM/yyyy:H:m:s Z" ]
}
useragent {
source => "[user_agent][original]"
target => "[user_agent]"
}
geoip {
source => "[source][ip]"
target => "[source][geo]"
}
}
output {
elasticsearch {
hosts => "localhost"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY-MM-dd}"
}
}