In my set-up I have 2 different servers. The first has Filebeat and the second one has Elasticsearch, Kibana and Logstash installed.
I have apache2.yml in /etc/filebeat/modules.d enabled.
In my filebeat.yml in /etc/filebeat, I have the following Filebeat prospectors:
# Apache2
- type: log
enabled: true
paths:
- /swan/log/apache-swan-intern-access.log.*
tags: ["apache"]
fields_under_root: true
- type: log
paths:
- /swan/log/apache-swan-intern-error.log.*
tags: ["apache"]
fields_under_root: true
And the output is Logstash.
On my other server I have the following Conf-File in /etc/logstash/conf.d: 01-apache2-filter.conf
input {
beats {
port => 5044
}
}
filter {
if [fileset][module] == "apache2" and [fileset][name] == "access" {
grok {
match => {
"message" => [
"%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?",
"%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"-\" %{NUMBER:[apache2][access][response_code]} -"
]
}
}
mutate {
rename => {
"@timestamp" => "read_timestamp"
}
}
date {
match => [
"[apache2][access][time]",
"dd/MMM/YYYY:H:m:s Z"
]
target => "@timestamp"
}
useragent {
source => "[apache2][access][agent]"
target => "[apache2][access][user_agent]"
}
geoip {
source => "[apache2][access][remote_ip]"
target => "[apache2][access][geoip]"
}
}
}
output {
elasticsearch { hosts => ["10.6.10.45:9200"] }
stdout { codec => rubydebug }
}
The logfile of Logstash says the following: /var/log/logstash/logstash-plain.log
[2018-05-23T10:45:21,249][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-23T10:45:21,355][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-23T10:45:22,795][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-23T10:45:23,190][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-05-23T10:45:23,863][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xd027ef4@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-23T10:45:23,873][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xd027ef4@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 dead>"}
[2018-05-23T10:45:23,884][INFO ][logstash.agent ] Pipelines running {:count=>0, :pipelines=>[]}
[2018-05-23T10:45:42,763][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-05-23T10:45:42,775][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-05-23T10:45:43,457][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-23T10:45:43,655][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-23T10:45:45,313][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-23T10:45:45,694][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-05-23T10:45:46,323][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x285a8644@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-23T10:45:46,329][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x285a8644@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 dead>"}
[2018-05-23T10:45:46,336][INFO ][logstash.agent ] Pipelines running {:count=>0, :pipelines=>[]}
[2018-05-23T10:46:04,883][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-05-23T10:46:04,890][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-05-23T10:46:05,444][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-23T10:46:05,603][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-23T10:46:06,925][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-23T10:46:07,322][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-05-23T10:46:07,501][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4107557e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-23T10:46:07,510][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-05-23T10:46:07,964][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x4107557e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
I can't see any errors here but when I look in my Dashboard, it seems there is no data. Any idea?