Need help with my Apache2 filter


#1

In my set-up I have 2 different servers. The first has Filebeat and the second one has Elasticsearch, Kibana and Logstash installed.

I have apache2.yml in /etc/filebeat/modules.d enabled.
In my filebeat.yml in /etc/filebeat, I have the following Filebeat prospectors:

# Apache2
- type: log
  enabled: true
  paths:
    - /swan/log/apache-swan-intern-access.log.*
  tags: ["apache"]
  fields_under_root: true

- type: log
  paths:
    - /swan/log/apache-swan-intern-error.log.*
  tags: ["apache"]
  fields_under_root: true

And the output is Logstash.

On my other server I have the following Conf-File in /etc/logstash/conf.d: 01-apache2-filter.conf

input {
  beats {
    port => 5044
  }
}

filter {

  if [fileset][module] == "apache2" and [fileset][name] == "access" {
    grok {
      match => {
        "message" => [
          "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?",
          "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"-\" %{NUMBER:[apache2][access][response_code]} -"
        ]
      }
    }

    mutate {
      rename => {
        "@timestamp" => "read_timestamp"
      }
    }

    date {
      match => [
        "[apache2][access][time]",
        "dd/MMM/YYYY:H:m:s Z"
      ]
      target => "@timestamp"
    }

    useragent {
      source => "[apache2][access][agent]"
      target => "[apache2][access][user_agent]"
    }

    geoip {
      source => "[apache2][access][remote_ip]"
      target => "[apache2][access][geoip]"
    }

  }
}

output {
  elasticsearch { hosts => ["10.6.10.45:9200"] }
  stdout { codec => rubydebug }
}

The logfile of Logstash says the following: /var/log/logstash/logstash-plain.log

[2018-05-23T10:45:21,249][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-23T10:45:21,355][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-23T10:45:22,795][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-23T10:45:23,190][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-05-23T10:45:23,863][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xd027ef4@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-23T10:45:23,873][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xd027ef4@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 dead>"}
[2018-05-23T10:45:23,884][INFO ][logstash.agent           ] Pipelines running {:count=>0, :pipelines=>[]}
[2018-05-23T10:45:42,763][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-05-23T10:45:42,775][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-05-23T10:45:43,457][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-23T10:45:43,655][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-23T10:45:45,313][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-23T10:45:45,694][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-05-23T10:45:46,323][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x285a8644@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-23T10:45:46,329][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x285a8644@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 dead>"}
[2018-05-23T10:45:46,336][INFO ][logstash.agent           ] Pipelines running {:count=>0, :pipelines=>[]}
[2018-05-23T10:46:04,883][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-05-23T10:46:04,890][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-05-23T10:46:05,444][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-23T10:46:05,603][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-23T10:46:06,925][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-23T10:46:07,322][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-05-23T10:46:07,501][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4107557e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-23T10:46:07,510][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-05-23T10:46:07,964][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x4107557e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}

I can't see any errors here but when I look in my Dashboard, it seems there is no data. Any idea?

Filebeat - [Filebeat Apache2] Access and error logs


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.