Visualizing apache2 logs in Kibana

[dhiraj_khanna]

I am trying to visualize apache2 logs in the dashboards which ship with Kibana (6.6) on Windows. Here's what I have done:

1. Installed the ingest-user-agent and ingest-geoip plugins in Elasticsearch.

2. Enabled the apache2 module in filebeat.

This is what my filebeat.yml looks like:

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  enabled: true

  paths: 'C:\Users\bigdataadmin\Downloads\ApacheLogs\*'
 

#============================= Filebeat modules ===============================

filebeat.config.modules:
 
  path: 'C:\Users\bigdataadmin\Downloads\Filebeat\modules.d\*.yml'

  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
#============================== Kibana =====================================

setup.kibana:

  
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

3. This is what my apache2.yml looks like:

- module: apache2
  # Access logs
  access:
    enabled: true
    var.paths: ['C:\Users\bigdataadmin\Downloads\ApacheLogs\*']

  # Error logs
  error:
    enabled: true
    var.paths: ['C:\Users\bigdataadmin\Downloads\ApacheLogs\*']

4. When I run filebeat after the setup -e, I don't see any harvesting. Only monitoring :

 2019-02-06T21:13:24.850+0530    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://localhost:920
0)) established
2019-02-06T21:13:25.236+0530    INFO    fileset/pipelines.go:80 Elasticsearch pipeline with ID 'filebeat-6.6.0-apache2-a
ccess-default' loaded
2019-02-06T21:13:25.594+0530    INFO    fileset/pipelines.go:80 Elasticsearch pipeline with ID 'filebeat-6.6.0-apache2-e
rror-pipeline' loaded
2019-02-06T21:13:25.595+0530    INFO    input/input.go:114      Starting input of type: log; ID: 4549795131320701132
2019-02-06T21:13:25.598+0530    INFO    input/input.go:114      Starting input of type: log; ID: 7729261239287606488
2019-02-06T21:13:25.599+0530    INFO    cfgfile/reload.go:205   Loading of config files completed.
2019-02-06T21:13:25.600+0530    INFO    log/harvester.go:255    Harvester started for file: C:\Users\bigdataadmin\Downlo
ads\ApacheLogs\apache-daily-access.log
2019-02-06T21:13:25.601+0530    INFO    log/harvester.go:255    Harvester started for file: C:\Users\bigdataadmin\Downlo
ads\ApacheLogs\apache-daily-access.log
2019-02-06T21:13:54.948+0530    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monito
ring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1343,"time":{"ms":1343}},"total":{"ticks":5452,"time":{"ms":5452},"
value":5452},"user":{"ticks":4109,"time":{"ms":4109}}},"handles":{"open":487},"info":{"ephemeral_id":"d9f2a5f1-8807-4d61
-a8cf-b4fa2a83a23b","uptime":{"ms":33116}},"memstats":{"gc_next":52843616,"memory_alloc":26419712,"memory_total":2118096
88,"rss":82108416}},"filebeat":{"events":{"active":4119,"added":20213,"done":16094},"harvester":{"open_files":3,"running
":3,"started":3}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":16086,"active":50
,"batches":323,"total":16136},"read":{"bytes":212189},"type":"elasticsearch","write":{"bytes":15609805}},"pipeline":{"cl
ients":5,"events":{"active":4119,"filtered":8,"published":20202,"retry":50,"total":20213},"queue":{"acked":16086}}},"reg
istrar":{"states":{"current":1,"update":16094},"writes":{"success":330,"total":330}},"system":{"cpu":{"cores":64}}}}}
2019-02-06T21:14:24.763+0530    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monito
ring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2375,"time":{"ms":1032}},"total":{"ticks":10718,"time":{"ms":5266},
"value":10718},"user":{"ticks":8343,"time":{"ms":4234}}},"handles":{"open":552},"info":{"ephemeral_id":"d9f2a5f1-8807-4d
61-a8cf-b4fa2a83a23b","uptime":{"ms":63117}},"memstats":{"gc_next":53496192,"memory_alloc":38162408,"memory_total":40130
0488,"rss":2789376}},"filebeat":{"events":{"active":50,"added":17382,"done":17332},"harvester":{"open_files":3,"running"
:3}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":17382,"batches":348,"total":17382},"read":
{"bytes":233764},"write":{"bytes":17228070}},"pipeline":{"clients":5,"events":{"active":4119,"published":17382,"total":1
7382},"queue":{"acked":17382}}},"registrar":{"states":{"current":1,"update":17382},"writes":{"success":347,"total":348}}
}}}
2019-02-06T21:14:54.764+0530    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monito
ring": {"metrics": {"beat":{"cpu":{"system":{"ticks":4734,"time":{"ms":2359}},"total":{"ticks":17343,"time":{"ms":6625},
"value":17343},"user":{"ticks":12609,"time":{"ms":4266}}},"handles":{"open":564},"info":{"ephemeral_id":"d9f2a5f1-8807-4
d61-a8cf-b4fa2a83a23b","uptime":{"ms":93117}},"memstats":{"gc_next":45311472,"memory_alloc":33090824,"memory_total":5659
98616,"rss":6270976}},"filebeat":{"events":{"active":-1934,"added":14250,"done":16184},"harvester":{"open_files":3,"runn
ing":3}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":16184,"batches":324,"total":16184},"re
ad":{"bytes":222363},"write":{"bytes":16482789}},"pipeline":{"clients":5,"events":{"active":2185,"published":14253,"tota
l":14250},"queue":{"acked":16184}}},"registrar":{"states":{"current":1,"update":16184},"writes":{"success":324,"total":3
24}}}}}

I do see a filebeat-* index pattern in Kibana, but nothing pops up in the builtin dashboards of Kibana for apache2 logs.

Can't figure out what's going on.

[dhiraj_khanna]

This is a screen grab from Kibana: