FileBeat not harvesting logs

Elastic Stack Beats
1

Hi,

I am using filebeat and in particular the apache module to send the logs to ElasticSearch and it doesn't appear that the logs are being sent to ElasticSearch and are not appearing in Kibana.

I have followed this page : https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-apache2.html

And run the following command : ./filebeat setup -e and the apache templates appear in kibana.

2017-11-29T10:45:54+10:00 DBG Disable stderr logging
2017-11-29T10:45:54+10:00 INFO Home path: [/u01/filebeat/current] Config path: [/u01/filebeat/current] Data path: [/u01/filebeat/current/data] Logs path: [/u01/filebeat/current/logs]
2017-11-29T10:45:54+10:00 DBG Beat metadata path: /u01/filebeat/current/data/meta.json
2017-11-29T10:45:54+10:00 INFO Beat UUID: 13f79a8f-e275-4fab-ba86-a5045c24363f
2017-11-29T10:45:54+10:00 INFO Setup Beat: filebeat; Version: 6.0.0
2017-11-29T10:45:54+10:00 DBG Initializing output plugins
2017-11-29T10:45:54+10:00 DBG Processors:
2017-11-29T10:45:54+10:00 INFO Elasticsearch url: http://cidclx062.qdot.qld.gov.au:9200
2017-11-29T10:45:54+10:00 INFO Metrics logging every 30s
2017-11-29T10:45:54+10:00 DBG start pipeline event consumer
2017-11-29T10:45:54+10:00 INFO Beat name: cidclx047
2017-11-29T10:45:54+10:00 INFO filebeat start running.
2017-11-29T10:45:54+10:00 INFO Registry file set to: /u01/filebeat/current/data/registry
2017-11-29T10:45:54+10:00 INFO Loading registrar data from /u01/filebeat/current/data/registry
2017-11-29T10:45:54+10:00 INFO States Loaded from registrar: 0
2017-11-29T10:45:54+10:00 INFO Loading Prospectors: 1
2017-11-29T10:45:54+10:00 DBG Checking module configs from: /u01/filebeat/current/modules.d/.yml
2017-11-29T10:45:54+10:00 DBG Load config from file: /u01/filebeat/current/modules.d/apache2.yml
2017-11-29T10:45:54+10:00 DBG Number of module configs found: 1
2017-11-29T10:45:54+10:00 DBG Processors:
2017-11-29T10:45:54+10:00 DBG recursive glob disabled
2017-11-29T10:45:54+10:00 DBG exclude_files: [(?-s:.)gz(?-m:$)]
2017-11-29T10:45:54+10:00 DBG Prospector with previous states loaded: 0
2017-11-29T10:45:54+10:00 DBG File Configs: [/u01/apache/logs/ /u01/apache/logs/]
2017-11-29T10:45:54+10:00 DBG Processors:
2017-11-29T10:45:54+10:00 DBG recursive glob disabled
2017-11-29T10:45:54+10:00 DBG exclude_files: [(?-s:.)gz(?-m:$)]
2017-11-29T10:45:54+10:00 DBG Prospector with previous states loaded: 0
2017-11-29T10:45:54+10:00 DBG File Configs: [/var/log/apache2/error.log /var/log/apache2/error.log*]
2017-11-29T10:45:54+10:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 0
2017-11-29T10:45:54+10:00 INFO Starting Registrar
2017-11-29T10:45:54+10:00 INFO Config reloader started
2017-11-29T10:45:54+10:00 DBG Scan for new config files
2017-11-29T10:45:54+10:00 DBG Load config from file: /u01/filebeat/current/modules.d/apache2.yml
2017-11-29T10:45:54+10:00 DBG Number of module configs found: 1
2017-11-29T10:45:54+10:00 DBG Remove module from stoplist: 20118039972280951
2017-11-29T10:45:54+10:00 DBG Add module to startlist: 20118039972280951
2017-11-29T10:45:54+10:00 DBG Processors:
2017-11-29T10:45:54+10:00 DBG recursive glob disabled
2017-11-29T10:45:54+10:00 DBG exclude_files: [(?-s:.)gz(?-m:$)]
2017-11-29T10:45:54+10:00 DBG Prospector with previous states loaded: 0
2017-11-29T10:45:54+10:00 DBG File Configs: [/u01/apache/logs/* /u01/apache/logs/]
2017-11-29T10:45:54+10:00 DBG Processors:
2017-11-29T10:45:54+10:00 DBG recursive glob disabled
2017-11-29T10:45:54+10:00 DBG exclude_files: [(?-s:.)gz(?-m:$)]
2017-11-29T10:45:54+10:00 DBG Prospector with previous states loaded: 0
2017-11-29T10:45:54+10:00 DBG File Configs: [/var/log/apache2/error.log /var/log/apache2/error.log*]
2017-11-29T10:45:54+10:00 INFO Starting 1 runners ...
2017-11-29T10:45:54+10:00 INFO Elasticsearch url: http://cidclx062.qdot.qld.gov.au:9200
2017-11-29T10:45:54+10:00 DBG ES Ping(url=http://cidclx062.qdot.qld.gov.au:9200)
2017-11-29T10:45:54+10:00 DBG Ping status code: 200
2017-11-29T10:45:54+10:00 INFO Connected to Elasticsearch version 5.6.4
2017-11-29T10:45:54+10:00 DBG Required processors: [{user_agent ingest-user-agent} {geoip ingest-geoip}]
2017-11-29T10:45:54+10:00 DBG GET http://cidclx062.qdot.qld.gov.au:9200/_nodes/ingest
2017-11-29T10:45:54+10:00 DBG GET http://cidclx062.qdot.qld.gov.au:9200/_ingest/pipeline/filebeat-6.0.0-apache2-access-default
2017-11-29T10:45:54+10:00 DBG Pipeline filebeat-6.0.0-apache2-access-default already loaded
2017-11-29T10:45:54+10:00 DBG Required processors: []
2017-11-29T10:45:54+10:00 DBG GET http://cidclx062.qdot.qld.gov.au:9200/_ingest/pipeline/filebeat-6.0.0-apache2-error-pipeline
2017-11-29T10:45:54+10:00 DBG Pipeline filebeat-6.0.0-apache2-error-pipeline already loaded
2017-11-29T10:45:54+10:00 INFO Starting prospector of type: log; id: 241050076287732636
2017-11-29T10:45:54+10:00 INFO Starting prospector of t

2017-12-01T10:04:24+10:00 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4194304 beat.memstats.memory_alloc=1511672 beat.memstats.memory_total=141713328 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=1 libbeat.pipeline.clients=4 libbeat.pipeline.events.active=0 registrar.states.current=0

I can't work out what the problem is. Any help is appreciated.

Thanks

ype: log; id: 12452675827382683975

2

Hi,
I discovered this problem but now have a new problem,

It was the directory was configured wrong.

Now I am getting this error.

2017-12-01T10:59:52+10:00 DBG Skipping directory: /u01/apache2/logs/dev.corp.qdot.qld.gov.au-http
2017-12-01T10:59:52+10:00 DBG Skipping directory: /u01/apache2/logs/dev.corp.qdot.qld.gov.au-https
2017-12-01T10:59:52+10:00 DBG Skipping directory: /u01/apache2/logs/dev.cssr.qdot.qld.gov.au-http
2017-12-01T10:59:52+10:00 DBG Skipping directory: /u01/apache2/logs/dev.cssr.qdot.qld.gov.au-https
2017-12-01T10:59:52+10:00 DBG Skipping directory: /u01/apache2/logs/dev.imd.qdot.qld.gov.au-http

Any ideas ?

3

Could you share your config and version that you are using? The above means these files are skipped, as they are directories and not files, so can't be tailed.

4

Hi,

We are running version 6.0 of filebeat.

filebeat.yml

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/*.log

apache2.yml

  • module: apache2

    Access logs

    access:
    enabled: true

    Set custom paths for the log files. If left empty,

    Filebeat will choose the paths depending on your OS.

    var.paths: ["/u01/apache2/logs/*"]

    Error logs

    error:
    enabled: true

    Set custom paths for the log files. If left empty,

    Filebeat will choose the paths depending on your OS.

    #var.paths:

5

Hi,

I have resolved the issue, I needed to change /u01/apache2/logs/* to /u01/apache2/logs/**/*.

Thanks.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.