Filebeat not reading apache logs?


(Tomiford) #1

Hi

I am new to filebeat and have set up the syslog files of our servers to be sent to logstash by filebeat. I am now trying to get apache logs into logstash but it does not seem to be reading the log.

When starting up filebeat this is the debug info:

sudo service filebeat restart

  • Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat 2016/01/18 06:45:06.457697 beat.go:97: DBG Initializing output plugins
    2016/01/18 06:45:06.457773 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
    2016/01/18 06:45:06.520679 outputs.go:111: INFO Activated logstash as output plugin.
    2016/01/18 06:45:06.521348 publish.go:198: DBG create output worker: 0x0, 0x0
    2016/01/18 06:45:06.523384 publish.go:235: DBG No output is defined to store the topology. The server fields might not be filled.
    2016/01/18 06:45:06.523684 publish.go:249: INFO Publisher name: myserver
    2016/01/18 06:45:06.524167 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=200)
    2016/01/18 06:45:06.524334 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1

When I look at the filebeat log it looks like this:

2016-01-18T15:45:06+09:00 DBG scan path /var/log/apache/.log
2016-01-18T15:45:06+09:00 DBG No pending prospectors. Finishing setup
2016-01-18T15:45:06+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-18T15:45:06+09:00 INFO Starting Registrar
2016-01-18T15:45:06+09:00 INFO Start sending events to output
2016-01-18T15:45:09+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T15:45:14+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T15:45:16+09:00 DBG Start next scan
2016-01-18T15:45:16+09:00 DBG scan path /var/log/apache/
.log
2016-01-18T15:45:21+09:00 DBG Flushing spooler because of timemout. Events flushed: 0

The access logs have definitely been written to during this period.

The apache prospector looks like this:

  • paths:

    • /var/log/apache/*.log
      document_type: apache

    encoding: utf-8

    Ignore files which are older then 24 hours

    ignore_older: 24h

    Additional fields which can be freely defined

    fields:
    type: apache

Any feedback would be appreciated.

Thanks


(Magnus B├Ąck) #2

I suspect the user that runs the Filebeat process doesn't have sufficient permissions to access the contents of /var/log/apache.


(Tomiford) #3

Thanks for the reply. Looking at top the Filebeat service is running as root so it should have access.


(ruflin) #4

Can you share a few more lines of the log file on starting up filebeat until the log file starts to repeat?


(Tomiford) #5

Thanks for the reply. Is this what you were after?

2016-01-18T18:51:03+09:00 DBG Disable stderr logging
2016-01-18T18:51:03+09:00 DBG Initializing output plugins
2016-01-18T18:51:03+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-01-18T18:51:03+09:00 INFO Activated logstash as output plugin.
2016-01-18T18:51:03+09:00 DBG create output worker: 0x0, 0x0
2016-01-18T18:51:03+09:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-01-18T18:51:03+09:00 INFO Publisher name: asia-web-2
2016-01-18T18:51:03+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)
2016-01-18T18:51:03+09:00 INFO Init Beat: filebeat; Version: 1.0.1
2016-01-18T18:51:03+09:00 INFO filebeat sucessfully setup. Start running.
2016-01-18T18:51:03+09:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-01-18T18:51:03+09:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-01-18T18:51:03+09:00 DBG Set idleTimeoutDuration to 5s
2016-01-18T18:51:03+09:00 DBG File Configs: [/var/log/apache/.log]
2016-01-18T18:51:03+09:00 DBG Set ignore_older duration to 24h0m0s
2016-01-18T18:51:03+09:00 DBG Set scan_frequency duration to 10s
2016-01-18T18:51:03+09:00 DBG Set backoff duration to 1s
2016-01-18T18:51:03+09:00 DBG Set max_backoff duration to 10s
2016-01-18T18:51:03+09:00 DBG Set partial_line_waiting duration to 5s
2016-01-18T18:51:03+09:00 DBG Waiting for 1 prospectors to initialise
2016-01-18T18:51:03+09:00 DBG Harvest path: /var/log/apache/
.log
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/
.log
2016-01-18T18:51:03+09:00 DBG No pending prospectors. Finishing setup
2016-01-18T18:51:03+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-18T18:51:03+09:00 INFO Starting Registrar
2016-01-18T18:51:03+09:00 INFO Start sending events to output
2016-01-18T18:51:03+09:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-01-18T18:51:05+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T18:51:13+09:00 DBG Start next scan
2016-01-18T18:51:13+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T18:51:13+09:00 DBG scan path /var/log/apache/*.log

I have disabled the syslog that I was also sending to logstash server while I try to get apache log working.


(ruflin) #6

It seems like the harvester doesn't even see the file, so @magnusbeack could be right.

As far as I understand you comment you are using filebeat also for other files that work as expected?


(Tomiford) #7

Yes, if I change the prospector to syslog filebeat reads the file, this is the start up output when syslog is read:

2016-01-19T08:05:16+09:00 DBG Disable stderr logging
2016-01-19T08:05:16+09:00 DBG Initializing output plugins
2016-01-19T08:05:16+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-01-19T08:05:16+09:00 INFO Activated logstash as output plugin.
2016-01-19T08:05:16+09:00 DBG create output worker: 0x0, 0x0
2016-01-19T08:05:16+09:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-01-19T08:05:16+09:00 INFO Publisher name: asia-web-2
2016-01-19T08:05:16+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)
2016-01-19T08:05:16+09:00 INFO Init Beat: filebeat; Version: 1.0.1
2016-01-19T08:05:16+09:00 INFO filebeat sucessfully setup. Start running.
2016-01-19T08:05:16+09:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-01-19T08:05:16+09:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-01-19T08:05:16+09:00 DBG Set idleTimeoutDuration to 5s
2016-01-19T08:05:16+09:00 DBG File Configs: [/var/log/syslog]
2016-01-19T08:05:16+09:00 DBG Set ignore_older duration to 24h0m0s
2016-01-19T08:05:16+09:00 DBG Set scan_frequency duration to 10s
2016-01-19T08:05:16+09:00 DBG Set backoff duration to 1s
2016-01-19T08:05:16+09:00 DBG Set max_backoff duration to 10s
2016-01-19T08:05:16+09:00 DBG Set partial_line_waiting duration to 5s
2016-01-19T08:05:16+09:00 DBG Waiting for 1 prospectors to initialise
2016-01-19T08:05:16+09:00 DBG Harvest path: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Start harvesting unknown file: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Same file as before found. Fetch the state and persist it.
2016-01-19T08:05:16+09:00 DBG Resuming harvester on a previously harvested file: /var/log/syslog
2016-01-19T08:05:16+09:00 INFO Harvester started for file: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG harvest: "/var/log/syslog" position:208785 (offset snapshot:208785)
2016-01-19T08:05:16+09:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-01-19T08:05:16+09:00 DBG Registrar will re-save state for /var/log/syslog
2016-01-19T08:05:16+09:00 DBG No pending prospectors. Finishing setup
2016-01-19T08:05:16+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-19T08:05:16+09:00 INFO Starting Registrar
2016-01-19T08:05:16+09:00 INFO Start sending events to output
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Update existing file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Not harvesting, file didn't change: /var/log/syslog
2016-01-19T08:05:18+09:00 DBG Flushing spooler because of timemout. Events flushed: 474
2016-01-19T08:05:18+09:00 DBG send event
2016-01-19T08:05:18+09:00 DBG Start Preprocessing
2016-01-19T08:05:18+09:00 DBG Publish: { ... }


(Tomiford) #8

I apologize, this is a really basic error and what you get for copying tutorials from Google. There was a typo in the apache log path.

/var/log/apache/

should have been

/var/log/apache2/

Sorry for wasting your time.


(ruflin) #9

Glad you found the solution :slight_smile:


(system) #10