Filebeat not reading apache logs?

tomiford · January 18, 2016, 7:10am

Hi

I am new to filebeat and have set up the syslog files of our servers to be sent to logstash by filebeat. I am now trying to get apache logs into logstash but it does not seem to be reading the log.

When starting up filebeat this is the debug info:

sudo service filebeat restart

Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat 2016/01/18 06:45:06.457697 beat.go:97: DBG Initializing output plugins
2016/01/18 06:45:06.457773 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/01/18 06:45:06.520679 outputs.go:111: INFO Activated logstash as output plugin.
2016/01/18 06:45:06.521348 publish.go:198: DBG create output worker: 0x0, 0x0
2016/01/18 06:45:06.523384 publish.go:235: DBG No output is defined to store the topology. The server fields might not be filled.
2016/01/18 06:45:06.523684 publish.go:249: INFO Publisher name: myserver
2016/01/18 06:45:06.524167 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=200)
2016/01/18 06:45:06.524334 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1

When I look at the filebeat log it looks like this:

2016-01-18T15:45:06+09:00 DBG scan path /var/log/apache/.log
2016-01-18T15:45:06+09:00 DBG No pending prospectors. Finishing setup
2016-01-18T15:45:06+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-18T15:45:06+09:00 INFO Starting Registrar
2016-01-18T15:45:06+09:00 INFO Start sending events to output
2016-01-18T15:45:09+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T15:45:14+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T15:45:16+09:00 DBG Start next scan
2016-01-18T15:45:16+09:00 DBG scan path /var/log/apache/.log
2016-01-18T15:45:21+09:00 DBG Flushing spooler because of timemout. Events flushed: 0

The access logs have definitely been written to during this period.

The apache prospector looks like this:

paths:
- /var/log/apache/*.log
  document_type: apache
encoding: utf-8

Ignore files which are older then 24 hours

ignore_older: 24h

Additional fields which can be freely defined

fields:
type: apache

Any feedback would be appreciated.

Thanks

magnusbaeck · January 18, 2016, 7:31am

I suspect the user that runs the Filebeat process doesn't have sufficient permissions to access the contents of /var/log/apache.

tomiford · January 18, 2016, 7:40am

Thanks for the reply. Looking at top the Filebeat service is running as root so it should have access.

ruflin · January 18, 2016, 8:13am

Can you share a few more lines of the log file on starting up filebeat until the log file starts to repeat?

tomiford · January 18, 2016, 9:54am

Thanks for the reply. Is this what you were after?

2016-01-18T18:51:03+09:00 DBG Disable stderr logging
2016-01-18T18:51:03+09:00 DBG Initializing output plugins
2016-01-18T18:51:03+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-01-18T18:51:03+09:00 INFO Activated logstash as output plugin.
2016-01-18T18:51:03+09:00 DBG create output worker: 0x0, 0x0
2016-01-18T18:51:03+09:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-01-18T18:51:03+09:00 INFO Publisher name: asia-web-2
2016-01-18T18:51:03+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)
2016-01-18T18:51:03+09:00 INFO Init Beat: filebeat; Version: 1.0.1
2016-01-18T18:51:03+09:00 INFO filebeat sucessfully setup. Start running.
2016-01-18T18:51:03+09:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-01-18T18:51:03+09:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-01-18T18:51:03+09:00 DBG Set idleTimeoutDuration to 5s
2016-01-18T18:51:03+09:00 DBG File Configs: [/var/log/apache/.log]
2016-01-18T18:51:03+09:00 DBG Set ignore_older duration to 24h0m0s
2016-01-18T18:51:03+09:00 DBG Set scan_frequency duration to 10s
2016-01-18T18:51:03+09:00 DBG Set backoff duration to 1s
2016-01-18T18:51:03+09:00 DBG Set max_backoff duration to 10s
2016-01-18T18:51:03+09:00 DBG Set partial_line_waiting duration to 5s
2016-01-18T18:51:03+09:00 DBG Waiting for 1 prospectors to initialise
2016-01-18T18:51:03+09:00 DBG Harvest path: /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG No pending prospectors. Finishing setup
2016-01-18T18:51:03+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-18T18:51:03+09:00 INFO Starting Registrar
2016-01-18T18:51:03+09:00 INFO Start sending events to output
2016-01-18T18:51:03+09:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-01-18T18:51:05+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T18:51:13+09:00 DBG Start next scan
2016-01-18T18:51:13+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T18:51:13+09:00 DBG scan path /var/log/apache/*.log

I have disabled the syslog that I was also sending to logstash server while I try to get apache log working.

ruflin · January 18, 2016, 11:59am

It seems like the harvester doesn't even see the file, so @magnusbeack could be right.

As far as I understand you comment you are using filebeat also for other files that work as expected?

tomiford · January 18, 2016, 11:08pm

Yes, if I change the prospector to syslog filebeat reads the file, this is the start up output when syslog is read:

2016-01-19T08:05:16+09:00 DBG Disable stderr logging
2016-01-19T08:05:16+09:00 DBG Initializing output plugins
2016-01-19T08:05:16+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-01-19T08:05:16+09:00 INFO Activated logstash as output plugin.
2016-01-19T08:05:16+09:00 DBG create output worker: 0x0, 0x0
2016-01-19T08:05:16+09:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-01-19T08:05:16+09:00 INFO Publisher name: asia-web-2
2016-01-19T08:05:16+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)
2016-01-19T08:05:16+09:00 INFO Init Beat: filebeat; Version: 1.0.1
2016-01-19T08:05:16+09:00 INFO filebeat sucessfully setup. Start running.
2016-01-19T08:05:16+09:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-01-19T08:05:16+09:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-01-19T08:05:16+09:00 DBG Set idleTimeoutDuration to 5s
2016-01-19T08:05:16+09:00 DBG File Configs: [/var/log/syslog]
2016-01-19T08:05:16+09:00 DBG Set ignore_older duration to 24h0m0s
2016-01-19T08:05:16+09:00 DBG Set scan_frequency duration to 10s
2016-01-19T08:05:16+09:00 DBG Set backoff duration to 1s
2016-01-19T08:05:16+09:00 DBG Set max_backoff duration to 10s
2016-01-19T08:05:16+09:00 DBG Set partial_line_waiting duration to 5s
2016-01-19T08:05:16+09:00 DBG Waiting for 1 prospectors to initialise
2016-01-19T08:05:16+09:00 DBG Harvest path: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Start harvesting unknown file: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Same file as before found. Fetch the state and persist it.
2016-01-19T08:05:16+09:00 DBG Resuming harvester on a previously harvested file: /var/log/syslog
2016-01-19T08:05:16+09:00 INFO Harvester started for file: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG harvest: "/var/log/syslog" position:208785 (offset snapshot:208785)
2016-01-19T08:05:16+09:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-01-19T08:05:16+09:00 DBG Registrar will re-save state for /var/log/syslog
2016-01-19T08:05:16+09:00 DBG No pending prospectors. Finishing setup
2016-01-19T08:05:16+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-19T08:05:16+09:00 INFO Starting Registrar
2016-01-19T08:05:16+09:00 INFO Start sending events to output
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Update existing file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Not harvesting, file didn't change: /var/log/syslog
2016-01-19T08:05:18+09:00 DBG Flushing spooler because of timemout. Events flushed: 474
2016-01-19T08:05:18+09:00 DBG send event
2016-01-19T08:05:18+09:00 DBG Start Preprocessing
2016-01-19T08:05:18+09:00 DBG Publish: { ... }

tomiford · January 18, 2016, 11:30pm

I apologize, this is a really basic error and what you get for copying tutorials from Google. There was a typo in the apache log path.

/var/log/apache/

should have been

/var/log/apache2/

Sorry for wasting your time.

ruflin · January 19, 2016, 7:22am

Glad you found the solution

Filebeat not reading apache logs?

Ignore files which are older then 24 hours

Additional fields which can be freely defined