Filebeat not reading apache logs?

tomiford · January 18, 2016, 7:10am

Hi

I am new to filebeat and have set up the syslog files of our servers to be sent to logstash by filebeat. I am now trying to get apache logs into logstash but it does not seem to be reading the log.

When starting up filebeat this is the debug info:

sudo service filebeat restart

Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat 2016/01/18 06:45:06.457697 beat.go:97: DBG Initializing output plugins
2016/01/18 06:45:06.457773 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/01/18 06:45:06.520679 outputs.go:111: INFO Activated logstash as output plugin.
2016/01/18 06:45:06.521348 publish.go:198: DBG create output worker: 0x0, 0x0
2016/01/18 06:45:06.523384 publish.go:235: DBG No output is defined to store the topology. The server fields might not be filled.
2016/01/18 06:45:06.523684 publish.go:249: INFO Publisher name: myserver
2016/01/18 06:45:06.524167 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=200)
2016/01/18 06:45:06.524334 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1

When I look at the filebeat log it looks like this:

2016-01-18T15:45:06+09:00 DBG scan path /var/log/apache/.log
2016-01-18T15:45:06+09:00 DBG No pending prospectors. Finishing setup
2016-01-18T15:45:06+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-18T15:45:06+09:00 INFO Starting Registrar
2016-01-18T15:45:06+09:00 INFO Start sending events to output
2016-01-18T15:45:09+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T15:45:14+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T15:45:16+09:00 DBG Start next scan
2016-01-18T15:45:16+09:00 DBG scan path /var/log/apache/.log
2016-01-18T15:45:21+09:00 DBG Flushing spooler because of timemout. Events flushed: 0

The access logs have definitely been written to during this period.

The apache prospector looks like this:

paths:
- /var/log/apache/*.log
  document_type: apache
encoding: utf-8

Ignore files which are older then 24 hours

ignore_older: 24h

Additional fields which can be freely defined

fields:
type: apache

Any feedback would be appreciated.

Thanks

magnusbaeck · January 18, 2016, 7:31am

I suspect the user that runs the Filebeat process doesn't have sufficient permissions to access the contents of /var/log/apache.

tomiford · January 18, 2016, 7:40am

Thanks for the reply. Looking at top the Filebeat service is running as root so it should have access.

ruflin · January 18, 2016, 8:13am

Can you share a few more lines of the log file on starting up filebeat until the log file starts to repeat?

tomiford · January 18, 2016, 9:54am

Thanks for the reply. Is this what you were after?

2016-01-18T18:51:03+09:00 DBG Disable stderr logging
2016-01-18T18:51:03+09:00 DBG Initializing output plugins
2016-01-18T18:51:03+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-01-18T18:51:03+09:00 INFO Activated logstash as output plugin.
2016-01-18T18:51:03+09:00 DBG create output worker: 0x0, 0x0
2016-01-18T18:51:03+09:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-01-18T18:51:03+09:00 INFO Publisher name: asia-web-2
2016-01-18T18:51:03+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)
2016-01-18T18:51:03+09:00 INFO Init Beat: filebeat; Version: 1.0.1
2016-01-18T18:51:03+09:00 INFO filebeat sucessfully setup. Start running.
2016-01-18T18:51:03+09:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-01-18T18:51:03+09:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-01-18T18:51:03+09:00 DBG Set idleTimeoutDuration to 5s
2016-01-18T18:51:03+09:00 DBG File Configs: [/var/log/apache/.log]
2016-01-18T18:51:03+09:00 DBG Set ignore_older duration to 24h0m0s
2016-01-18T18:51:03+09:00 DBG Set scan_frequency duration to 10s
2016-01-18T18:51:03+09:00 DBG Set backoff duration to 1s
2016-01-18T18:51:03+09:00 DBG Set max_backoff duration to 10s
2016-01-18T18:51:03+09:00 DBG Set partial_line_waiting duration to 5s
2016-01-18T18:51:03+09:00 DBG Waiting for 1 prospectors to initialise
2016-01-18T18:51:03+09:00 DBG Harvest path: /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/.log
2016-01-18T18:51:03+09:00 DBG No pending prospectors. Finishing setup
2016-01-18T18:51:03+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-18T18:51:03+09:00 INFO Starting Registrar
2016-01-18T18:51:03+09:00 INFO Start sending events to output
2016-01-18T18:51:03+09:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-01-18T18:51:05+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T18:51:13+09:00 DBG Start next scan
2016-01-18T18:51:13+09:00 DBG Flushing spooler because of timemout. Events flushed: 0
2016-01-18T18:51:13+09:00 DBG scan path /var/log/apache/*.log

I have disabled the syslog that I was also sending to logstash server while I try to get apache log working.

ruflin · January 18, 2016, 11:59am

It seems like the harvester doesn't even see the file, so @magnusbeack could be right.

As far as I understand you comment you are using filebeat also for other files that work as expected?

tomiford · January 18, 2016, 11:08pm

Yes, if I change the prospector to syslog filebeat reads the file, this is the start up output when syslog is read:

2016-01-19T08:05:16+09:00 DBG Disable stderr logging
2016-01-19T08:05:16+09:00 DBG Initializing output plugins
2016-01-19T08:05:16+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-01-19T08:05:16+09:00 INFO Activated logstash as output plugin.
2016-01-19T08:05:16+09:00 DBG create output worker: 0x0, 0x0
2016-01-19T08:05:16+09:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-01-19T08:05:16+09:00 INFO Publisher name: asia-web-2
2016-01-19T08:05:16+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)
2016-01-19T08:05:16+09:00 INFO Init Beat: filebeat; Version: 1.0.1
2016-01-19T08:05:16+09:00 INFO filebeat sucessfully setup. Start running.
2016-01-19T08:05:16+09:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-01-19T08:05:16+09:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-01-19T08:05:16+09:00 DBG Set idleTimeoutDuration to 5s
2016-01-19T08:05:16+09:00 DBG File Configs: [/var/log/syslog]
2016-01-19T08:05:16+09:00 DBG Set ignore_older duration to 24h0m0s
2016-01-19T08:05:16+09:00 DBG Set scan_frequency duration to 10s
2016-01-19T08:05:16+09:00 DBG Set backoff duration to 1s
2016-01-19T08:05:16+09:00 DBG Set max_backoff duration to 10s
2016-01-19T08:05:16+09:00 DBG Set partial_line_waiting duration to 5s
2016-01-19T08:05:16+09:00 DBG Waiting for 1 prospectors to initialise
2016-01-19T08:05:16+09:00 DBG Harvest path: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Start harvesting unknown file: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Same file as before found. Fetch the state and persist it.
2016-01-19T08:05:16+09:00 DBG Resuming harvester on a previously harvested file: /var/log/syslog
2016-01-19T08:05:16+09:00 INFO Harvester started for file: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG harvest: "/var/log/syslog" position:208785 (offset snapshot:208785)
2016-01-19T08:05:16+09:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-01-19T08:05:16+09:00 DBG Registrar will re-save state for /var/log/syslog
2016-01-19T08:05:16+09:00 DBG No pending prospectors. Finishing setup
2016-01-19T08:05:16+09:00 INFO All prospectors initialised with 1 states to persist
2016-01-19T08:05:16+09:00 INFO Starting Registrar
2016-01-19T08:05:16+09:00 INFO Start sending events to output
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Update existing file for harvesting: /var/log/syslog
2016-01-19T08:05:16+09:00 DBG Not harvesting, file didn't change: /var/log/syslog
2016-01-19T08:05:18+09:00 DBG Flushing spooler because of timemout. Events flushed: 474
2016-01-19T08:05:18+09:00 DBG send event
2016-01-19T08:05:18+09:00 DBG Start Preprocessing
2016-01-19T08:05:18+09:00 DBG Publish: { ... }

tomiford · January 18, 2016, 11:30pm

I apologize, this is a really basic error and what you get for copying tutorials from Google. There was a typo in the apache log path.

/var/log/apache/

should have been

/var/log/apache2/

Sorry for wasting your time.

ruflin · January 19, 2016, 7:22am

Glad you found the solution

Topic		Replies	Views
Filebeat service doesn't send apache logs Beats filebeat	2	219	November 20, 2022
Filebeat's logs doesn't create Logstash	11	265	November 14, 2022
Filebeat not sending logs to Elasticsearch Beats filebeat	5	3509	July 6, 2016
Filebeat is not sending logs to Logstash, logstash is localhost Logstash	5	882	October 29, 2020
Filebeat does not send data to logstash Beats filebeat	10	1353	July 2, 2018

Filebeat not reading apache logs?

Ignore files which are older then 24 hours

Additional fields which can be freely defined

Related Topics