# Filebeat not reading apache logs?

**URL:** <https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 18, 2016, 7:10am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419 "2016-01-18T07:10:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomiford](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@tomiford](https://discuss.elastic.co/u/tomiford)\
**Post date:** [January 18, 2016, 7:10am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/1 "2016-01-18T07:10:16Z")

</div>

Hi

I am new to filebeat and have set up the syslog files of our servers to be sent to logstash by filebeat. I am now trying to get apache logs into logstash but it does not seem to be reading the log.

When starting up filebeat this is the debug info:

sudo service filebeat restart

- Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat 2016/01/18 06:45:06.457697 beat.go:97: DBG Initializing output plugins  
2016/01/18 06:45:06.457773 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths  
2016/01/18 06:45:06.520679 outputs.go:111: INFO Activated logstash as output plugin.  
2016/01/18 06:45:06.521348 publish.go:198: DBG create output worker: 0x0, 0x0  
2016/01/18 06:45:06.523384 publish.go:235: DBG No output is defined to store the topology. The server fields might not be filled.  
2016/01/18 06:45:06.523684 publish.go:249: INFO Publisher name: myserver  
2016/01/18 06:45:06.524167 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=200)  
2016/01/18 06:45:06.524334 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1

When I look at the filebeat log it looks like this:

2016-01-18T15:45:06+09:00 DBG scan path /var/log/apache/_.log  
2016-01-18T15:45:06+09:00 DBG No pending prospectors. Finishing setup  
2016-01-18T15:45:06+09:00 INFO All prospectors initialised with 1 states to persist  
2016-01-18T15:45:06+09:00 INFO Starting Registrar  
2016-01-18T15:45:06+09:00 INFO Start sending events to output  
2016-01-18T15:45:09+09:00 DBG Flushing spooler because of timemout. Events flushed: 0  
2016-01-18T15:45:14+09:00 DBG Flushing spooler because of timemout. Events flushed: 0  
2016-01-18T15:45:16+09:00 DBG Start next scan  
2016-01-18T15:45:16+09:00 DBG scan path /var/log/apache/_.log  
2016-01-18T15:45:21+09:00 DBG Flushing spooler because of timemout. Events flushed: 0

The access logs have definitely been written to during this period.

The apache prospector looks like this:

- paths:

Any feedback would be appreciated.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 18, 2016, 7:31am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/2 "2016-01-18T07:31:24Z")

</div>

I suspect the user that runs the Filebeat process doesn't have sufficient permissions to access the contents of /var/log/apache.

---

<div class="post-metadata">

**Author:** ![tomiford](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@tomiford](https://discuss.elastic.co/u/tomiford)\
**Post date:** [January 18, 2016, 7:40am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/3 "2016-01-18T07:40:53Z")

</div>

Thanks for the reply. Looking at top the Filebeat service is running as root so it should have access.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [January 18, 2016, 8:13am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/4 "2016-01-18T08:13:57Z")

</div>

Can you share a few more lines of the log file on starting up filebeat until the log file starts to repeat?

---

<div class="post-metadata">

**Author:** ![tomiford](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@tomiford](https://discuss.elastic.co/u/tomiford)\
**Post date:** [January 18, 2016, 9:54am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/5 "2016-01-18T09:54:06Z")

</div>

Thanks for the reply. Is this what you were after?

2016-01-18T18:51:03+09:00 DBG Disable stderr logging  
2016-01-18T18:51:03+09:00 DBG Initializing output plugins  
2016-01-18T18:51:03+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths  
2016-01-18T18:51:03+09:00 INFO Activated logstash as output plugin.  
2016-01-18T18:51:03+09:00 DBG create output worker: 0x0, 0x0  
2016-01-18T18:51:03+09:00 DBG No output is defined to store the topology. The server fields might not be filled.  
2016-01-18T18:51:03+09:00 INFO Publisher name: asia-web-2  
2016-01-18T18:51:03+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)  
2016-01-18T18:51:03+09:00 INFO Init Beat: filebeat; Version: 1.0.1  
2016-01-18T18:51:03+09:00 INFO filebeat sucessfully setup. Start running.  
2016-01-18T18:51:03+09:00 INFO Registry file set to: /var/lib/filebeat/registry  
2016-01-18T18:51:03+09:00 INFO Loading registrar data from /var/lib/filebeat/registry  
2016-01-18T18:51:03+09:00 DBG Set idleTimeoutDuration to 5s  
2016-01-18T18:51:03+09:00 DBG File Configs: [/var/log/apache/_.log]  
2016-01-18T18:51:03+09:00 DBG Set ignore\_older duration to 24h0m0s  
2016-01-18T18:51:03+09:00 DBG Set scan\_frequency duration to 10s  
2016-01-18T18:51:03+09:00 DBG Set backoff duration to 1s  
2016-01-18T18:51:03+09:00 DBG Set max\_backoff duration to 10s  
2016-01-18T18:51:03+09:00 DBG Set partial\_line\_waiting duration to 5s  
2016-01-18T18:51:03+09:00 DBG Waiting for 1 prospectors to initialise  
2016-01-18T18:51:03+09:00 DBG Harvest path: /var/log/apache/_.log  
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/_.log  
2016-01-18T18:51:03+09:00 DBG scan path /var/log/apache/_.log  
2016-01-18T18:51:03+09:00 DBG No pending prospectors. Finishing setup  
2016-01-18T18:51:03+09:00 INFO All prospectors initialised with 1 states to persist  
2016-01-18T18:51:03+09:00 INFO Starting Registrar  
2016-01-18T18:51:03+09:00 INFO Start sending events to output  
2016-01-18T18:51:03+09:00 INFO Starting spooler: spool\_size: 1024; idle\_timeout: 5s  
2016-01-18T18:51:05+09:00 DBG Flushing spooler because of timemout. Events flushed: 0  
2016-01-18T18:51:13+09:00 DBG Start next scan  
2016-01-18T18:51:13+09:00 DBG Flushing spooler because of timemout. Events flushed: 0  
2016-01-18T18:51:13+09:00 DBG scan path /var/log/apache/\*.log

I have disabled the syslog that I was also sending to logstash server while I try to get apache log working.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [January 18, 2016, 11:59am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/6 "2016-01-18T11:59:02Z")

</div>

It seems like the harvester doesn't even see the file, so @magnusbeack could be right.

As far as I understand you comment you are using filebeat also for other files that work as expected?

---

<div class="post-metadata">

**Author:** ![tomiford](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@tomiford](https://discuss.elastic.co/u/tomiford)\
**Post date:** [January 18, 2016, 11:08pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/7 "2016-01-18T23:08:48Z")

</div>

Yes, if I change the prospector to syslog filebeat reads the file, this is the start up output when syslog is read:

2016-01-19T08:05:16+09:00 DBG Disable stderr logging  
2016-01-19T08:05:16+09:00 DBG Initializing output plugins  
2016-01-19T08:05:16+09:00 INFO GeoIP disabled: No paths were set under output.geoip.paths  
2016-01-19T08:05:16+09:00 INFO Activated logstash as output plugin.  
2016-01-19T08:05:16+09:00 DBG create output worker: 0x0, 0x0  
2016-01-19T08:05:16+09:00 DBG No output is defined to store the topology. The server fields might not be filled.  
2016-01-19T08:05:16+09:00 INFO Publisher name: asia-web-2  
2016-01-19T08:05:16+09:00 DBG create bulk processing worker (interval=1s, bulk size=200)  
2016-01-19T08:05:16+09:00 INFO Init Beat: filebeat; Version: 1.0.1  
2016-01-19T08:05:16+09:00 INFO filebeat sucessfully setup. Start running.  
2016-01-19T08:05:16+09:00 INFO Registry file set to: /var/lib/filebeat/registry  
2016-01-19T08:05:16+09:00 INFO Loading registrar data from /var/lib/filebeat/registry  
2016-01-19T08:05:16+09:00 DBG Set idleTimeoutDuration to 5s  
2016-01-19T08:05:16+09:00 DBG File Configs: [/var/log/syslog]  
2016-01-19T08:05:16+09:00 DBG Set ignore\_older duration to 24h0m0s  
2016-01-19T08:05:16+09:00 DBG Set scan\_frequency duration to 10s  
2016-01-19T08:05:16+09:00 DBG Set backoff duration to 1s  
2016-01-19T08:05:16+09:00 DBG Set max\_backoff duration to 10s  
2016-01-19T08:05:16+09:00 DBG Set partial\_line\_waiting duration to 5s  
2016-01-19T08:05:16+09:00 DBG Waiting for 1 prospectors to initialise  
2016-01-19T08:05:16+09:00 DBG Harvest path: /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG Start harvesting unknown file: /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG Same file as before found. Fetch the state and persist it.  
2016-01-19T08:05:16+09:00 DBG Resuming harvester on a previously harvested file: /var/log/syslog  
2016-01-19T08:05:16+09:00 INFO Harvester started for file: /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG harvest: "/var/log/syslog" position:208785 (offset snapshot:208785)  
2016-01-19T08:05:16+09:00 INFO Starting spooler: spool\_size: 1024; idle\_timeout: 5s  
2016-01-19T08:05:16+09:00 DBG Registrar will re-save state for /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG No pending prospectors. Finishing setup  
2016-01-19T08:05:16+09:00 INFO All prospectors initialised with 1 states to persist  
2016-01-19T08:05:16+09:00 INFO Starting Registrar  
2016-01-19T08:05:16+09:00 INFO Start sending events to output  
2016-01-19T08:05:16+09:00 DBG scan path /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG Check file for harvesting: /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG Update existing file for harvesting: /var/log/syslog  
2016-01-19T08:05:16+09:00 DBG Not harvesting, file didn't change: /var/log/syslog  
2016-01-19T08:05:18+09:00 DBG Flushing spooler because of timemout. Events flushed: 474  
2016-01-19T08:05:18+09:00 DBG send event  
2016-01-19T08:05:18+09:00 DBG Start Preprocessing  
2016-01-19T08:05:18+09:00 DBG Publish: { ... }

---

<div class="post-metadata">

**Author:** ![tomiford](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@tomiford](https://discuss.elastic.co/u/tomiford)\
**Post date:** [January 18, 2016, 11:30pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/8 "2016-01-18T23:30:21Z")

</div>

I apologize, this is a really basic error and what you get for copying tutorials from Google. There was a typo in the apache log path.

/var/log/apache/

should have been

/var/log/apache2/

Sorry for wasting your time.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [January 19, 2016, 7:22am UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/9 "2016-01-19T07:22:20Z")

</div>

Glad you found the solution 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:56pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-apache-logs/39419/10 "2017-07-05T21:56:36Z")

</div>