How to process apache log files in Filebeat?


I have set up Filebeat and Logstash to process the apache access logs. In my filebeat.yml file, I have specified a director from where logs should be picked.

When I first copied the file, it got picked and got processed too. However, if I am modifying the same file, file beat is not processing it. Similarly if i add a new log file at the same location that is also not getting picked.

Can someone explain the reason for it?

The configuration of filebeat.yml file is as follows:


Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • input_type: log

Paths that should be crawled and fetched. Glob based paths.

#- /var/log/*.log
- D:\Softwares\LogMonitoring\logs*

Can you share the log output from filebeat when you do the above. There you should see in more detail on what happens. Which version are you using?

The version of filebeat is 5.2

The log snapshot is as follows:

2017-02-16T11:45:53+05:30 INFO Home path: [D:\Softwares\LogMonitoring\filebeat] Config path: [D:\Softwares\LogMonitoring\filebeat] Data path: [D:\Softwares\LogMonitoring\filebeat\data] Logs path: [D:\Softwares\LogMonitoring\filebeat\logs]
2017-02-16T11:45:53+05:30 INFO Setup Beat: filebeat; Version: 5.2.0
2017-02-16T11:45:53+05:30 INFO Max Retries set to: 3
2017-02-16T11:45:53+05:30 INFO Activated logstash as output plugin.
2017-02-16T11:45:53+05:30 INFO Publisher name: LT0004658
2017-02-16T11:45:53+05:30 INFO Flush Interval set to: 1s
2017-02-16T11:45:53+05:30 INFO Max Bulk Size set to: 2048
2017-02-16T11:45:53+05:30 INFO filebeat start running.
2017-02-16T11:45:53+05:30 INFO Registry file set to: D:\Softwares\LogMonitoring\filebeat\data\registry
2017-02-16T11:45:53+05:30 INFO Loading registrar data from D:\Softwares\LogMonitoring\filebeat\data\registry
2017-02-16T11:45:53+05:30 INFO States Loaded from registrar: 1
2017-02-16T11:45:53+05:30 INFO Loading Prospectors: 1
2017-02-16T11:45:53+05:30 INFO Starting Registrar
2017-02-16T11:45:53+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-02-16T11:45:53+05:30 INFO Start sending events to output
2017-02-16T11:45:53+05:30 INFO Prospector with previous states loaded: 1
2017-02-16T11:45:53+05:30 INFO Loading Prospectors completed. Number of prospectors: 1
2017-02-16T11:45:53+05:30 INFO All prospectors are initialised and running with 1 states to persist
2017-02-16T11:45:53+05:30 INFO Starting prospector of type: log
2017-02-16T11:46:23+05:30 INFO Non-zero metrics in the last 30s: registrar.writes=1 registar.states.current=1 registrar.states.update=1
2017-02-16T11:46:53+05:30 INFO No non-zero metrics in the last 30s
2017-02-16T11:47:10+05:30 INFO Stopping filebeat
2017-02-16T11:47:10+05:30 INFO Stopping Crawler
2017-02-16T11:47:10+05:30 INFO Stopping 1 prospectors
2017-02-16T11:47:10+05:30 INFO Stopping Prospector
2017-02-16T11:47:10+05:30 INFO Prospector channel stopped
2017-02-16T11:47:10+05:30 INFO Prospector ticker stopped
2017-02-16T11:47:10+05:30 INFO Crawler stopped
2017-02-16T11:47:10+05:30 INFO Stopping spooler
2017-02-16T11:47:10+05:30 INFO Stopping Registrar
2017-02-16T11:47:10+05:30 INFO Ending Registrar
2017-02-16T11:47:10+05:30 INFO Total non-zero values: registar.states.current=1 registrar.writes=2 registrar.states.update=1
2017-02-16T11:47:10+05:30 INFO Uptime: 1m17.3757368s
2017-02-16T11:47:10+05:30 INFO filebeat stopped.

Could you start filebeat with -e -d "*" and share the output?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.