How to process apache log files in Filebeat?

Elastic Stack Beats
1

Hi,

I have set up Filebeat and Logstash to process the apache access logs. In my filebeat.yml file, I have specified a director from where logs should be picked.

When I first copied the file, it got picked and got processed too. However, if I am modifying the same file, file beat is not processing it. Similarly if i add a new log file at the same location that is also not getting picked.

Can someone explain the reason for it?

The configuration of filebeat.yml file is as follows:

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • input_type: log

Paths that should be crawled and fetched. Glob based paths.

paths:
#- /var/log/*.log
- D:\Softwares\LogMonitoring\logs*

2

Can you share the log output from filebeat when you do the above. There you should see in more detail on what happens. Which version are you using?

3

The version of filebeat is 5.2

The log snapshot is as follows:

2017-02-16T11:45:53+05:30 INFO Home path: [D:\Softwares\LogMonitoring\filebeat] Config path: [D:\Softwares\LogMonitoring\filebeat] Data path: [D:\Softwares\LogMonitoring\filebeat\data] Logs path: [D:\Softwares\LogMonitoring\filebeat\logs]
2017-02-16T11:45:53+05:30 INFO Setup Beat: filebeat; Version: 5.2.0
2017-02-16T11:45:53+05:30 INFO Max Retries set to: 3
2017-02-16T11:45:53+05:30 INFO Activated logstash as output plugin.
2017-02-16T11:45:53+05:30 INFO Publisher name: LT0004658
2017-02-16T11:45:53+05:30 INFO Flush Interval set to: 1s
2017-02-16T11:45:53+05:30 INFO Max Bulk Size set to: 2048
2017-02-16T11:45:53+05:30 INFO filebeat start running.
2017-02-16T11:45:53+05:30 INFO Registry file set to: D:\Softwares\LogMonitoring\filebeat\data\registry
2017-02-16T11:45:53+05:30 INFO Loading registrar data from D:\Softwares\LogMonitoring\filebeat\data\registry
2017-02-16T11:45:53+05:30 INFO States Loaded from registrar: 1
2017-02-16T11:45:53+05:30 INFO Loading Prospectors: 1
2017-02-16T11:45:53+05:30 INFO Starting Registrar
2017-02-16T11:45:53+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-02-16T11:45:53+05:30 INFO Start sending events to output
2017-02-16T11:45:53+05:30 INFO Prospector with previous states loaded: 1
2017-02-16T11:45:53+05:30 INFO Loading Prospectors completed. Number of prospectors: 1
2017-02-16T11:45:53+05:30 INFO All prospectors are initialised and running with 1 states to persist
2017-02-16T11:45:53+05:30 INFO Starting prospector of type: log
2017-02-16T11:46:23+05:30 INFO Non-zero metrics in the last 30s: registrar.writes=1 registar.states.current=1 registrar.states.update=1 publish.events=1
2017-02-16T11:46:53+05:30 INFO No non-zero metrics in the last 30s
2017-02-16T11:47:10+05:30 INFO Stopping filebeat
2017-02-16T11:47:10+05:30 INFO Stopping Crawler
2017-02-16T11:47:10+05:30 INFO Stopping 1 prospectors
2017-02-16T11:47:10+05:30 INFO Stopping Prospector
2017-02-16T11:47:10+05:30 INFO Prospector channel stopped
2017-02-16T11:47:10+05:30 INFO Prospector ticker stopped
2017-02-16T11:47:10+05:30 INFO Crawler stopped
2017-02-16T11:47:10+05:30 INFO Stopping spooler
2017-02-16T11:47:10+05:30 INFO Stopping Registrar
2017-02-16T11:47:10+05:30 INFO Ending Registrar
2017-02-16T11:47:10+05:30 INFO Total non-zero values: registar.states.current=1 publish.events=1 registrar.writes=2 registrar.states.update=1
2017-02-16T11:47:10+05:30 INFO Uptime: 1m17.3757368s
2017-02-16T11:47:10+05:30 INFO filebeat stopped.

4

Could you start filebeat with -e -d "*" and share the output?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.