Filebeat not monitoring data


(Sahitya Uddaraju) #1

Hi Friends,

I set up Elasticsearch in our Lab, there in filebeat.yml I given the following configuration

filebeat:
paths:
- /var/log/*
document_type: syslog
scan_frequency: 10s

logstash:
# The Logstash hosts
hosts: ["...:5044"]

even though Scan_frequency:10s, Filebeat is not Monitoring files.
It just Adding the files at very first time when it see new file path is given.

Could anyone help me to sort it out.


(Magnus Bäck) #2

Please post full logs, i.e. start Filebeat with -v -d "*".

(I suspect -v is superfluous when -d is specified but it won't hurt.)


(Sahitya Uddaraju) #3

Hi Magnus, Please find the below lines are in my Filebeat.yml file.

filebeat:
prospectors:
paths:
- /var/log/*
type: log
paths:
- /var/log/cron
type: cron_log
document_type: syslog
scan_frequency: 10s
output:
elasticsearch:
hosts: ["localhost:9200"]
logstash:
hosts: ["ip_add:5044"] // here output as Logstash
shipper:
logging:
files:

this is total file , means remaining all lines contains # front of them.

File is adding, I can discover through kibana.
but, later changes in files are not monitoring by Filebeat.


(Magnus Bäck) #4

I asked for logs, not your configuration (although it's good to have the full configuration, but please make sure you format it as code since YAML is indentation-sensitive and this forum strips leading whitespace).


(Steffen Siering) #5

why have you configured both, elasticsearch and logstash output? This on purpose?


(Sahitya Uddaraju) #6

Hi Magnus, I would like to monitor syslog. and I didn't found syslog so I given the path like,

paths:

  • /var/log/*

in that folder all the below log files are there. problem is eventhough scan_frequency is 10s , it is not monitoring.

anaconda.log, audit, boot.log, btmp, cron, cups, dmesg, dracut.log, gdm, httpd, kern, maillog, mcelog, messages, mysqld.log, pm-powersave.log, sa-update.log, secure, spooler, tallylog, up2date, wtmp, Xorg.0.log, yum.log


(Sahitya Uddaraju) #7

Hi Steffens,

there is no purpose for putting output both as elasticsearch and logstash. It is accidentally forget to comment one.


(system) #8