I am using the filebeat 5.3.2 and I want to run filebeat as a non-root user.
Before I change the filebeat's directory privileges, filebeat works good.
the /etc/filebeat/filebeat.yml :
filebeat.prospectors:
- input_type: log
paths:
- /data/logs/logstash_*.log
output.console:
pretty: true
the command run the filebeat:
./bin/filebeat -c /etc/filebeat/filebeat.yml -e -d "*"
meanwhile, I echo some JSON text to /data/logs/logstash_*.log.
But after I use ansible change filebeat working directory privileges, prospector don't deal withs the files changes, the ansible playbook like this:
---
- hosts: test-server
remote_user: man
sudo: yes
tasks:
- name: download filebeat deb
get_url:
url: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.3.2-amd64.deb
dest: /tmp/filebeat.deb
- name: install filebeat
shell: sudo dpkg -i /tmp/filebeat.deb
- copy:
src: /home/emersonwang/filebeat.yml
dest: /etc/filebeat/filebeat.yml
owner: root
group: root
mode: 0644
- user:
name: filebeat
comment: "Filebeat Service Account"
shell: /bin/false
createhome: no
- copy:
src: /home/emersonwang/filebeat
dest: /etc/init.d/filebeat
owner: root
group: root
mode: 0755
- name: create filebeat log dir
file:
path: /var/log/filebeat
state: directory
owner: filebeat
group: filebeat
- name: create filebeat pid dir
file:
path: /var/run/filebeat
state: directory
owner: filebeat
group: filebeat
- name: change /var/lib/filebeat privilege
file:
path: /var/lib/filebeat
state: directory
owner: filebeat
group: filebeat
recurse: yes
- name: change /usr/share/filebeat privilege
file:
path: /var/share/filebeat
state: directory
owner: filebeat
group: filebeat
recurse:
the debug info like this:
2017/05/03 02:58:44.144975 console.go:83: CRIT Unable to publish events to console: <nil>
2017/05/03 02:58:44.145010 sync.go:47: DBG Shutting down sync publisher
2017/05/03 02:58:46.130934 log_file.go:84: DBG End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:58:49.129586 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:58:49.129916 prospector.go:183: DBG Run prospector
2017/05/03 02:58:49.129971 prospector_log.go:70: DBG Start next scan
2017/05/03 02:58:49.130114 prospector_log.go:226: DBG Check file for harvesting: /data/logs/logstash_gateway.log
2017/05/03 02:58:49.130174 prospector_log.go:259: DBG Update existing file for harvesting: /data/logs/logstash_gateway.log, offset: 10953
2017/05/03 02:58:49.130254 prospector_log.go:311: DBG Harvester for file is still running: /data/logs/logstash_gateway.log
2017/05/03 02:58:49.130301 prospector_log.go:91: DBG Prospector states cleaned up. Before: 1, After: 1
2017/05/03 02:58:54.129860 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:58:54.131325 log_file.go:84: DBG End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:58:59.130049 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:58:59.130481 prospector.go:183: DBG Run prospector
2017/05/03 02:58:59.130491 prospector_log.go:70: DBG Start next scan
2017/05/03 02:58:59.130556 prospector_log.go:226: DBG Check file for harvesting: /data/logs/logstash_gateway.log
2017/05/03 02:58:59.130565 prospector_log.go:259: DBG Update existing file for harvesting: /data/logs/logstash_gateway.log, offset: 10953
2017/05/03 02:58:59.130569 prospector_log.go:311: DBG Harvester for file is still running: /data/logs/logstash_gateway.log
2017/05/03 02:58:59.130575 prospector_log.go:91: DBG Prospector states cleaned up. Before: 1, After: 1
2017/05/03 02:59:04.130215 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:59:04.131552 log_file.go:84: DBG End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:05.131749 log_file.go:84: DBG End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:07.131960 log_file.go:84: DBG End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:09.129105 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.publisher.published_events=30
2017/05/03 02:59:09.130403 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 1
2017/05/03 02:59:09.130730 prospector.go:183: DBG Run prospector
2017/05/03 02:59:09.130739 prospector_log.go:70: DBG Start next scan
2017/05/03 02:59:09.130790 prospector_log.go:226: DBG Check file for harvesting: /data/logs/logstash_gateway.log
2017/05/03 02:59:09.130798 prospector_log.go:259: DBG Update existing file for harvesting: /data/logs/logstash_gateway.log, offset: 11208
2017/05/03 02:59:09.130802 prospector_log.go:311: DBG Harvester for file is still running: /data/logs/logstash_gateway.log
2017/05/03 02:59:09.130808 prospector_log.go:91: DBG Prospector states cleaned up. Before: 1, After: 1
2017/05/03 02:59:11.132163 log_file.go:84: DBG End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:14.130591 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:59:19.130778 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:59:19.130946 prospector.go:183: DBG Run prospector
2017/05/03 02:59:19.130956 prospector_log.go:70: DBG Start next scan
the Harvester detect the file's changes, but not update the registry file, Prospector also not ouput the text to console.