Filebeat harvester detect the file's changes, but prospector don't deal with it


(Zijian Wang) #1

I am using the filebeat 5.3.2 and I want to run filebeat as a non-root user.

Before I change the filebeat's directory privileges, filebeat works good.

the /etc/filebeat/filebeat.yml :

filebeat.prospectors:
- input_type: log
  paths:
    - /data/logs/logstash_*.log

output.console:
  pretty: true

the command run the filebeat:

./bin/filebeat -c /etc/filebeat/filebeat.yml -e -d "*"

meanwhile, I echo some JSON text to /data/logs/logstash_*.log.

But after I use ansible change filebeat working directory privileges, prospector don't deal withs the files changes, the ansible playbook like this:

---
- hosts: test-server
  remote_user: man
  sudo: yes

  tasks:
  - name: download filebeat deb
    get_url:
      url: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.3.2-amd64.deb
      dest: /tmp/filebeat.deb
  - name: install filebeat
    shell: sudo dpkg -i /tmp/filebeat.deb
  - copy:
      src: /home/emersonwang/filebeat.yml
      dest: /etc/filebeat/filebeat.yml
      owner: root
      group: root
      mode: 0644
  - user:
      name: filebeat
      comment: "Filebeat Service Account"
      shell: /bin/false
      createhome: no
  - copy:
      src: /home/emersonwang/filebeat
      dest: /etc/init.d/filebeat
      owner: root
      group: root
      mode: 0755
  - name: create filebeat log dir
    file: 
      path: /var/log/filebeat
      state: directory
      owner: filebeat
      group: filebeat
  - name: create filebeat pid dir
    file: 
      path: /var/run/filebeat
      state: directory
      owner: filebeat
      group: filebeat
  - name: change /var/lib/filebeat privilege
    file:
      path: /var/lib/filebeat
      state: directory
      owner: filebeat
      group: filebeat
      recurse: yes
  - name: change /usr/share/filebeat privilege
    file:
      path: /var/share/filebeat
      state: directory
      owner: filebeat
      group: filebeat
      recurse: 

the debug info like this:

2017/05/03 02:58:44.144975 console.go:83: CRIT Unable to publish events to console: <nil>
2017/05/03 02:58:44.145010 sync.go:47: DBG  Shutting down sync publisher
2017/05/03 02:58:46.130934 log_file.go:84: DBG  End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:58:49.129586 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:58:49.129916 prospector.go:183: DBG  Run prospector
2017/05/03 02:58:49.129971 prospector_log.go:70: DBG  Start next scan
2017/05/03 02:58:49.130114 prospector_log.go:226: DBG  Check file for harvesting: /data/logs/logstash_gateway.log
2017/05/03 02:58:49.130174 prospector_log.go:259: DBG  Update existing file for harvesting: /data/logs/logstash_gateway.log, offset: 10953
2017/05/03 02:58:49.130254 prospector_log.go:311: DBG  Harvester for file is still running: /data/logs/logstash_gateway.log
2017/05/03 02:58:49.130301 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 1, After: 1
2017/05/03 02:58:54.129860 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:58:54.131325 log_file.go:84: DBG  End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:58:59.130049 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:58:59.130481 prospector.go:183: DBG  Run prospector
2017/05/03 02:58:59.130491 prospector_log.go:70: DBG  Start next scan
2017/05/03 02:58:59.130556 prospector_log.go:226: DBG  Check file for harvesting: /data/logs/logstash_gateway.log
2017/05/03 02:58:59.130565 prospector_log.go:259: DBG  Update existing file for harvesting: /data/logs/logstash_gateway.log, offset: 10953
2017/05/03 02:58:59.130569 prospector_log.go:311: DBG  Harvester for file is still running: /data/logs/logstash_gateway.log
2017/05/03 02:58:59.130575 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 1, After: 1
2017/05/03 02:59:04.130215 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:59:04.131552 log_file.go:84: DBG  End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:05.131749 log_file.go:84: DBG  End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:07.131960 log_file.go:84: DBG  End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:09.129105 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.publisher.published_events=30
2017/05/03 02:59:09.130403 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 1
2017/05/03 02:59:09.130730 prospector.go:183: DBG  Run prospector
2017/05/03 02:59:09.130739 prospector_log.go:70: DBG  Start next scan
2017/05/03 02:59:09.130790 prospector_log.go:226: DBG  Check file for harvesting: /data/logs/logstash_gateway.log
2017/05/03 02:59:09.130798 prospector_log.go:259: DBG  Update existing file for harvesting: /data/logs/logstash_gateway.log, offset: 11208
2017/05/03 02:59:09.130802 prospector_log.go:311: DBG  Harvester for file is still running: /data/logs/logstash_gateway.log
2017/05/03 02:59:09.130808 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 1, After: 1
2017/05/03 02:59:11.132163 log_file.go:84: DBG  End of file reached: /data/logs/logstash_gateway.log; Backoff now.
2017/05/03 02:59:14.130591 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:59:19.130778 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017/05/03 02:59:19.130946 prospector.go:183: DBG  Run prospector
2017/05/03 02:59:19.130956 prospector_log.go:70: DBG  Start next scan

the Harvester detect the file's changes, but not update the registry file, Prospector also not ouput the text to console.


(ruflin) #2

Could it be that you hit this bug here? https://github.com/elastic/beats/pull/4045


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.