Filebeat redis output less than 4096 events

Elastic Stack Beats
1

when use the redis output, repeat generating the log file (with the same filename) and removing it. the filebeat would send events no more than 4096.

even use one log file contains 6000 rows, the filebeat only send 4096 events.

filebeat version

filebeat version 6.0.0-beta2 (amd64), libbeat 6.0.0-beta2

redis-server version

Redis server v=3.2.100

configuation

filebeat:
  registry_file: /home/xxx/filebeat/data/filebeat_registry
  config:
    prospectors:
      path: filebeat.yml
      reload.enabled: false
      reload.period: 10s
  prospectors:
  - type: log
    paths:
      - /home/xxx/filebeat/input/*.log
    tail_files: true
output:
  drop_events_older: 1m
  redis:
    hosts: ["192.168.1.119:6379"]
    data_type: list
    key: logs
    db: 0
    timeout: 5
    bulk_max_size: 2048

logging:
  to_syslog: false
  to_files: true
  files:
    path: /home/xxx/filebeat/logs
    name: filebeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 5
  # Other available selectors are beat, publish, service
  #selectors: [ ]
  # Available log levels are: critical, error, warning, info, debug
  level: debug

redis list size

127.0.0.1:6379> llen logs
(integer) 2000
127.0.0.1:6379> llen logs
(integer) 4000
127.0.0.1:6379> llen logs
(integer) 4096
127.0.0.1:6379> llen logs
(integer) 4096

filebeat log

2017-09-27T15:33:20+08:00 DBG  Start next scan
2017-09-27T15:33:20+08:00 DBG  Check file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:20+08:00 DBG  Update existing file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log, offset: 37746
2017-09-27T15:33:20+08:00 DBG  Harvester for file is still running: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:20+08:00 DBG  Prospector states cleaned up. Before: 3, After: 3
2017-09-27T15:33:20+08:00 DBG  State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:20+08:00 DBG  Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:20+08:00 DBG  State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:20+08:00 DBG  Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:25+08:00 DBG  End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now.
2017-09-27T15:33:25+08:00 DBG  End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now.
2017-09-27T15:33:30+08:00 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=6633024 beat.memstats.memory_alloc=5499736 beat.memstats.memory_total=28098120 filebeat.events.active=2119 filebeat.events.added=2119 filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.output.events.acked=2096 libbeat.output.events.batches=10 libbeat.output.events.total=2096 libbeat.output.read.bytes=70 libbeat.output.write.bytes=1357354 libbeat.pipeline.events.active=2117 libbeat.pipeline.events.filtered=2 libbeat.pipeline.events.published=2116 libbeat.pipeline.events.total=2119
2017-09-27T15:33:30+08:00 DBG  Run prospector
2017-09-27T15:33:30+08:00 DBG  Start next scan
2017-09-27T15:33:30+08:00 DBG  Check file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:30+08:00 DBG  Update existing file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log, offset: 37746
2017-09-27T15:33:30+08:00 DBG  Harvester for file is still running: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:30+08:00 DBG  Prospector states cleaned up. Before: 3, After: 3
2017-09-27T15:33:30+08:00 DBG  State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:30+08:00 DBG  Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:30+08:00 DBG  State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:30+08:00 DBG  Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log
2017-09-27T15:33:35+08:00 DBG  End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now.
2017-09-27T15:33:35+08:00 DBG  End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now.
2017-09-27T15:33:40+08:00 DBG  Run prospector
2

Did you restart filebeat? If so, did you delete the registry file?

Please check the registry file for the actual file offset. The offset in the registry file contains the last ACKed event offsets (acked by redis) in your file.

The log shows some events are still waiting in filebeats pipeline.

3

I had deleted the registry file and restarted the filebeat, the redis received no more than 4096 events.
I event test both on windows and centos.
When I change the filebeat to another version, It works well.

4

Which filebeat version is it working for? Which filebeat version is failing for you?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.