# Filebeat redis output less than 4096 events **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 27, 2017, 7:37am UTC](https://discuss.elastic.co/t/filebeat-redis-output-less-than-4096-events/101942 "2017-09-27T07:37:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![zhouyue](https://avatars.discourse-cdn.com/v4/letter/z/6bbea6/32.png) [@zhouyue](https://discuss.elastic.co/u/zhouyue)\ **Post date:** [September 27, 2017, 7:37am UTC](https://discuss.elastic.co/t/filebeat-redis-output-less-than-4096-events/101942/1 "2017-09-27T07:37:55Z")
when use the redis output, repeat generating the log file (with the same filename) and removing it. the filebeat would send events no more than 4096. even use one log file contains 6000 rows, the filebeat only send 4096 events. ## filebeat version filebeat version 6.0.0-beta2 (amd64), libbeat 6.0.0-beta2 ## redis-server version Redis server v=3.2.100 ## configuation ```auto filebeat: registry_file: /home/xxx/filebeat/data/filebeat_registry config: prospectors: path: filebeat.yml reload.enabled: false reload.period: 10s prospectors: - type: log paths: - /home/xxx/filebeat/input/*.log tail_files: true output: drop_events_older: 1m redis: hosts: ["192.168.1.119:6379"] data_type: list key: logs db: 0 timeout: 5 bulk_max_size: 2048 logging: to_syslog: false to_files: true files: path: /home/xxx/filebeat/logs name: filebeat.log rotateeverybytes: 10485760 # = 10MB keepfiles: 5 # Other available selectors are beat, publish, service #selectors: [] # Available log levels are: critical, error, warning, info, debug level: debug ``` ## redis list size ```auto 127.0.0.1:6379> llen logs (integer) 2000 127.0.0.1:6379> llen logs (integer) 4000 127.0.0.1:6379> llen logs (integer) 4096 127.0.0.1:6379> llen logs (integer) 4096 ``` ## filebeat log ```auto 2017-09-27T15:33:20+08:00 DBG Start next scan 2017-09-27T15:33:20+08:00 DBG Check file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:20+08:00 DBG Update existing file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log, offset: 37746 2017-09-27T15:33:20+08:00 DBG Harvester for file is still running: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:20+08:00 DBG Prospector states cleaned up. Before: 3, After: 3 2017-09-27T15:33:20+08:00 DBG State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:20+08:00 DBG Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:20+08:00 DBG State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:20+08:00 DBG Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:25+08:00 DBG End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now. 2017-09-27T15:33:25+08:00 DBG End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now. 2017-09-27T15:33:30+08:00 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=6633024 beat.memstats.memory_alloc=5499736 beat.memstats.memory_total=28098120 filebeat.events.active=2119 filebeat.events.added=2119 filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.output.events.acked=2096 libbeat.output.events.batches=10 libbeat.output.events.total=2096 libbeat.output.read.bytes=70 libbeat.output.write.bytes=1357354 libbeat.pipeline.events.active=2117 libbeat.pipeline.events.filtered=2 libbeat.pipeline.events.published=2116 libbeat.pipeline.events.total=2119 2017-09-27T15:33:30+08:00 DBG Run prospector 2017-09-27T15:33:30+08:00 DBG Start next scan 2017-09-27T15:33:30+08:00 DBG Check file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:30+08:00 DBG Update existing file for harvesting: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log, offset: 37746 2017-09-27T15:33:30+08:00 DBG Harvester for file is still running: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:30+08:00 DBG Prospector states cleaned up. Before: 3, After: 3 2017-09-27T15:33:30+08:00 DBG State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:30+08:00 DBG Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:30+08:00 DBG State for file not removed because harvester not finished: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:30+08:00 DBG Remove state for file as file removed or renamed: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log 2017-09-27T15:33:35+08:00 DBG End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now. 2017-09-27T15:33:35+08:00 DBG End of file reached: /home/zhouyue/filebeat/input/bitsharesapi-access-2.log; Backoff now. 2017-09-27T15:33:40+08:00 DBG Run prospector ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 4, 2017, 11:10am UTC](https://discuss.elastic.co/t/filebeat-redis-output-less-than-4096-events/101942/2 "2017-10-04T11:10:45Z")
Did you restart filebeat? If so, did you delete the registry file? Please check the registry file for the actual file offset. The offset in the registry file contains the last ACKed event offsets (acked by redis) in your file. The log shows some events are still waiting in filebeats pipeline. ---
**Author:** ![zhouyue](https://avatars.discourse-cdn.com/v4/letter/z/6bbea6/32.png) [@zhouyue](https://discuss.elastic.co/u/zhouyue)\ **Post date:** [October 11, 2017, 2:44am UTC](https://discuss.elastic.co/t/filebeat-redis-output-less-than-4096-events/101942/3 "2017-10-11T02:44:31Z")
I had deleted the registry file and restarted the filebeat, the redis received no more than 4096 events. I event test both on windows and centos. When I change the filebeat to another version, It works well. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 11, 2017, 10:12am UTC](https://discuss.elastic.co/t/filebeat-redis-output-less-than-4096-events/101942/4 "2017-10-11T10:12:40Z")
Which filebeat version is it working for? Which filebeat version is failing for you? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 8, 2017, 10:12am UTC](https://discuss.elastic.co/t/filebeat-redis-output-less-than-4096-events/101942/5 "2017-11-08T10:12:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.