Filebeat fails to pickup other files

Elastic Stack Beats
21

Registry cleanup is not implemented it. We are still discussion the options. Follow this issue for more details: https://github.com/elastic/beats/issues/1600

About the files like /flow_base_dir/dump/flows-201606150045.dump. So not change happened to these files? Because the above means, that offset > file.Size() so it assumes the file was truncated.

There are some minor changes in alpha4 related to the offset, but should not be related to the above issues. But worth a try :slight_smile:

22

What I ended up doing is to update the bash script that generates these dump files, to rotate them between 12 files with 5 min increments. So then I don't have to worry about disk cleanup. In this case, I do see the file truncate in the logs, but this time it's for overwriting the file. So I think from that perspective it's good.

Every ten minutes I will see this in the logs, is this normal?

2016-06-16T15:00:17Z WARN client/metadata fetching metadata for all topics from broker broker-2.service.consul:7000
2016-06-16T15:00:17Z WARN client/metadata fetching metadata for all topics from broker broker-0.service..consul:7000

23

Good to hear you found a solution. I hope to fix this in the long term from the filebeat side.

About the warning message: This seems to come from the Kafka library (sarama) we use. TBH it sounds reasonable to me, not sure why it goes into WARN. @steffens knows perhaps more.

24

sarama lib only provides log.Println (used for INFO and ERROR messages). WARN level is used as 'compromise'. The message about fetching metadata is very normal.