Hello,
When the registry file is about 13M, read log is about 1w/s. But when I remove the registry file then restart filebeat, read log grow up to 7w/s.
can the size of registry file affect reading speed ?
How do you measure the speed of reading? What does w stand for? write?
I measure the speed of reading by monitoring log of filebeat, this w stands for ten thousand.
Size of the registry should not affect the speed. The registry has a hash table to speed up lookup.
At what speed are new logs are written to the input files of Filebeat? I think Filebeat reads "slower" when the registry file is present, because it follows the file. It's speed cannot be higher than the speed of the writing of those files. When the registry is removed, Filebeat can read quicker, as it has more lines to process as it starts from the beginning of the file.
1 Like
I don't understand your explanation, the input files are downloaded from external part, and writing speed is very high.
You mean that you are not reading a local file?
It is many local files, and those files are downloaded from external part.
hey @jasper-zhang
i would like to understand the flow a bit better, so far i see some partial information so to help you better we will need to learn the whole flow.
you say you have local files downloaded from external source. when are these files downloaded? is it that the file is being downloaded from external source and at the same time they are processed by filebeat.
or is it the case where files are pre-downloaded and processed afterwards? or are you using some kind of network filesystem?
Hi， @Michal_Pristas
- These files are downloaded very minute;
- These files are downloaded as a zip package, then I unzip them, and then filebeat start to harvest them;
- All files are in local server, not in any network filesystem.
May these info can help you to understand the flow.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.