How can filebeat queue.mem send more than queue.mem.events to output?

Elastic Stack Beats
1

I am just trying to understand how, after a log load test has stopped producing logs and files have been rotated and deleted, filebeat continues to send 100s of thousands of log messages (in my case 1MM) to my output over 5-10 minutes, even though the queue.mem.events is set well below that number?

If the max events I want to handle is X, where on earth is filebeat getting the more than X events from? If queue.mem does not configure the buffer then what does?

2

I'm guessing this is linux? Maybe Windows works like that too anyway, not a Windows expert.

If filebeat(or anything) is reading a file (has a open handle on it) which is probably the case in your scenario, then deleting the file as you say, IS impossible. The filesystem will acknowledge the delete without error to the program making the delete call but in reality the inode will remain and the data too.
Essentially the file is still there but invisible to commands like "ls". The pointer that "ls" would see is gone but the inode for this file still exists and filebeat can continue reading it as long as it wishes to. (as long as it doesn't close it, allowing the filesystem to reclaim the inode for real.)
I suspect this is what you're describing, Filebeat is just continuing to read the files that you think are deleted. Because you don't see them... but Filebeat does.

As this is more OS level stuff and outside the scope of filebeat itself I would point to something like:

as a reference and you can go as deep as you want on your own, if that is useful or is actually the case in your scenario. That, in and of itself has nothing to do with filebeat.
An httpd server can send log to a "deleted" file for years if it keeps it open, for example. (and filebeat could read it too... depending on the config of filebeat...)
I can fill my disks with files, then deleted them... and my disk will still be full. (If something or someone, has a handle on those files.)

As it relates to filebeat harvesting files what you need to know and the options available to influence what happens are in the filebeat documentation.
All the settings called:
ignore_*
close_*
clean_*
scan_* / scan.*
tail_*
etc.
In Log input | Filebeat Reference [8.11] | Elastic

Exists because filebeat is specialized software for harvesting files and tries to give you as much knobs to obtain the desired behavior you need and for filebeat to do what you want considering all the different life cycles that are possible for files and inodes on filesystems.

For example, I use close_timeout to force filebeat to close its handles on file periodically to prevent situations where a disk gets FULL of "deleted" files that still exists because filebeat keeps them open.
Carefully read the description for all those settings and you'll see what can happen and how filebeat can be configured to behave like you expect.

3

I wanted to add, as you can see, filebeat is trying REAL hard to harvest everything and not loose the events you configured it to harvest. With your current config, that means filebeat harvests "deleted" files because it's not finished... It doesn't want to loose events as it wants to guarantee their delivery to ES.

4

Thanks for your detailed response.

For context, I am running the load test in GKE, which has file rotation set to 10MB and a keeps (I think) 3-4 rotated files around.

For our config, we are setting close_inactive: 1m and harvester_limit: 80, so given 10 containers each logging around 10MB every 15-20s, it seems off to me that handlers will stick around long enough to capture 1M+ latent events. Is there something else going on?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.