Hi,
I've been successfully using the IMAP plugin for the past few months and now I'm doing some tweaking and would love to have some insight on the inner workings.
These questions originate from using an office365 account and occasionally the pipeline gets stopped because my results get throttled because I reach the 60 requests/min maximum. I'm not sure why I am reaching that because I'm only sending about 15 requests/min according to my intervals of all my inputs. I changed the batch.delay from 5 to 10000 because maybe 5ms is too quick for imap to keep up with. The mailbox I'm using has upward of 500k messages in some of the folders I'm accessing, where sometimes 475k are read and 25k are still unread if you can foresee that to be an issue.
-
When imap goes into the mailbox, does it start from the oldest and then scroll up to find the oldest unread message and then run through the fetch_count value of unreads until # of emails retrieved = fetch_count?
-
When multiple folders are being pulled in, does the first one on the input .conf file get sent off first or all simultaneously? Let's say the fetch_count is 100 for each folder, the interval is 20s and I'm importing 3 folders. If the first folder has 100 unread messages, the second 50 and the third 20, how does LS go through the 3? Does it import the first folder, wait 20 seconds, then import the second and because the fetch count isn't reached, goes to the third? If my batch.size is 200, does it go through all 3 before sending it through?
2a) I'm aware that in logstash.yml one can adjust the pipeline.batch.size and pipeline.batch.delay. So does LS go through all of the inputs and once an accumulation of the inputs add up to the pipeline.batch.size will it then send the data through? Or does the pipeline.batch.size criterion only apply for each individual imap input?
- I can't seem to tweak the logstash.yml file that makes any effective difference. Given the state of my system, would you recommend a batch size of 1000 to be reasonable? Is a 10s batch delay outrageous?
Thanks if you have any input! 