# Strange Filebeat alerts processing **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 16, 2020, 12:57pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345 "2020-10-16T12:57:06Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 12:57pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/1 "2020-10-16T12:57:06Z")
Hi, dear community! We've faced the following problem in our Elastic stack: Filebeat sends the processed logs - about 100-150 Gb in average per day, but instead of straight line as it always be now we see such peaks (see screenshot). We've checked both Elastic and Filebeat logs in the debug mode but found nothing. Also we've checked network - traffic is nice and smooth. Could you advise, please? We use Elasticsearch/Filebeat 7.8.1. Thanks in advance. ![screenshot](https://us1.discourse-cdn.com/elastic/original/3X/c/a/cab17c77910fd15c389b873852aea55d3148d781.png) ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 5:59pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/2 "2020-10-16T17:59:58Z")
> [@MichaelA](#): > > but instead of straight line as it always be now we see such peaks (see screenshot). The chart seems to be cut off in the screenshot so it's not possible to tell what those lines (series) mean or what their scales are (no y-axis). Could you please post the full chart? Also, were you seeing the straight line behavior before and then it changed? If so, did you change something in your environment around the time the behavior changed? ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 6:37pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/3 "2020-10-16T18:37:56Z")
Hi, @shaunak! Thank you for the reply! I hope this chart looks better then the first. At the pointer normal behaviour line breaks into the peaks of re-sending logs by the Filebeat. We didn't do any changes to the Elastic cluster or our servers, it happens absolutely unexpected. We've checked our servers that send the logs, we've checked both Elastic and Filebeat logs in debug mode and found nothing suspicious. We've tried to restart Elastic cluster. We've checked network traffic and it looks nice. We are have no more ideas what else to do. So we're here 🙂 ![screenshotgfc](https://us1.discourse-cdn.com/elastic/original/3X/0/2/02c7d2cb5f7cf9cd61671568be6b888016bf0841.png) ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 6:43pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/4 "2020-10-16T18:43:56Z")
> [@MichaelA](#): > > re-sending logs This is interesting. How are you able to tell that Filebeat is trying to _re-send_ logs? This would happen if Filebeat was having trouble talking to Elasticsearch but then you'd see errors about in the Filebeat log. ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 6:54pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/5 "2020-10-16T18:54:18Z")
> [@shaunak](#): > > This is interesting. How are you able to tell that Filebeat is trying to _re-send_ logs? It looks like this, i suppose. If it didn't send it in the usual way, then it send stash of the logs, and we see these peaks. Is it right? ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 7:10pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/6 "2020-10-16T19:10:52Z")
Right, that was my first _theory_ as well (given the extremes — 0 for some time and then a peak, 0 and then a peak, etc.). But this theory can be easily tested by looking at Filebeat logs around the times of 0 activity — we should see errors about being able to send data to Elasticsearch, retrying, etc. You mentioned in your earlier comment that you found nothing suspicious in the logs though. So I'm not sure this is due to Filebeat retrying. 🤔 ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 7:15pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/7 "2020-10-16T19:15:49Z")
We didn't pay too much attention to the fact is there data in the Elastic or not. Let me explain: we're turn on the debug mode on the several servers that send the logs and left it for a couple of hours. Then we've filtered these logs for the key words as error, warning, disconnect and so on and found nothing. Also we've seen in the Filebeat debug log how it's send processed logs. But in the Elastic we still have nothing or these peaks. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 7:17pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/8 "2020-10-16T19:17:34Z")
Any chance you could shut down all but one of your Filebeat instances, then observe this chart for a bit? If it continues this zero-then-spike pattern, could you post the Filebeat logs (debug level would be nice but not required) here (appropriated redacted) from when the activity is 0? Shaunak ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 7:21pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/9 "2020-10-16T19:21:57Z")
Unfortunately, no, this is critical system ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 7:23pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/10 "2020-10-16T19:23:30Z")
Understood. In that case, is there any way you could filter that chart to only show traffic coming from one Filebeat instance? I'm thinking of different ways we could narrow down the scope of this problem so we might get some clues and/or make it easier to troubleshoot. ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 7:30pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/11 "2020-10-16T19:30:23Z")
This one i've posted earlier isn't suitable? If not, i will made one more ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 7:43pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/12 "2020-10-16T19:43:13Z")
> [@MichaelA](#): > > This one i've posted earlier isn't suitable? If not, i will made one more Oh, I didn't realize that was for traffic from a single Filebeat instance. In that case, you don't need to post a new one. If you could post some logs here (maybe a minute's worth) from that Filebeat instance (with sensitive data redacted) during the timestamps when its apparently not sending any logs over the network, that would be great. Thanks! ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 7:50pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/13 "2020-10-16T19:50:18Z")
Okay, i will try to find something. Thanks 🙂 ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 8:19pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/14 "2020-10-16T20:19:09Z")
This is the Filebeat log file during the timestamps when we not receiving any logs on the Elastic [https://file.io/xy2a5mAS3z36](https://file.io/xy2a5mAS3z36) ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 16, 2020, 8:30pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/15 "2020-10-16T20:30:49Z")
> [@MichaelA](#): > > [Deleted](https://file.io/xy2a5mAS3z36) That link 404s for me. ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 16, 2020, 8:35pm UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/16 "2020-10-16T20:35:35Z")
Hm, this is strange. Sorry, once again: [https://dropmefiles.com/TLgVW](https://dropmefiles.com/TLgVW) pwd: o1G62F ---
**Author:** ![MichaelA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaela/32/66353_2.png) [@MichaelA](https://discuss.elastic.co/u/MichaelA)\ **Post date:** [October 22, 2020, 7:35am UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/17 "2020-10-22T07:35:13Z")
Problem was solved. Issue was on the AWS side. Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 19, 2020, 9:35am UTC](https://discuss.elastic.co/t/strange-filebeat-alerts-processing/252345/18 "2020-11-19T09:35:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.