# Stopping any incoming data from filebeat **URL:** \ **Category:** Elasticsearch\ **Created:** [June 26, 2019, 7:12am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497 "2019-06-26T07:12:48Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [June 26, 2019, 7:12am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497/1 "2019-06-26T07:12:48Z")
Since a while, I've been suffering from [timeout each time I wanted to make a change through REST API calls](https://discuss.elastic.co/t/es-6-x-timeout-by-curl-operations/180773). I've given the issue some thoughts and came to the idea of temporary blocking any incoming data so it wouldn't trigger any ingest or bulk operation. I have a few concerns: - I don't want to lose any data coming from the filebeat clients because the elastic search server was unavailable for too long. - If possible, I don't want to touch the firewall settings. I thought about removing the data and ingest role to the node? Is it a viable way to do that? ---
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [June 27, 2019, 9:00am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497/2 "2019-06-27T09:00:19Z")
Removing all the roles didn't help, instead it generated more errors and I still had the timeout by callling of a REST API call. Any suggestion? And if the only option I have is to block incoming communication through Firewall, what rule should I add/modify? ---
**Author:** ![chamilad](https://avatars.discourse-cdn.com/v4/letter/c/b77776/32.png) [@chamilad](https://discuss.elastic.co/u/chamilad)\ **Post date:** [July 8, 2019, 12:38am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497/3 "2019-07-08T00:38:09Z")
Hi, If you are using Logstash to collect logs, have you tried working with Persistent Queues? IIUC your use case can be helped by using a pre-calculated size for the queue at logstash and back-pressure filebeat when the queue is full. The filebeat logs will only be accepted when the queue is not filled again. I was able to work through a spike in incoming data using this flow. ---
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [July 8, 2019, 7:15am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497/4 "2019-07-08T07:15:50Z")
No I'm feeding the logs directly into the ingest nodes, but I am open to any idea which might help. ---
**Author:** ![chamilad](https://avatars.discourse-cdn.com/v4/letter/c/b77776/32.png) [@chamilad](https://discuss.elastic.co/u/chamilad)\ **Post date:** [July 11, 2019, 12:06am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497/5 "2019-07-11T00:06:24Z")
Hi, Let me explain how I tackled this. However it would be highly likely there's a better way of achieving the same thing (provided I understood your problem correctly). This worked for me 🙂 So the way log ingestion happens in this specific deployment was through log file -\> filebeat -\> logstash -\> elasticsearch. Logstash is used to throttle, parse, and enrich log entries. In my case it was a given that spikes could occur during the log ingestion. Persistent Queues seems to be the Elastic recommended way to buffer such spikes [1]. With Persistent Queues enabled, logstash will write all incoming logs from filebeat to a file based queue, and process them in order. If the queue fills up (you can define the size on disk or the number of events in flight as a limit) logstash will stop accepting events from filebeat until the queue becomes free again. Enabling persistent queues is pretty easy. Refer to the guide for that [2]. [1] - [https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#scaling-ingest](https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#scaling-ingest) [2] - [https://www.elastic.co/guide/en/logstash/current/persistent-queues.html](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2019, 12:06am UTC](https://discuss.elastic.co/t/stopping-any-incoming-data-from-filebeat/187497/6 "2019-08-08T00:06:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.