# Netflow gaps between filebeat and elasticsearch

**URL:** <https://discuss.elastic.co/t/netflow-gaps-between-filebeat-and-elasticsearch/224105>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 18, 2020, 12:20pm UTC](https://discuss.elastic.co/t/netflow-gaps-between-filebeat-and-elasticsearch/224105 "2020-03-18T12:20:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kupauw](https://avatars.discourse-cdn.com/v4/letter/k/848f3c/32.png) [@Kupauw](https://discuss.elastic.co/u/Kupauw)\
**Post date:** [March 18, 2020, 12:20pm UTC](https://discuss.elastic.co/t/netflow-gaps-between-filebeat-and-elasticsearch/224105/1 "2020-03-18T12:20:49Z")

</div>

Hi everyone,

I have been struggeling with parsing netflow from my Cisco FTD (cluster mode) with around 12k events per second. Through filebeat -\> elasticsearch.  
(Using version 7.6.0 for filebeat/elastic)

I have 2 servers, 1 for filebeat the other for elasticsearch. They have the following hardware config:  
Filebeat: 8xCPU, 16Gb Memory, 250Gb storage (SSD).  
Elastic: 16xCPU, 32Gb Memory, 7Tb storage (SSD).  
Networking is 10gb.  
(probably a bit oversized but we can change that in the future)

On my filebeat node i can see a constant flow of netflow packets comming from my firewall. (I checked this with tcpdump -nni any port 2055). But when i do a tcpdump on the output side of filebeat to elastic (tcpdump -nny any port 9200) i can see that sometimes filebeat stops sending data to my elastic node all together but resumes after some time. You can see this in this picture:

 ![Screenshot_2020-03-18 Stack Monitoring - Elasticsearch - Indices - filebeat-7 6 0-2020 03 18-000001 - Overview - Kibana](https://us1.discourse-cdn.com/elastic/original/3X/1/5/15ced19fc5331e013b8f84023be51ec9b3bdddeb.png)

When using htop during these outages i can see barely any usage on my CPU and memory. So it looks like filebeat is dropping traffic to my elastic node but i cant figure out why. I have been toying around with the filebeat output settings:

```auto
    bulk_max_size: 4096
          worker: 2

```

I tryed smaller bulk sizes and more or less workers. also i have been testing the queue.mem settings from filebeat with multiple settings:

```auto
    queue.mem:
          events: 4096
          flush.min_events: 0

```

On the elastic side i temporarely disabled replication and changed index refresh to 30seconds.

Could anyone point me in the right direction or could give me any insight?

Thanks in advance

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 15, 2020, 12:20pm UTC](https://discuss.elastic.co/t/netflow-gaps-between-filebeat-and-elasticsearch/224105/2 "2020-04-15T12:20:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.