# Bottleneck Data Pipeline
**URL:** https://discuss.elastic.co/t/bottleneck-data-pipeline/160756
**Category:** Beats
**Tags:** filebeat
**Created:** [December 13, 2018, 2:31pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756 "2018-12-13T14:31:08Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)
#### Post date: [December 13, 2018, 2:31pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/1 "2018-12-13T14:31:08Z")
I have a csv with more than 300kr of rows per hr. I use filebeat to ship data into elasticsearch.
My problem is that, the sending of data is very slow like 2k of rows per 3-5mins only and sometimes it stops for a while.
Are there any config hacks which will make the data shipping faster?
TIA
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 13, 2018, 2:55pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/2 "2018-12-13T14:55:15Z")
Where are you sending the data?
---
### Author:  [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [December 13, 2018, 3:08pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/3 "2018-12-13T15:08:07Z")
Also, what is the filebeat OS and is the CSV on local or shared disk?
---
### Author:  [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)
#### Post date: [December 14, 2018, 2:12am UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/4 "2018-12-14T02:12:37Z")
from filebeat -\> logstash -\> elasticsearch
---
### Author:  [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)
#### Post date: [December 14, 2018, 2:13am UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/5 "2018-12-14T02:13:20Z")
i'm using filebeat for Win OS. the CSV is stored in the local
---
### Author:  [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [December 14, 2018, 5:09pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/6 "2018-12-14T17:09:04Z")
Well, I guess the next step is to see if the delay is in harvesting or publishing. Have you checked the logs for filebeat and logstash? Do you have other beats sending OK?
---
### Author:  [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)
#### Post date: [December 15, 2018, 6:05am UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/7 "2018-12-15T06:05:12Z")
yes. there's no error in the filebeat logs.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 15, 2018, 12:00pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/8 "2018-12-15T12:00:34Z")
What is the specification of your Elasticsearch cluster? What kind of hardware and storage are you using?
If you want to test if Elasticsearch is limiting throughput, you can e.g. temporarily replace the Elasticsearch output with a file output and see if that changes the throughput of data collected.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 12, 2019, 12:00pm UTC](https://discuss.elastic.co/t/bottleneck-data-pipeline/160756/9 "2019-01-12T12:00:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.