# Increasing logstash throughput for the file input and file output **URL:** \ **Category:** Logstash\ **Created:** [January 11, 2016, 4:49pm UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893 "2016-01-11T16:49:20Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![puneet035](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@puneet035](https://discuss.elastic.co/u/puneet035)\ **Post date:** [January 11, 2016, 4:49pm UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/1 "2016-01-11T16:49:20Z")
Dear all, How to increase logstash throughput for file input and file output plugins. I am using logstash on windows environment. My sample test produces 1.7GB of log file via stream writer, and I am moving these log messages via logstash to some other location in the same machine (and on same drive). To move 1.7 GB content, log stash takes more than 30 minutes or so. Is there any way I can improve this time? Below is my sample log stash configuration, and sample file to generate logs. ---------------------Log stash configuration---------------------------------- input { file{ path =\> "D:/Log/\*" start\_position =\> "beginning" } } output { file{ path =\> "D:\SampleLog.txt" message\_format =\> "%{message}" flush\_interval =\> 0 } } -------------------------Testing program to generate logs--------------------------------------------------------------- static void Main(string[] args) { using (StreamWriter writer = new StreamWriter(@"D:\Log\SampleLog.txt")) { for (int i = 0; i \< 100000000; i++) { writer.WriteLine("Testing: " + i); } } } My machine has 16GB of RAM & intel i7 processor. Thanks, Puneet ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 12, 2016, 3:08am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/2 "2016-01-12T03:08:18Z")
How many workers are you giving to Logstash when starting it? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 12, 2016, 3:32am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/3 "2016-01-12T03:32:48Z")
> [@puneet035](#): > > flush\_interval =\> 0 Why are you flushing for every single message? What does performance look like with the default setting? ---
**Author:** ![puneet035](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@puneet035](https://discuss.elastic.co/u/puneet035)\ **Post date:** [January 12, 2016, 3:46am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/4 "2016-01-12T03:46:42Z")
@warkolm: I believe it's default. I haven't specified any worker parameters ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 12, 2016, 3:59am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/5 "2016-01-12T03:59:19Z")
When you start LS you can define workers, see the `-w` flag - [https://www.elastic.co/guide/en/logstash/current/command-line-flags.html](https://www.elastic.co/guide/en/logstash/current/command-line-flags.html) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 12, 2016, 4:09am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/6 "2016-01-12T04:09:43Z")
If there is no filter section in the configuration, I would not expect the number of filter workers to have an impact on performance. ---
**Author:** ![puneet035](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@puneet035](https://discuss.elastic.co/u/puneet035)\ **Post date:** [January 12, 2016, 4:21am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/7 "2016-01-12T04:21:24Z")
@Christian_Dahlqvist: I tried after removing flush\_interval =\>0 and still it takes more than 30 minutes to sync 1.8 GB file. And I agree that increasing worker threads suggested by @warkolm will not have any impact, as it impacts only filters worker threads. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 12, 2016, 4:28am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/8 "2016-01-12T04:28:02Z")
Using logstash for copying files seems like an unusual use case. What is the rationale behind this performance test? ---
**Author:** ![puneet035](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@puneet035](https://discuss.elastic.co/u/puneet035)\ **Post date:** [January 12, 2016, 4:32am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/9 "2016-01-12T04:32:12Z")
We have various clients machines which are generating logs in txt files. We want to move all these text logs to one centralized server where it be searched & provided it to the user. My test case is just to check how fast is the logstash syncing process with file input and output. The time which logstash takes to move the files is crucial as well. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 12, 2016, 7:16am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/10 "2016-01-12T07:16:01Z")
Ahh yes, good point. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 12, 2016, 8:12am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/11 "2016-01-12T08:12:24Z")
How are the logs going to be searched once they have been moved to the central location? If you are planning on using Elasticsearch for this, there is generally no need to write logs into a central location before processing them and then indexing them directly into Elasticsearch. ---
**Author:** ![puneet035](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@puneet035](https://discuss.elastic.co/u/puneet035)\ **Post date:** [January 12, 2016, 5:00pm UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/12 "2016-01-12T17:00:47Z")
We are not planning to use elastic search for now. In our initial plan, logs will be moved in real time by logstash to centralized server and we provide some basic search utility to query the logs like query based on folder name. So the scope of our initiative is very limited, we want to use logstash to move logs from different client machines to the centralized server. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:16am UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-for-the-file-input-and-file-output/38893/13 "2017-07-06T05:16:00Z")