# Logstash/Elasticsearch Slow CSV Import **URL:** \ **Category:** Elasticsearch\ **Created:** [March 6, 2015, 7:30pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565 "2015-03-06T19:30:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![guardianmoon](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@guardianmoon](https://discuss.elastic.co/u/guardianmoon)\ **Post date:** [March 6, 2015, 7:30pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565/1 "2015-03-06T19:30:45Z")
I'm testing out the ELK stack on my desktop (ie 1 node) and thought I'd start by pulling a flat file, having logstash parse and output it to Elasticsearch. The setup was easy, but working through the flat file is painfully slow. The flat file is tab delimited, about 6million rows and 10 fields. I've messed around with the refresh\_interval, flush\_size, and workers, but the most I've been able to get is about 300 documents a second, which means 5-6hours. I'm having a hard time believing that that's right. In addition to this, logstash stops reading in the file at 579,242 documents every single time (about an hour in), but throws no errors. If I pull the index field out or the mapping template out (which is mostly specifying integers, dates and non-analyzed fields), then I start getting 4-6k documents loading per second. Any guesses as to what I'm doing wrong? If it's relevant, my desktop is set at 10gb (with a 4gb heap setting for ES) and 4 cores. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/043d9573-c07d-49f9-9410-9cb1424b2b78%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/043d9573-c07d-49f9-9410-9cb1424b2b78%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 6, 2015, 8:50pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565/2 "2015-03-06T20:50:19Z")
This may be worth taking to [Redirecting to Google Groups](https://groups.google.com/forum/?hl=en-GB#!forum/logstash-users), but can you show us your Logstash config? On 7 March 2015 at 06:30, Econgineer wrote: > I'm testing out the ELK stack on my desktop (ie 1 node) and thought I'd > start by pulling a flat file, having logstash parse and output it to > Elasticsearch. The setup was easy, but working through the flat file is > painfully slow. The flat file is tab delimited, about 6million rows and 10 > fields. I've messed around with the refresh\_interval, flush\_size, and > workers, but the most I've been able to get is about 300 documents a > second, which means 5-6hours. I'm having a hard time believing that that's > right. > > In addition to this, logstash stops reading in the file at 579,242 > documents every single time (about an hour in), but throws no errors. > > If I pull the index field out or the mapping template out (which is mostly > specifying integers, dates and non-analyzed fields), then I start getting > 4-6k documents loading per second. > > Any guesses as to what I'm doing wrong? > > If it's relevant, my desktop is set at 10gb (with a 4gb heap setting for > ES) and 4 cores. ---
**Author:** ![cdahlqvist](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@cdahlqvist](https://discuss.elastic.co/u/cdahlqvist)\ **Post date:** [March 8, 2015, 4:27pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565/3 "2015-03-08T16:27:28Z")
Hi, Can you please share you logstash configuration, some sample data as well as your mappings? Best regards, Christian On Friday, March 6, 2015 at 11:30:45 AM UTC-8, Econgineer wrote: > I'm testing out the ELK stack on my desktop (ie 1 node) and thought I'd > start by pulling a flat file, having logstash parse and output it to > Elasticsearch. The setup was easy, but working through the flat file is > painfully slow. The flat file is tab delimited, about 6million rows and 10 > fields. I've messed around with the refresh\_interval, flush\_size, and > workers, but the most I've been able to get is about 300 documents a > second, which means 5-6hours. I'm having a hard time believing that that's > right. > > In addition to this, logstash stops reading in the file at 579,242 > documents every single time (about an hour in), but throws no errors. > > If I pull the index field out or the mapping template out (which is mostly > specifying integers, dates and non-analyzed fields), then I start getting > 4-6k documents loading per second. > > Any guesses as to what I'm doing wrong? > > If it's relevant, my desktop is set at 10gb (with a 4gb heap setting for > ES) and 4 cores. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0e3bfebb-cbb1-4500-a1fa-3c784cf42cb4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0e3bfebb-cbb1-4500-a1fa-3c784cf42cb4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![guardianmoon](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@guardianmoon](https://discuss.elastic.co/u/guardianmoon)\ **Post date:** [March 8, 2015, 4:30pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565/4 "2015-03-08T16:30:11Z")
Turns out I just had the wrong character encoding set. Everythings working great at 2-3k documents a second now! Thanks! On Friday, March 6, 2015 at 11:30:45 AM UTC-8, Econgineer wrote: > I'm testing out the ELK stack on my desktop (ie 1 node) and thought I'd > start by pulling a flat file, having logstash parse and output it to > Elasticsearch. The setup was easy, but working through the flat file is > painfully slow. The flat file is tab delimited, about 6million rows and 10 > fields. I've messed around with the refresh\_interval, flush\_size, and > workers, but the most I've been able to get is about 300 documents a > second, which means 5-6hours. I'm having a hard time believing that that's > right. > > In addition to this, logstash stops reading in the file at 579,242 > documents every single time (about an hour in), but throws no errors. > > If I pull the index field out or the mapping template out (which is mostly > specifying integers, dates and non-analyzed fields), then I start getting > 4-6k documents loading per second. > > Any guesses as to what I'm doing wrong? > > If it's relevant, my desktop is set at 10gb (with a 4gb heap setting for > ES) and 4 cores. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3d4a986-56d9-4080-93d0-1bc17eb880be%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3d4a986-56d9-4080-93d0-1bc17eb880be%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [June 8, 2016, 8:57am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565/5 "2016-06-08T08:57:20Z")
Can you share the config values before and after to help others with a similar problem? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:45pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-slow-csv-import/22565/6 "2017-07-05T22:45:23Z")