# Logstash taking too long to run **URL:** \ **Category:** Logstash\ **Created:** [April 17, 2020, 2:15pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536 "2020-04-17T14:15:39Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Robert\_Ga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_ga/32/51129_2.png) [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\ **Post date:** [April 17, 2020, 2:15pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/1 "2020-04-17T14:15:39Z")
Hello everyone, I started to index aprox. 1.000.000 json files into elastic seach using logstash. The logstash config contains some lowercase, renaming operations and if-else statements. I`m trying to index the data using 16 threads and batch size = 256. The problem is that if I look in Kibana, at Index Management, I can`t see the index created yet and the logstas is running since 3 hours ago From what I know, logstash should start the documents indexing after finnishing each batch. Is my understanding wrong? Also, I tried with about 30k documents and everything was fine. I want to know if something is wrong with my conf file/elastic search instance or it`s just normal to take that much. Thank you! ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [April 17, 2020, 3:36pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/2 "2020-04-17T15:36:21Z")
it is not normal. index will be created as soon as first batch is arrived via logstash. I have many pipleline running and when I introduce new pipeline index gets created right away. 7.6.1 had some problem I believe where in some cases it wasn't creating index. ---
**Author:** ![Robert\_Ga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_ga/32/51129_2.png) [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\ **Post date:** [April 17, 2020, 4:03pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/3 "2020-04-17T16:03:13Z")
It is pretty strange, because with less documents, it works as expected. It could be a problem if I have close order set with 1, or maybe the fact that sincedb\_path is /dev/null? ``` file { codec => json path => "user/documents/*.json" start_position => "beginning" sincedb_path => "/dev/null" close_older => 1 } ``` Also, I checked again and the entire stack is 7.6.2 Besides all that, if I check de log file, I don`t see any error/warning. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [April 17, 2020, 5:03pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/4 "2020-04-17T17:03:00Z")
I am not using that close\_older because logstash will remove file which is already read. don't you need '/' in front of user input { file { path =\> "/logstash/csv\_files/login\_stats/_\_hourly.csv_" start\_position =\> "beginning" mode =\> "read" sincedb\_path =\> "/dev/null" codec =\> plain { charset =\> "Windows-1252" } } } ---
**Author:** ![Robert\_Ga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_ga/32/51129_2.png) [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\ **Post date:** [April 17, 2020, 5:49pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/5 "2020-04-17T17:49:39Z")
Yes, I was using close\_older because firstly I ran into a problem regarding having way too many files open in the same time. Do you think that is the problem ? Regarding the path, I was providing a dummy path here, but the path is good. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [April 17, 2020, 6:36pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/6 "2020-04-17T18:36:47Z")
how many files we are talking about? check /etc/logstash/startup.options file any message in your logstash log file? ---
**Author:** ![Robert\_Ga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_ga/32/51129_2.png) [@Robert\_Ga](https://discuss.elastic.co/u/Robert_Ga)\ **Post date:** [April 18, 2020, 12:33pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/7 "2020-04-18T12:33:05Z")
For now, is about 1 milion json files. No error message in the log file. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [April 20, 2020, 1:24pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/8 "2020-04-20T13:24:29Z")
ok. then it might be hitting some other limit. but from configuration stand point it looks ok. can you try some kind of wild card match in like _ab_.json and test. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 18, 2020, 1:24pm UTC](https://discuss.elastic.co/t/logstash-taking-too-long-to-run/228536/9 "2020-05-18T13:24:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.