# Import of huge quantity of csv files in elasticsearch with logstash **URL:** \ **Category:** Logstash\ **Created:** [November 17, 2017, 9:05am UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092 "2017-11-17T09:05:46Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Wills1](https://avatars.discourse-cdn.com/v4/letter/w/85e7bf/32.png) [@Wills1](https://discuss.elastic.co/u/Wills1)\ **Post date:** [November 17, 2017, 9:05am UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/1 "2017-11-17T09:05:46Z")
Hi ! I'm new in ELK and have some questions concerning the import of multiple csv-files in elasticsearch via logstash. I have a folder with more than 425 csv-files within and each csv file have different column header; So I want to import all this cvs files in elasticsearch via logstash with one config file. Can you please tell me how my config-file must look like ? Thanks. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2017, 9:13am UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/2 "2017-11-17T09:13:44Z")
If every file is different then you will likely need to have seperate configs ☹ ---
**Author:** ![Wills1](https://avatars.discourse-cdn.com/v4/letter/w/85e7bf/32.png) [@Wills1](https://discuss.elastic.co/u/Wills1)\ **Post date:** [November 17, 2017, 10:21am UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/3 "2017-11-17T10:21:30Z")
thanks for your quick reply @warkolm. So when I good understand it mean if I want absolutly import them with one config-file, I must differentiate each path of each csv-file with the "conditionals" right ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2017, 7:59pm UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/4 "2017-11-17T19:59:48Z")
You'd probably be better off running logstash with `-e` and pass the config in via the command line. Then you can programatically iterate over the various files and generate appropriate configs. ---
**Author:** ![Wills1](https://avatars.discourse-cdn.com/v4/letter/w/85e7bf/32.png) [@Wills1](https://discuss.elastic.co/u/Wills1)\ **Post date:** [November 17, 2017, 9:32pm UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/5 "2017-11-17T21:32:34Z")
Sorry @warkolm I'm new in ELK and I don't really understand what you mean 😅 Cant you please give me an example ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 18, 2017, 9:56am UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/6 "2017-11-18T09:56:24Z")
You can call Logstash with the `-e` flag and then pass in a config in the shell/cli, per [https://www.elastic.co/guide/en/logstash/6.0/running-logstash-command-line.html#command-line-flags](https://www.elastic.co/guide/en/logstash/6.0/running-logstash-command-line.html#command-line-flags). Then you could script something to pass in changing configs based on the files and the headers. Also we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out! 😉 ---
**Author:** ![Wills1](https://avatars.discourse-cdn.com/v4/letter/w/85e7bf/32.png) [@Wills1](https://discuss.elastic.co/u/Wills1)\ **Post date:** [November 21, 2017, 12:02pm UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/7 "2017-11-21T12:02:54Z")
Hi @warkolm, I didn't find something about iterations in Logstash. Can you please tell me how I can iterate over this various files ? It would be great if you can give me an example and tell me how the config file must look like. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 21, 2017, 6:50pm UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/8 "2017-11-21T18:50:49Z")
That'll really depend on what OS you are on. Even then you're asking a lot of things to be done, and I don't want to be rude but I don't have the time to properly help you I am sorry to say ☹ If you're relatively new to doing this sort of thing, are you able to ask someone at your company that might be able to walk you through it? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 21, 2017, 7:12pm UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/9 "2017-11-21T19:12:47Z")
If the first line of each file has the column headers then you might be able to use the same config, assuming you want the fields of the document to be named after the columns. ``` filter { csv { autodetect_column_names => true } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 19, 2017, 7:12pm UTC](https://discuss.elastic.co/t/import-of-huge-quantity-of-csv-files-in-elasticsearch-with-logstash/108092/10 "2017-12-19T19:12:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.