# Bulk importing CSVs with different headers **URL:** \ **Category:** Elasticsearch\ **Created:** [January 23, 2015, 2:21pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799 "2015-01-23T14:21:52Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ron\_3](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Ron\_3](https://discuss.elastic.co/u/Ron_3)\ **Post date:** [January 23, 2015, 2:21pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/1 "2015-01-23T14:21:52Z")
Hello all, So I'm trying to import a large number of CSV files into Elasticsearch. All the files have different content in them, with different headers. My goal is to have a directory we can drop CSVs into, and some plugin or process would pick them up, read the header and place the data in Elasticsearch mapping data to fields (gotten from the header). I've looked at the CSV River plugin and Fluentd. It looks like they both support half of the plan. The issue we run into is it looks like both want static field names before import. Am I wrong? Anyone's help would be wonderful. Thanks Ron -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 23, 2015, 10:23pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/2 "2015-01-23T22:23:19Z")
You could use Logstash for this as well, it has a CSV filter. On 24 January 2015 at 01:21, Ron [rulko313@gmail.com](mailto:rulko313@gmail.com) wrote: > Hello all, > > So I'm trying to import a large number of CSV files into Elasticsearch. > All the files have different content in them, with different headers. > > My goal is to have a directory we can drop CSVs into, and some plugin or > process would pick them up, read the header and place the data in > Elasticsearch mapping data to fields (gotten from the header). > > I've looked at the CSV River plugin and Fluentd. It looks like they both > support half of the plan. The issue we run into is it looks like both want > static field names before import. > > Am I wrong? Anyone's help would be wonderful. > > Thanks > > Ron > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-a7K-soUP\_800gF3bD5R1-gWutsYEE8ow6VSJB9QEaSA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-a7K-soUP_800gF3bD5R1-gWutsYEE8ow6VSJB9QEaSA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ron\_3](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Ron\_3](https://discuss.elastic.co/u/Ron_3)\ **Post date:** [January 23, 2015, 11:16pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/3 "2015-01-23T23:16:58Z")
So I also looked at logstash, but you have to define the field names for each file. Same as the others. Logstash is almost worse because you need a restart of the service when you update the config file. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/589d5cd6-063f-4f17-ada9-9cd9664a0e09%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/589d5cd6-063f-4f17-ada9-9cd9664a0e09%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ron\_3](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Ron\_3](https://discuss.elastic.co/u/Ron_3)\ **Post date:** [January 25, 2015, 2:40am UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/4 "2015-01-25T02:40:22Z")
The same issue exists with the Logstash plugin. You have to define the field names in the Logstash config file. Almost worse, each time you update field names, you have to restart logstash. On Friday, January 23, 2015 at 5:23:49 PM UTC-5, Mark Walkom wrote: > You could use Logstash for this as well, it has a CSV filter. > > On 24 January 2015 at 01:21, Ron \<[rulk...@gmail.com](mailto:rulk...@gmail.com) \\> wrote: > > > Hello all, > > > > So I'm trying to import a large number of CSV files into Elasticsearch. > > All the files have different content in them, with different headers. > > > > My goal is to have a directory we can drop CSVs into, and some plugin or > > process would pick them up, read the header and place the data in > > Elasticsearch mapping data to fields (gotten from the header). > > > > I've looked at the CSV River plugin and Fluentd. It looks like they both > > support half of the plan. The issue we run into is it looks like both want > > static field names before import. > > > > Am I wrong? Anyone's help would be wonderful. > > > > Thanks > > > > Ron > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 25, 2015, 5:25am UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/5 "2015-01-25T05:25:50Z")
I think you are looking for this? > > > CSV files are fairly common and almost always include a header line with name of… fields. > Right now logstash requires to: > \- somehow skip the header, by doing a messy match + drop > \- specify manually the columns to understand the rest of files. > > It would be really awesome if LS would just grab the field names from 1st line and then discard it, maybe controlled by a 'header-line' option. > I understand that it may not be easy to do if filters have no control over the input, and also when resuming a load it would have to read the header line 1st. > But it could be pretty powerful if the input could be controlled by the filter, or just add more input options, to generally handle the concept of header and feed it to the filter. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 25 janv. 2015 à 03:40, Ron [rulko313@gmail.com](mailto:rulko313@gmail.com) a écrit : > > The same issue exists with the Logstash plugin. You have to define the field names in the Logstash config file. > > Almost worse, each time you update field names, you have to restart logstash. > > > On Friday, January 23, 2015 at 5:23:49 PM UTC-5, Mark Walkom wrote: > > You could use Logstash for this as well, it has a CSV filter. > > > > > On 24 January 2015 at 01:21, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) wrote: > > > Hello all, > > > > > > So I'm trying to import a large number of CSV files into Elasticsearch. All the files have different content in them, with different headers. > > > > > > My goal is to have a directory we can drop CSVs into, and some plugin or process would pick them up, read the header and place the data in Elasticsearch mapping data to fields (gotten from the header). > > > > > > I've looked at the CSV River plugin and Fluentd. It looks like they both support half of the plan. The issue we run into is it looks like both want static field names before import. > > > > > > Am I wrong? Anyone's help would be wonderful. > > > > > > Thanks > > > > > > ## Ron > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/896B3DE1-123C-44AB-A6B2-2D1996F30B57%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/896B3DE1-123C-44AB-A6B2-2D1996F30B57%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ron\_3](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Ron\_3](https://discuss.elastic.co/u/Ron_3)\ **Post date:** [January 25, 2015, 6:55am UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/6 "2015-01-25T06:55:22Z")
Yup.. I'm one of the guys posting in that Issue log.. On Sunday, January 25, 2015 at 12:26:04 AM UTC-5, David Pilato wrote: > I think you are looking for this? > > [Make it easier to ingest CSV files · Issue #2088 · elastic/logstash · GitHub](https://github.com/elasticsearch/logstash/issues/2088) > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 25 janv. 2015 à 03:40, Ron \<[rulk...@gmail.com](mailto:rulk...@gmail.com) \\> a écrit : > > The same issue exists with the Logstash plugin. You have to define the > field names in the Logstash config file. > > Almost worse, each time you update field names, you have to restart > logstash. > > On Friday, January 23, 2015 at 5:23:49 PM UTC-5, Mark Walkom wrote: > > > You could use Logstash for this as well, it has a CSV filter. > > > > On 24 January 2015 at 01:21, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) wrote: > > > > > Hello all, > > > > > > So I'm trying to import a large number of CSV files into Elasticsearch. > > > All the files have different content in them, with different headers. > > > > > > My goal is to have a directory we can drop CSVs into, and some plugin or > > > process would pick them up, read the header and place the data in > > > Elasticsearch mapping data to fields (gotten from the header). > > > > > > I've looked at the CSV River plugin and Fluentd. It looks like they both > > > support half of the plan. The issue we run into is it looks like both want > > > static field names before import. > > > > > > Am I wrong? Anyone's help would be wonderful. > > > > > > Thanks > > > > > > Ron > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 25, 2015, 12:57pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/7 "2015-01-25T12:57:15Z")
I finally managed to modify the CsvJdbc driver from [http://csvjdbc.sourceforge.net/](http://csvjdbc.sourceforge.net/) so it behaves nicely and pushed it to > **[GitHub - jprante/jdbc-driver-csv: JDBC driver for CSV](https://github.com/jprante/jdbc-driver-csv)** > > JDBC driver for CSV. Contribute to jprante/jdbc-driver-csv development by creating an account on GitHub. So the news is, if you install JDBC river 1.4.0.9 onwards, a CSV driver is included. > **[GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch](https://github.com/jprante/elasticsearch-jdbc)** > > JDBC importer for Elasticsearch. Contribute to jprante/elasticsearch-jdbc development by creating an account on GitHub. To index CSV files from a directory without knowing the column names, just use curl -XPUT 'localhost:9200/\_river/my\_csv\_river/\_meta' -d ' { "type": "jdbc", "jdbc": { "url": "jdbc:xbib:csv:/Users/joerg/Desktop/csv", "sql": "select \* from airports", "index": "csv", "type": "csv" } }' In the URL, you specify a directory, in the SQL statement table name, you select a file from within the directory (without .csv suffix, here: airports.csv) Jörg On Sun, Jan 25, 2015 at 7:55 AM, Ron [rulko313@gmail.com](mailto:rulko313@gmail.com) wrote: > Yup.. I'm one of the guys posting in that Issue log.. > > On Sunday, January 25, 2015 at 12:26:04 AM UTC-5, David Pilato wrote: > > > I think you are looking for this? > > > > [Make it easier to ingest CSV files · Issue #2088 · elastic/logstash · GitHub](https://github.com/elasticsearch/logstash/issues/2088) > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 25 janv. 2015 à 03:40, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) a écrit : > > > > The same issue exists with the Logstash plugin. You have to define the > > field names in the Logstash config file. > > > > Almost worse, each time you update field names, you have to restart > > logstash. > > > > On Friday, January 23, 2015 at 5:23:49 PM UTC-5, Mark Walkom wrote: > > > > > You could use Logstash for this as well, it has a CSV filter. > > > > > > On 24 January 2015 at 01:21, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) wrote: > > > > > > > Hello all, > > > > > > > > So I'm trying to import a large number of CSV files into Elasticsearch. > > > > All the files have different content in them, with different headers. > > > > > > > > My goal is to have a directory we can drop CSVs into, and some plugin > > > > or process would pick them up, read the header and place the data in > > > > Elasticsearch mapping data to fields (gotten from the header). > > > > > > > > I've looked at the CSV River plugin and Fluentd. It looks like they > > > > both support half of the plan. The issue we run into is it looks like both > > > > want static field names before import. > > > > > > > > Am I wrong? Anyone's help would be wonderful. > > > > > > > > Thanks > > > > > > > > Ron > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoETGaDFHbZX3ncmGi\_-sFAVCwpYOjoWkkX%3DCjrLkoXb5A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoETGaDFHbZX3ncmGi_-sFAVCwpYOjoWkkX%3DCjrLkoXb5A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ron\_3](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Ron\_3](https://discuss.elastic.co/u/Ron_3)\ **Post date:** [January 26, 2015, 12:13pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/8 "2015-01-26T12:13:49Z")
Looks solid, I'll give it a run today. On Sunday, January 25, 2015 at 7:57:29 AM UTC-5, Jörg Prante wrote: > I finally managed to modify the CsvJdbc driver from > [http://csvjdbc.sourceforge.net/](http://csvjdbc.sourceforge.net/) so it behaves nicely and pushed it to > > [GitHub - jprante/jdbc-driver-csv: JDBC driver for CSV](https://github.com/jprante/jdbc-driver-csv) > > So the news is, if you install JDBC river 1.4.0.9 onwards, a CSV driver is > included. > > [GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch](https://github.com/jprante/elasticsearch-river-jdbc) > > To index CSV files from a directory without knowing the column names, just > use > > curl -XPUT 'localhost:9200/\_river/my\_csv\_river/\_meta' -d ' > { > "type": "jdbc", > "jdbc": { > "url": "jdbc:xbib:csv:/Users/joerg/Desktop/csv", > "sql": "select \* from airports", > "index": "csv", > "type": "csv" > } > }' > > In the URL, you specify a directory, in the SQL statement table name, you > select a file from within the directory (without .csv suffix, here: > airports.csv) > > Jörg > > On Sun, Jan 25, 2015 at 7:55 AM, Ron \<[rulk...@gmail.com](mailto:rulk...@gmail.com) \\> > wrote: > > > Yup.. I'm one of the guys posting in that Issue log.. > > > > On Sunday, January 25, 2015 at 12:26:04 AM UTC-5, David Pilato wrote: > > > > > I think you are looking for this? > > > > > > [Make it easier to ingest CSV files · Issue #2088 · elastic/logstash · GitHub](https://github.com/elasticsearch/logstash/issues/2088) > > > > > > -- > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 25 janv. 2015 à 03:40, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) a écrit : > > > > > > The same issue exists with the Logstash plugin. You have to define the > > > field names in the Logstash config file. > > > > > > Almost worse, each time you update field names, you have to restart > > > logstash. > > > > > > On Friday, January 23, 2015 at 5:23:49 PM UTC-5, Mark Walkom wrote: > > > > > > > You could use Logstash for this as well, it has a CSV filter. > > > > > > > > On 24 January 2015 at 01:21, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) wrote: > > > > > > > > > Hello all, > > > > > > > > > > So I'm trying to import a large number of CSV files into > > > > > Elasticsearch. All the files have different content in them, with different > > > > > headers. > > > > > > > > > > My goal is to have a directory we can drop CSVs into, and some plugin > > > > > or process would pick them up, read the header and place the data in > > > > > Elasticsearch mapping data to fields (gotten from the header). > > > > > > > > > > I've looked at the CSV River plugin and Fluentd. It looks like they > > > > > both support half of the plan. The issue we run into is it looks like both > > > > > want static field names before import. > > > > > > > > > > Am I wrong? Anyone's help would be wonderful. > > > > > > > > > > Thanks > > > > > > > > > > Ron > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86c136d2-006b-4eaf-83ef-7ce3e1dfc32f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86c136d2-006b-4eaf-83ef-7ce3e1dfc32f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ron\_3](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Ron\_3](https://discuss.elastic.co/u/Ron_3)\ **Post date:** [January 28, 2015, 7:01pm UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/9 "2015-01-28T19:01:34Z")
Not having a lot of luck here Installed the plugin based on Github, and tried the request. I'm getting a fail.. curl -XPUT 'localhost:9200/\_river/my\_csv\_river/\_meta' -d ' { "type": "jdbc", "jdbc": { "url": "jdbc:xbib:csv:/home/user", "sql": "select \* from garages", "index": "csv", "type": "csv" } }' {"\_index":"\_river","\_type":"my\_csv\_river","\_id":"\_meta","\_version":4,"created":false} On Sun Jan 25 2015 at 7:57:29 AM [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I finally managed to modify the CsvJdbc driver from > [http://csvjdbc.sourceforge.net/](http://csvjdbc.sourceforge.net/) so it behaves nicely and pushed it to > > [GitHub - jprante/jdbc-driver-csv: JDBC driver for CSV](https://github.com/jprante/jdbc-driver-csv) > > So the news is, if you install JDBC river 1.4.0.9 onwards, a CSV driver is > included. > > [GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch](https://github.com/jprante/elasticsearch-river-jdbc) > > To index CSV files from a directory without knowing the column names, just > use > > curl -XPUT 'localhost:9200/\_river/my\_csv\_river/\_meta' -d ' > { > "type": "jdbc", > "jdbc": { > "url": "jdbc:xbib:csv:/Users/joerg/Desktop/csv", > "sql": "select \* from airports", > "index": "csv", > "type": "csv" > } > }' > > In the URL, you specify a directory, in the SQL statement table name, you > select a file from within the directory (without .csv suffix, here: > airports.csv) > > Jörg > > On Sun, Jan 25, 2015 at 7:55 AM, Ron [rulko313@gmail.com](mailto:rulko313@gmail.com) wrote: > > > Yup.. I'm one of the guys posting in that Issue log.. > > > > On Sunday, January 25, 2015 at 12:26:04 AM UTC-5, David Pilato wrote: > > > > > I think you are looking for this? > > > > > > [Make it easier to ingest CSV files · Issue #2088 · elastic/logstash · GitHub](https://github.com/elasticsearch/logstash/issues/2088) > > > > > > -- > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 25 janv. 2015 à 03:40, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) a écrit : > > > > > > The same issue exists with the Logstash plugin. You have to define the > > > field names in the Logstash config file. > > > > > > Almost worse, each time you update field names, you have to restart > > > logstash. > > > > > > On Friday, January 23, 2015 at 5:23:49 PM UTC-5, Mark Walkom wrote: > > > > > > > You could use Logstash for this as well, it has a CSV filter. > > > > > > > > On 24 January 2015 at 01:21, Ron [rulk...@gmail.com](mailto:rulk...@gmail.com) wrote: > > > > > > > > > Hello all, > > > > > > > > > > So I'm trying to import a large number of CSV files into > > > > > Elasticsearch. All the files have different content in them, with different > > > > > headers. > > > > > > > > > > My goal is to have a directory we can drop CSVs into, and some plugin > > > > > or process would pick them up, read the header and place the data in > > > > > Elasticsearch mapping data to fields (gotten from the header). > > > > > > > > > > I've looked at the CSV River plugin and Fluentd. It looks like they > > > > > both support half of the plan. The issue we run into is it looks like both > > > > > want static field names before import. > > > > > > > > > > Am I wrong? Anyone's help would be wonderful. > > > > > > > > > > Thanks > > > > > > > > > > Ron > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/54f555d9-1d66-4b19-9334-90dde4fc658a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/620537a8-7bf4-4acf-b5ea-46c2d24c3812%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e5f58365-65e7-45b6-a1ce-b86f55d655df%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/wDathej\_ySQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/wDathej_ySQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoETGaDFHbZX3ncmGi\_-sFAVCwpYOjoWkkX%3DCjrLkoXb5A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoETGaDFHbZX3ncmGi_-sFAVCwpYOjoWkkX%3DCjrLkoXb5A%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoETGaDFHbZX3ncmGi\_-sFAVCwpYOjoWkkX%3DCjrLkoXb5A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoETGaDFHbZX3ncmGi_-sFAVCwpYOjoWkkX%3DCjrLkoXb5A%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKKgDVEeMa%3Dp9QyCtjuNN%3Dzx528cPHtP2yP%2BKb6hmYdqj3f43A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKKgDVEeMa%3Dp9QyCtjuNN%3Dzx528cPHtP2yP%2BKb6hmYdqj3f43A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:36am UTC](https://discuss.elastic.co/t/bulk-importing-csvs-with-different-headers/21799/10 "2017-07-06T00:36:06Z")