# JDBC input multiple tables with table name as a parameter

**URL:** <https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970>\
**Category:** Logstash\
**Created:** [September 27, 2017, 11:17am UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970 "2017-09-27T11:17:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maxclac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxclac/32/29107_2.png) [@Maxclac](https://discuss.elastic.co/u/Maxclac)\
**Post date:** [September 27, 2017, 11:17am UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970/1 "2017-09-27T11:17:04Z")

</div>

Hello everyone!

I am working currently with a Postgres database and trying to convert it to a json file for Elasticsearch.  
I found out that Logstash is perfect for doing this with the JDBC input plugin.  
I have several tables in my DB and I wanted to know how to process them all at the same time.

To be more specfic, I have my .conf file like this:

```
input {
    jdbc {
        # Postgres jdbc connection string to our database, db
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/db"
        # The user we wish to execute our statement as
        jdbc_user => "postgres"
        #password
        jdbc_password => "postgres"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "blabla.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "org.postgresql.Driver"
        # our query
        statement => "SELECT * FROM some_table LIMIT 10;"
    }
}

```

but the thing is that I have 40 tables. Obviously, I am not going to have 40 conf files for every table of my database.  
What would be a smart way to do this? Could I pass the table name as a parameter in the statement?

I was thinking of using this statement:

```
SELECT TABLE_NAME
FROM information_schema.tables
WHERE table_type = 'BASE TABLE'
  AND table_schema NOT IN ('pg_catalog','information_schema')

```

to get all the table names of my database and use them to create other statements of the form

SELECT \* FROM some\_table LIMIT 10

where 'some\_table' would be a parameter.

Any idea on how I could achieve this with logstash?

Thank you in advance for your attention!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 27, 2017, 1:41pm UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970/2 "2017-09-27T13:41:05Z")

</div>

Why not use a script to generate the necessary configuration blocks?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 27, 2017, 1:53pm UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970/3 "2017-09-27T13:53:35Z")

</div>

Before you transfer table by table into Elasticsearch, think about how you want to be able to query the data. As Elasticsearch does not support joins, it may make sense to denormalise or restructure some of the data before sending it to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![Maxclac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxclac/32/29107_2.png) [@Maxclac](https://discuss.elastic.co/u/Maxclac)\
**Post date:** [September 27, 2017, 1:55pm UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970/4 "2017-09-27T13:55:18Z")

</div>

Yes, that could be an idea, but I was wondering if I could use some features if the plugin to do this.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 27, 2017, 2:03pm UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970/5 "2017-09-27T14:03:27Z")

</div>

That depends on your data and data model, so I don't think there is any way to automate that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2017, 2:03pm UTC](https://discuss.elastic.co/t/jdbc-input-multiple-tables-with-table-name-as-a-parameter/101970/6 "2017-10-25T14:03:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.