# Logstash jdbc document\_id =\> "%{uid}" problem **URL:** \ **Category:** Logstash\ **Created:** [November 20, 2015, 5:44pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174 "2015-11-20T17:44:59Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [November 20, 2015, 5:44pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/1 "2015-11-20T17:44:59Z")
Hi Logstash newbie here 😄 Using logstash 2.0.0 and elastic 2.0.0 I'm trying to follow [https://www.elastic.co/blog/logstash-jdbc-input-plugin](https://www.elastic.co/blog/logstash-jdbc-input-plugin) and MySQL driver. I'm setting in config ```auto ``` The data is being read ok-ish, (got multiple records returned) however on Elastic side I'm getting only one document indexed. In place of usual `_id` it's geting `_id:%{uid}` ![](https://us1.discourse-cdn.com/elastic/original/2X/8/820ee917ca91ba964640b6a99e2bd653addc479c.png) If I don't use `document_id => "%{uid}"` I'm getting multiple documents indexed reflecting the SQL query output. However I wanted to avoid data duplication. Any ideas? Marcin ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 21, 2015, 5:54am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/2 "2015-11-21T05:54:16Z")
Can you provide your entire config? ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [November 21, 2015, 8:09pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/3 "2015-11-21T20:09:09Z")
Hi Mark, thanks for asking. ```auto input { jdbc { jdbc_driver_library => "/opt/logstash-2.0.0/vendor/mysql/mysql-connector-java-5.1.37-bin.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_connection_string => "jdbc:mysql:// ***:*** /***" jdbc_user => "***" jdbc_password => "***" statement_filepath => "/opt/logstash-2.0.0/sql_queries/query3.sql" } } output { # stdout { codec=>rubydebug } elasticsearch { index => " ****" document_type => " ****" document_id => "%{uid}" hosts => "localhost:9200" } } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 21, 2015, 9:44pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/4 "2015-11-21T21:44:27Z")
When you enable the stdout output, do you have a field named 'uid' that the document\_id can be built from? ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [November 21, 2015, 10:30pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/5 "2015-11-21T22:30:32Z")
Good question Christian! In the result I don't have any ID field actually. However in the example from the URL the result of SQL query also has no 'uid' thing. It's there seen in json output though. And there you go - I've modified my sql query to return an id and used `document_id => "%{id}"` and this populated the row IDs into `_id` field. I'm guessing it's possible to create a hash of returned values and have it as `document_id`? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 22, 2015, 9:20am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/6 "2015-11-22T09:20:16Z")
You can use the fingerprint or checksum plugins to create a hash based on specific field(s) and then use this to set document\_id. If you have one or more fields in your data that make up a unique key, you may however be better off concatenating these into a key without hashing it. ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [November 22, 2015, 7:10pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/7 "2015-11-22T19:10:33Z")
Thanks! ---
**Author:** ![Architha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/architha/32/92477_2.png) [@Architha](https://discuss.elastic.co/u/Architha)\ **Post date:** [January 8, 2016, 10:49am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/8 "2016-01-08T10:49:29Z")
Hi , I have been facing the same issue. Could you tell me how you have solved it ? Thanks ! ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [January 18, 2016, 7:05am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/9 "2016-01-18T07:05:43Z")
@Architha From your SQL query you need to return one column with unique ID - a number (or maybe a hash). Lets say, a name of that field will be `my_ID` Then in your `output` section of logstash config you add (under `elasticsearch { }`) ``` document_id => "%{my_ID}" ``` And that's all. Hope this helps! ---
**Author:** ![Architha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/architha/32/92477_2.png) [@Architha](https://discuss.elastic.co/u/Architha)\ **Post date:** [January 20, 2016, 11:23am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/11 "2016-01-20T11:23:43Z")
@Marcin_Kubica Is this possible with an already existing column with unique ID ? Supposed i have an existing column `Employee_id` and config in the same way as mentioned (under `elasticsearch{ }` in Output filter) > document\_id =\> "%{Employee\_id}" I still see `"_id" : "%{Employee_id}"` in the search results. Kindly help. I am not very clear with "From your SQL query you need to return one column with unique ID - a number (or maybe a hash)." Please elaborate 🙂 Thanks a lot ! ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [January 20, 2016, 8:41pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/12 "2016-01-20T20:41:09Z")
Hi is Employee\_id (number) returned ( displayed ) as result of your SQL select statement? I take it is not 😉 Had exactly this problem. Make sure to include it in SELECT statement. Marcin ---
**Author:** ![Architha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/architha/32/92477_2.png) [@Architha](https://discuss.elastic.co/u/Architha)\ **Post date:** [January 21, 2016, 5:59am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/13 "2016-01-21T05:59:19Z")
Hi Marcin , The problem was solved when i used `"employee_id"` instead of `"Employee_id"` in the output section of Logstash conf file. It is case sensitive. Thanks for your reply 🙂 Cheers !! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:14am UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-uid-problem/35174/14 "2017-07-06T05:14:56Z")