# Sending Data from MySql to ES **URL:** \ **Category:** Logstash\ **Created:** [May 14, 2020, 10:10am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609 "2020-05-14T10:10:05Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jasmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasmin/32/56586_2.png) [@Jasmin](https://discuss.elastic.co/u/Jasmin)\ **Post date:** [May 14, 2020, 10:10am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/1 "2020-05-14T10:10:05Z")
Hi , i am sending 447 documents from logstash but ES is recieving only 43 , my config file : input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://ipaddress:3306/zendb" jdbc\_user =\> "\*\*\*\*" jdbc\_password =\> "\*\*\*\*\*" jdbc\_driver\_library =\> "/root/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" statement =\> "SELECT \* FROM zen\_orders WHERE current\_state = 3" } } output { elasticsearch { "hosts" =\> "localhost:9200" "index" =\> "zen" "document\_id" =\> "%{id\_order}" } stdout { codec =\> json\_lines } } ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [May 14, 2020, 11:10am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/2 "2020-05-14T11:10:23Z")
> [@Jasmin](#): > > i am sending 447 documents from logstash but ES is recieving only 43 , You are setting the `"document_id" => "%{id_order}"` in your output plugin. If there are duplicate values for that field in your database then Elasticsearch will update those documents instead of creating duplicates at the time of indexing. Try to run a `COUNT DISTINCT` on that field in your database and see what number it shows up. If you want all those 447 documents indexed, then remove the document\_id setting and let Elasticsearch dynamically generate that for you. ---
**Author:** ![Jasmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasmin/32/56586_2.png) [@Jasmin](https://discuss.elastic.co/u/Jasmin)\ **Post date:** [May 14, 2020, 11:51am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/3 "2020-05-14T11:51:10Z")
COUNT DISCNIT give me 447 , so it's not a problem of duplication ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [May 14, 2020, 12:22pm UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/4 "2020-05-14T12:22:33Z")
if you set the output to files, how many documents are created? ---
**Author:** ![Jasmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasmin/32/56586_2.png) [@Jasmin](https://discuss.elastic.co/u/Jasmin)\ **Post date:** [May 14, 2020, 12:44pm UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/5 "2020-05-14T12:44:11Z")
even when output is a file : 43 document ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [May 14, 2020, 5:21pm UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/6 "2020-05-14T17:21:06Z")
then you’re not sending 447 documents to ES, you only sent 43 🙂 you’re not seeing any jdbc error in the log? if your jdbc input produces different result compared to executing sql statement to the db directly, i would think that’s it’s either the problem in the jdbc input plugin or the jdbc library. ---
**Author:** ![Jasmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasmin/32/56586_2.png) [@Jasmin](https://discuss.elastic.co/u/Jasmin)\ **Post date:** [May 15, 2020, 8:05am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/7 "2020-05-15T08:05:54Z")
debug mode give me this WARN : [2020-05-15T08:52:44,754][WARN][logstash.inputs.jdbc][main] Exception when executing JDBC query {:exception=\>#\ 3\>} [2020-05-15T08:52:45,028][DEBUG][logstash.javapipeline][main] Input plugins stopped! Will shutdown filter/output workers. {:pipeline\_id=\>"main", :thread=\>"#\"} [2020-05-15T08:52:45,043][DEBUG][logstash.javapipeline][main] Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"} [2020-05-15T08:52:45,120][DEBUG][logstash.javapipeline][main] Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"} ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [May 15, 2020, 11:08am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/8 "2020-05-15T11:08:11Z")
> [@Jasmin](#): > > 2020-05-15T08:52:44,754][WARN][logstash.inputs.jdbc][main] Exception when executing JDBC query {:exception=\>#\ 3\>} > [2020-05-15T08:52:45,028][DEBUG][logstash.javapipeline][main] Input plugins stopped! Will shutdown filter/output you had sql error, that’s why the record returned is less than expected. you need to fix sql statement. if the statement is working fine when you connect directly to the database, you probably hit a jdbc driver bug. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 12, 2020, 11:08am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609/9 "2020-06-12T11:08:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.