# Logstash - MariaDB to ElasticSearch - No Error, but doesnt load **URL:** \ **Category:** Logstash\ **Created:** [July 31, 2017, 6:50am UTC](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127 "2017-07-31T06:50:00Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![pawanyv](https://avatars.discourse-cdn.com/v4/letter/p/b5e925/32.png) [@pawanyv](https://discuss.elastic.co/u/pawanyv)\ **Post date:** [July 31, 2017, 6:50am UTC](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127/1 "2017-07-31T06:50:00Z")
Hi, I am trying to load data from MariaDB to elasticsearch using JDBC input plugin and elasticsearch output plugin. both logstash and elasticsearch versions are 5.5.0 My logstash.conf file is: input { jdbc { jdbc\_driver\_library =\> "mysql-connector-java-5.1.43-bin.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/exampledb" jdbc\_user =\> "**" parameters =\> { "source" =\> "example" } jdbc\_password =\> "**_" jdbc\_fetch\_size =\> "5000" schedule =\> "_ \* \* \* \*" statement =\> "SELECT \* from schedules" } } output { stdout { codec =\> json\_lines } elasticsearch { hosts =\> "localhost:9200" index =\> "/schedules" } } When i run logstash with the above config file, i can see the data fetched from MariaDB, but after few seconds or a minute i see the data is refetched from MARIADB and it never goes to elasticsearch. I have been scratching on this issue since very log, but unable to crack. Output log file: root@abt-dev:/logstash/logstash-5.5.0/bin# ./logstash -f logstash.conf ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Sending Logstash's logs to /logstash/logstash-5.5.0/logs which is now configured via log4j2.properties [2017-07-31T11:41:05,606][INFO][logstash.setting.writabledirectory] Creating directory {:setting=\>"path.queue", :path=\>"/logstash/logstash-5.5.0/data/queue"} [2017-07-31T11:41:05,609][INFO][logstash.setting.writabledirectory] Creating directory {:setting=\>"path.dead\_letter\_queue", :path=\>"/logstash/logstash-5.5.0/data/dead\_letter\_queue"} [2017-07-31T11:41:05,627][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=\>"3dd9ad06-c15e-4a9f-bd58-15969d4534bd", :path=\>"/logstash/logstash-5.5.0/data/uuid"} [2017-07-31T11:41:06,078][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2017-07-31T11:41:06,083][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"} [2017-07-31T11:41:06,170][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>#Java::JavaNet::URI:0x531660e2} [2017-07-31T11:41:06,171][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} [2017-07-31T11:41:06,226][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2017-07-31T11:41:06,232][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>[#Java::JavaNet::URI:0x7b7cbd95]} [2017-07-31T11:41:06,234][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>250} [2017-07-31T11:41:06,341][INFO][logstash.pipeline] Pipeline main started [2017-07-31T11:41:06,406][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2017-07-31T11:42:00,824][INFO][logstash.inputs.jdbc] (0.010000s) SELECT \* from schedules {"enddate":"2017-07-20T11:30:38.000Z","@timestamp":"2017-07-31T06:12:00.846Z","logfile":" \nJob name: Full\_Data\_Load\nLogging started on: 20-07-2017 17:00:36\n \nMariaDB connection successfully established - function: dbconnect\n \nSuccessfully extracted DATA TIME from head of TXT file\n \nSuccessfully extracted Total no of records from tail of TXT file - function: gettail\n \nSuccessfully inserted job details in the database - function: insertdb\n \nSuccessfully extracted relevant lines from source file - function pulllines\n \nNo of records from data source: 370\nNo of inserts in the database: 370\nSuccessfully completed data load in to the database. Updating Schedule in the database.\n","datatime":"07/19/2017 18:51:14","@version":"1","id":12,"source":"ed2go","startdate":"2017-07-20T11:30:36.000Z","script\_name":"Full\_Data\_Load","status":"SUCCESS"} {"enddate":"2017-07-20T11:31:40.000Z","@timestamp":"2017-07-31T06:12:00.848Z","logfile":" \nJob name: Delta\_Data\_Load\nLogging started on: 20-07-2017 17:01:38\n \nMariaDB connection successfully established - function: dbconnect\n \nSuccessfully extracted DATA TIME from head of TXT file\n \nSuccessfully extracted Total no of records from tail of TXT file - function: gettail\n \nSuccessfully inserted job details in the database - function: insertdb\n \nSuccessfully extracted relevant lines from source file - function pulllines\n \nSuccessfully completed splitting lines into INSERT, UPDATE and DELETE lists - function: splitdmls\n \nNo of records from data source: 89\nNo of inserts in the database: 0\nNo of updates in the database: 89\nNo of deletes in the database: 0\nSuccessfully completed data load in to the database. Updating job schedule in the database.\n","datatime":"07/19/2017 18:51:14","@version":"1","id":13,"source":"ed2go","startdate":"2017-07-20T11:31:38.000Z","script\_name":"Delta\_Data\_Load","status":"SUCCESS"} And the same select output repeats on the screen. ---
**Author:** ![pawanyv](https://avatars.discourse-cdn.com/v4/letter/p/b5e925/32.png) [@pawanyv](https://discuss.elastic.co/u/pawanyv)\ **Post date:** [July 31, 2017, 7:07am UTC](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127/2 "2017-07-31T07:07:56Z")
I have changed the following: From: index =\> “/schedules” To: index =\> “schedules” it seems to be working now. But, after the job, logstash doesnt stop. It tries to re-run the same job again. Please advise. ---
**Author:** ![pawanyv](https://avatars.discourse-cdn.com/v4/letter/p/b5e925/32.png) [@pawanyv](https://discuss.elastic.co/u/pawanyv)\ **Post date:** [July 31, 2017, 9:07am UTC](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127/3 "2017-07-31T09:07:26Z")
I found my solution. I have configured this to run for every 24 hours. SEems to be ok for now. Cheers !!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2017, 9:07am UTC](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127/4 "2017-08-28T09:07:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.