# Logstash 6.2.2 with JDBC data on not updated **URL:** \ **Category:** Logstash\ **Created:** [April 5, 2018, 2:28pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949 "2018-04-05T14:28:40Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ongun](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@Ongun](https://discuss.elastic.co/u/Ongun)\ **Post date:** [April 5, 2018, 2:28pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/1 "2018-04-05T14:28:40Z")
Hello all, I have an **ELK Stack 6.2.2** with X-pack trial license. I am running a multiple pipeline configuration with JDBC drivers and sometimes I notice that my data on Elasticsearch is not updated (particularly when multiple pipelines are run simultaneously). I notice this through `@timestamp` field in the discover portion of Kibana. Currently I have **7 pipelines** that I want to run simultaneously and they are **all scheduled** to run queries against the database in intervals of 5 seconds. What do you suggest for my situation? Are intervals of 5 seconds too tight and possibly overwhelm the SQL Server? I wonder if somehow the scheduler can be adjusted to wait for the previous pipelines to finish before executing the next scheduled queries. Thanks in advance My `pipelines.yml` file is as follows: ``` ## Never give the full path on Windows - pipeline.id: Training path.config: "./pipelines/sql_tables_Training.conf" - pipeline.id: YKBÜ path.config: "./pipelines/sql_tables_YKBÜ.conf" - pipeline.id: YKBÜ Session path.config: "./pipelines/sql_tables_YKBÜ_session.conf" - pipeline.id: Training Processes path.config: "./pipelines/sql_Training_processes.conf" - pipeline.id: YKBÜ Processes path.config: "./pipelines/sql_YKBÜ_processes.conf" - pipeline.id: Training Queues path.config: "/Users/ongun.arisev/Downloads/ElasticStack/logstash-6.2.2/pipelines/sql_Training_queues.conf" - pipeline.id: YKBÜ Queues path.config: "./pipelines/sql_YKBÜ_queues.conf" ``` Each configuration pipeline file is similar and I post here one of them as an example: ``` input { jdbc { jdbc_driver_library => "C:\Program Files\sqljdbc_6.0\enu\jre8\sqljdbc42.jar" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_connection_string => "jdbc:sqlserver://;integratedSecurity=true" schedule => "*/5 * * * * *" jdbc_user => "ongun.arisev" #statement_filepath => "C:\Users\ongun.arisev\Downloads\ElasticStack\logstash-6.2.2\sqlprocedure.txt" statement => " EXEC [YKBÜ].[dbo].[BPDS_QueueVolumesNow];" tags => "YKBÜ_queues" } } filter { mutate { add_field => { "Veritabani" => "YKBÜ" } } } output { # stdout { codec => "rubydebug" } elasticsearch{ hosts => ["localhost:9200"] index => "sql_queues_ykb" document_id => "%{name}" user => "logstash_internal" password => "x-pack-test-password" } } ``` ---
**Author:** ![PatmanAA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patmanaa/32/29642_2.png) [@PatmanAA](https://discuss.elastic.co/u/PatmanAA)\ **Post date:** [April 5, 2018, 6:48pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/2 "2018-04-05T18:48:54Z")
@Ongun I'm really not an expert but maybe is as something to do with pipeline.workers: [https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) Best regard's pat ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 5, 2018, 8:21pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/3 "2018-04-05T20:21:20Z")
Its hard to understand what you are trying to achieve. The sql statements give no clues. What does the data look like? How does it get updated? How many rows per second are inserted? Unless you are planning to always ingest all records each time, you need to add some form of progress tracking i.e. `WHERE some_id > :sql_last_value`. To be honest, I have never used the jdbc input in a multiple pipeline scenario. ---
**Author:** ![Ongun](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@Ongun](https://discuss.elastic.co/u/Ongun)\ **Post date:** [April 5, 2018, 8:43pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/4 "2018-04-05T20:43:43Z")
The SQL statement (actually a stored procedure) returns a table consisting of like at most 30 rows with 5 columns. Column number of the SQL query result is always the same and row number also mostly does not change. The only dynamic part then are the values in the table, their change rate depends. But I can say that it varies between 2 seconds to 2 minutes. So this is not a very loaded table. However, when used together with multiple pipelines sometimes data is not updated. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 5, 2018, 10:49pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/5 "2018-04-05T22:49:48Z")
I suppose you are trying to "mirror" the state (as it changes) into ES buy overwriting the ES document with an index per table. Correct? If so have you considered [CDC](https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server). > **[Streaming ETL: SQL Change Data Capture (CDC) to Azure Event Hub](https://mrfoxsql.wordpress.com/2017/07/12/streaming-etl-send-sql-change-data-capture-cdc-to-azure-event-hub/)** > > I had a recent requirement to capture and stream real-time data changes on several SQL database tables from an on-prem SQL Server to Azure for downstream processing. Specifically we needed to creat… I guess I am saying that it seems to me that you need a more reliable async way of capturing these changes. ---
**Author:** ![Ongun](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@Ongun](https://discuss.elastic.co/u/Ongun)\ **Post date:** [April 6, 2018, 7:54am UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/6 "2018-04-06T07:54:02Z")
Exactly I overwrite the table each time Logstash runs, I have not considered CDC. Thanks for the suggestion. I was also considering not overwriting but recording the tables together with their timestamps to have time series data. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2018, 7:54am UTC](https://discuss.elastic.co/t/logstash-6-2-2-with-jdbc-data-on-not-updated/126949/7 "2018-05-04T07:54:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.