# Pipeline is being aborted due to logstash\_jdbc\_last\_run File Not Found **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [July 6, 2020, 3:25pm UTC](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029 "2020-07-06T15:25:59Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![anurag397](https://avatars.discourse-cdn.com/v4/letter/a/df788c/32.png) [@anurag397](https://discuss.elastic.co/u/anurag397)\ **Post date:** [July 6, 2020, 3:25pm UTC](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029/1 "2020-07-06T15:25:59Z")
I have two pipelines which sync data from different tables in a postgresql database. ``` - pipeline.id: logstash_test_user_table path.config: "/usr/share/logstash/pipeline/logstash.conf" pipeline.workers: 1 - pipeline.id: logstash_test_addresses_table path.config: "/usr/share/logstash/pipeline/addresses.conf" pipeline.workers: 1 ``` The first pipeline works fine. Here is the pipeline: ``` input { jdbc { jdbc_driver_library => "/usr/share/logstash/plugins/jdbc/postgresql-42.2.14.jar" jdbc_driver_class => "org.postgresql.Driver" jdbc_connection_string => "jdbc:postgresql://localhost:5432/db_name" jdbc_user => "user" jdbc_password => "pass" jdbc_paging_enabled => true tracking_column => "unix_ts_in_secs" use_column_value => true clean_run => true tracking_column_type => "numeric" schedule => "*/20 * * * * *" statement => "SELECT *, EXTRACT(epoch from modification_time) AS unix_ts_in_secs FROM users WHERE (EXTRACT(epoch from modification_time) > :sql_last_value AND modification_time < NOW()) ORDER BY modification_time ASC" } } filter { mutate { copy => { "id" => "[@metadata][_id]"} remove_field => ["id", "@version", "unix_ts_in_secs"] } } output { # stdout { codec => "rubydebug"} elasticsearch { hosts => ["localhost:9200"] index => "rdbms_sync_idx" document_id => "%{[@metadata][_id]}" } } ``` The second one however gives me an error: `14:54:39.207 [[logstash_test_addresses_table]-pipeline-manager] ERROR logstash.javapipeline - Pipeline aborted due to error {:pipeline_id=>"logstash_test_addresses_table", :exception=>#, :backtrace=>["org/jruby/RubyFile.java:1285:in `delete'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:112:in `clean'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:13:in `build_last_value_tracker'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/inputs/jdbc.rb:240:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:326:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:286:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/addresses.conf"], :thread=>"#"}` Second pipeline: ``` input { jdbc { jdbc_driver_library => "/usr/share/logstash/plugins/jdbc/postgresql-42.2.14.jar" jdbc_driver_class => "org.postgresql.Driver" jdbc_connection_string => "jdbc:postgresql://localhost:5432/db_name" jdbc_user => "user" jdbc_password => "pass" jdbc_paging_enabled => true tracking_column => "id" use_column_value => true clean_run => true tracking_column_type => "numeric" schedule => "*/20 * * * * *" statement => "SELECT * FROM addresses WHERE id > :sql_last_value" } } filter { mutate { copy => { "id" => "[@metadata][_id]"} remove_field => ["id", "@version"] } } output { # stdout { codec => "rubydebug"} elasticsearch { hosts => ["localhost:9200"] index => "rdbms_sync_idx2" document_id => "%{[@metadata][_id]}" } } ``` Please help me out I'm unable to understand what this error is. I assume the first pipeline is setting the metadata and second one is trying to read the same metadata. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 6, 2020, 11:14pm UTC](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029/2 "2020-07-06T23:14:39Z")
Both pipelines have jdbc inputs that are persisting metadata. Since you have not set the last\_run\_metadata\_path option on either input there will be a conflict, since both will try to use "#{ENV['HOME']}/.logstash\_jdbc\_last\_run". Set last\_run\_metadata\_path on one or both inputs. ---
**Author:** ![anurag397](https://avatars.discourse-cdn.com/v4/letter/a/df788c/32.png) [@anurag397](https://discuss.elastic.co/u/anurag397)\ **Post date:** [July 7, 2020, 11:27am UTC](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029/3 "2020-07-07T11:27:52Z")
Thanks a lot! I set last\_run\_metadata\_path on both inputs and now both pipelines are able to run simultaneously. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 4, 2020, 11:27am UTC](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029/4 "2020-08-04T11:27:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.