JDBC input error when using schedule without last run

Elastic Stack Logstash
1

Hi, I need to query a database every 1 minute and get all the results of the query, so I use schedule but no last_run_metadata_path.

logstash give an error, but still que the data indexed in ES.

logstash look for this file /usr/share/logstash/.logstash_jdbc_last_run

when I run the command ll -a /usr/share/logstash/ to see hidden files the file is not there

altough the pipeline works, the error in the log bothers a lot

What can I do to avoid this error message?

{
    :exception=>Errno: :EACCES,
    :message=>"Permission denied - /usr/share/logstash/.logstash_jdbc_last_run",
    :backtrace=>[
        "org/jruby/RubyIO.java:1237:in `sysopen'",
        "org/jruby/RubyIO.java:3800:in `write'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:122:in `write'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:46:in `write'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/inputs/jdbc.rb:354:in `execute_query'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/inputs/jdbc.rb:308:in `block in run'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block in start_work_thread'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:in `block in start_work_thread'",
        "org/jruby/RubyKernel.java:1442:in `loop'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'"
    ],
    :now=>"2023-06-05T11:26:01.045",
    :last_time=>"2023-06-05T11:26:00.807",
    :next_time=>"2023-06-05T11:27:00.000",
    :job=>#<Rufus: :Scheduler: :CronJob: 0xd11274b @last_at=nil, @tags=[], @scheduled_at=2023-06-05 10: 44: 42 -0400, @cron_line=#<Rufus: :Scheduler: :CronLine: 0x2576f21b @timezone=nil, @weekdays=nil, @days=nil, @seconds=[
        0
    ], @minutes=[
        0,
        1,
        2,
        ...
        58,
        59
    ], @hours=nil, @months=nil, @monthdays=nil, @original="*/1 * * * *">, @last_time=2023-06-05 11: 26: 00 -0400, @times=nil, @locals={}, @unscheduled_at=nil, @callable=#<Proc: 0x5f2fa38d@/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/inputs/jdbc.rb: 307>, @next_time=2023-06-05 11: 27: 00 -0400, @local_mutex=#<Thread: :Mutex: 0x4ddb7459>, @mean_work_time=0.32294414634146335, @count=42, @last_work_time=0.232489, @scheduler=#<LogStash: :PluginMixins: :Jdbc: :Scheduler: 0x5a0359c1 @jobs=#<Rufus: :Scheduler: :JobArray: 0x63990fc @mutex=#<Thread: :Mutex: 0x24a8787d>, @array=[#<Rufus: :Scheduler: :CronJob: 0xd11274b ...>
    ]>, @scheduler_lock=#<Rufus: :Scheduler: :NullLock: 0x72ca23c5>, @started_at=2023-06-05 10: 44: 42 -0400, @thread=#<Thread: 0x2a5053e9@[
        4a4255846bcb2ba19b8e67827aff0b3ee0ea127e464f74d15aa158f5ccfa0840
    ]<jdbc__scheduler sleep>, @mutexes={}, @work_queue=#<Thread: :Queue: 0x383ef310>, @frequency=1.0, @_work_threads=[#<Thread: 0x5032dac0@[
            4a4255846bcb2ba19b8e67827aff0b3ee0ea127e464f74d15aa158f5ccfa0840
        ]<jdbc__scheduler_worker-00@/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/plugin_mixins/jdbc/scheduler.rb: 113 run>
    ], @paused=false, @trigger_lock=#<Rufus: :Scheduler: :NullLock: 0x73451f08>, @opts={
        :max_work_threads=>1,
        :thread_name=>"[4a4255846bcb2ba19b8e67827aff0b3ee0ea127e464f74d15aa158f5ccfa0840]<jdbc__scheduler",
        :frequency=>1.0
    }, @thread_key="rufus_scheduler_2174", @max_work_threads=1, @stderr=#<IO:<STDERR>>>, @paused_at=nil, @first_at=2023-06-05 10: 44: 42 -0400, @opts={}, @id="cron_1685976282.6209729_8049722172939095725", @handler=#<Proc: 0x5f2fa38d@/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.3/lib/logstash/inputs/jdbc.rb: 307>, @original="*/1 * * * *">,
    :opts=>{
        :max_work_threads=>1,
        :thread_name=>"[4a4255846bcb2ba19b8e67827aff0b3ee0ea127e464f74d15aa158f5ccfa0840]<jdbc__scheduler",
        :frequency=>1.0
    },
    :started_at=>2023-06-05 10: 44: 42 -0400,
    :thread=>"#<Thread:0x2a5053e9@[4a4255846bcb2ba19b8e67827aff0b3ee0ea127e464f74d15aa158f5ccfa0840]<jdbc__scheduler sleep>",
    :jobs_size=>1,
    :work_threads_size=>1,
    :work_queue_size=>0
}
2

Set the last_run_metadata_path option to something that you have permission to write to.

1 Like
3

Hi Badger, thanks for you answer, it works, so as long I dont set:

use_column_value => true
record_last_run => true
tracking_column => "clock"

the value in last_run_metadata_path wont be taking to account, so I can point multiples pipeline to the same dummy file, is that correct?

4

If you set record_last_run => false the path is never looked at, so you should be OK sharing it in that case.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.