Question about JDBC input plugin's last_run_metadata_path setting

Elastic Stack Logstash
1

I am using the last_run_metadata_path do define where the file it uses for persistence is kept. The documentation says that the default value for this is "/home/ph/.logstash_jdbc_last_run", but there is no such file or directory on my server. I believe this is causing it to not work. I read in a forum that others were having the same issue, and that trying to set it to a file in $HOME also has problems because of read/write user permissions.

As suggested in the forum, I set my path to /tmp/.jdbc_metrics-buffalonews_last_run and it is now working for me, but I am concerned because, from what I understand, the tmp directory is only temporary. Wouldn't that mean whenever tmp gets cleared that that file will be removed and that Logstash won't remember where it left off?

Am I misunderstanding something?

Thank you.

2

No, you're right. /tmp is usually cleared upon reboot and is therefore not suitable for this purpose.

I see no reason why there should be a general problem storing the file somewhere under $HOME.

3

Ah, I might have been misdiagnosing the issue. I have one very long running import using JDBC paging so that it splits it up into queries of 100000 limits. When does Logstash generate the jdbc_last_run file? When the entire pipeline ends? Or when the last output happened?

4

Not sure OTOH. Check the source code?

5

Hi @magnusbaeck, I tried changing the path and it's giving me the following error now:

  Error: Permission denied - /home/ubuntu/.jdbc_metrics_last_run
  Exception: Errno::EACCES
  Stack: org/jruby/RubyIO.java:3622:in `write'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.2/lib/logstash/inputs/jdbc.rb:284:in `update_state_file'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.2/lib/logstash/inputs/jdbc.rb:257:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:524:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:517:in `block in start_input'

Am I setting the path wrong?

6

No, but it looks like the user running Logstash doesn't have write access to ~ubuntu/.jdbc_metrics_last_run.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.