LogStash JDBC plugin - Issues with last_run_metadata_path

[mulgar (Michael Green)]

I'm having some problems with the last_run_metadata_path in the LogStash JDBC Input plugin (ref: Jdbc input plugin | Logstash Reference [8.11] | Elastic ).

I have the sql_last_value parameter working as expected previously using the default path (not specifying last_run_metadata_path). I need to have 2 inputs one which is pulling data from staging environment, the other production - so I need to customise the last run sql file so I can track the last run timestamp separately for the 2 environments.

I added the following in my LogStash configuration file in the jdbc plugin configuration:

last_run_metadata_path => "/etc/logstash/conf.d/lastrun/.logstash_jdbc_test_last_run"

Restarted the LogStash service, however it won't take the configuration and seems to keep writing to the default location. Now when I was first testing the sql_last_value feature I discovered that the default location for this file is:

"$USER_HOME/.logstash_jdbc_last_run"

(source: Default value for last_run_metadata_path config option is specific to OS X · Issue #113 · logstash-plugins/logstash-input-jdbc · GitHub ).

However while I could find the file there before (not sure why or when this bit changed) but now I can find the file in the following location, so it seems to be using this as the default (I'm not sure if this is relevant or not but it could be important):

"/usr/bin/logstash/.logstash_jdbc_last_run"

The file gets updated fine etc (although I'm not sure why it is in that location), but it won't seem to accept my configuration trying to using a custom file, which is needed to track multiple inputs with separate last sql values.

Any help appreciated on this, happy to provide additional information as needed, thanks.

Edit:
Also increasing the logging to DEBUG I don't see any errors or anything out of the ordinary here, the only DEBUG logs I can see related to this problem is where it picks up the configuration on start-up and this looks like it should be picking up the correct (custom) path to me:

[2017-06-13T14:34:59,302][DEBUG][logstash.inputs.jdbc ] config LogStash::Inputs::Jdbc/@last_run_metadata_path = "/etc/logstash/conf.d/lastrun/.logstash_jdbc_test_last_run"

[mulgar (Michael Green)]

I had some confusing observations along the way which I don't have the time to dig into, but seems this was a simple permissions issue which ultimately fixed this.

Using this configuration in JDBC input plugin:

last_run_metadata_path => "/etc/logstash/conf.d/lastrun/.logstash_jdbc_test_last_run"

I just had to create the directory and change the ownership to logstash such as:

sudo chown logstash:logstash /etc/logstash/conf.d/lastrun

After doing this the last run file gets created okay in the lastrun folder by the logstash service. I know a simple fix here, but wanted to update in case anyone else stumbles across this and it helps.