Logstash - File mentioned in last_run_metadata_path is not getting created

Elastic Stack Logstash
1

I am using Logstash 5.4

File mentioned in last_run_metadata_path is not getting created after starting Logstash.

Logstash config file:

input
{
jdbc
{
jdbc_driver_library => "/app/logstash/dbdrivers/mariadb/mariadb-java-client-2.1.0.jar"
jdbc_driver_class => "org.mariadb.jdbc.Driver"
jdbc_connection_string => "############"
jdbc_user => "######"
jdbc_password => "######"
schedule => "* * * * *"
use_column_value => true
tracking_column => "id"
jdbc_fetch_size => "100000"
last_run_metadata_path => "/app/logstash/var/.logstash_jdbc_last_run"
statement => "select id,nodeinstanceid,nodename,processinstanceid,@timestamp,processid from NodeInstanceLog where id > :sql_last_value"

    }

}

output
{
file
{
path => "/app/applications/logs/bpm/NodeInstanceLog.log"
}
}

The file NodeInstanceLog.log gets updated as shown below,

[root@##### bin]# tail -f /app/applications/logs/bpm/NodeInstanceLog.log
{"nodename":"ULIO Signal Point","processinstanceid":94853,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400949,"nodeinstanceid":"17"}
{"nodename":"PERFORM_VISION","processinstanceid":94775,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400952,"nodeinstanceid":"37"}
{"nodename":"","processinstanceid":94781,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400955,"nodeinstanceid":"32"}
{"nodename":"Initialize Request","processinstanceid":96392,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.PERFORM_VISION","@version":"1","id":3400958,"nodeinstanceid":"1"}
{"nodename":"","processinstanceid":94826,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400961,"nodeinstanceid":"36"}
{"nodename":"Parse MTAS Response and Validate Critical Elements","processinstanceid":95837,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400964,"nodeinstanceid":"6"}
{"nodename":"VP Signal Point","processinstanceid":94754,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400967,"nodeinstanceid":"23"}
{"nodename":"","processinstanceid":94838,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400970,"nodeinstanceid":"24"}
{"nodename":"Initialize Request","processinstanceid":96380,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.PERFORM_VISION","@version":"1","id":3400973,"nodeinstanceid":"1"}
{"nodename":"PERFORM_VISION","processinstanceid":94778,"@timestamp":"2017-08-14T14:19:26.884Z","processid":"M2MActivation.M2M_Subprocess","@version":"1","id":3400976,"nodeinstanceid":"37"}

But file not getting created in last_run_metadata_path,

[root@10-119-12-32 var]# pwd
/app/logstash/var
[root@10-119-12-32 var]# ls -al
total 24
drwxrwxrwx 2 root root 4096 Aug 2 12:05 .
drwxrwxr-x 14 root root 4096 Jul 31 09:52 ..
-rw-r--r-- 1 root root 276 Jul 31 10:14 orbpm_bpmnodes_cluster.sincedb
-rw-r--r-- 1 root root 254 Jul 31 10:14 orbpm_bpmnodes_nitcluster.sincedb
-rw-r--r-- 1 root root 256 Jul 31 10:14 orbpm_bpmprocess_cluster.sincedb
-rw-r--r-- 1 root root 224 Jul 31 10:14 orbpm_bpmprocess_nitcluster.sincedb

Log file contents:

[root@####### logs]# tail -f logstash-plain.log
[2017-08-14T10:08:00,466][INFO ][logstash.outputs.file ] Opening file {:path=>"/app/applications/logs/bpm/NodeInstanceLog.log"}
[2017-08-14T10:08:40,600][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-08-14T10:08:40,613][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-08-14T10:17:12,032][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-08-14T10:17:12,587][INFO ][logstash.pipeline ] Pipeline main started
[2017-08-14T10:17:12,657][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-08-14T10:18:00,229][INFO ][logstash.inputs.jdbc ] (0.131000s) select id,nodeinstanceid,nodename,processinstanceid,@timestamp,processid from NodeInstanceLog where id > 0
[2017-08-14T10:18:00,497][INFO ][logstash.outputs.file ] Opening file {:path=>"/app/applications/logs/bpm/NodeInstanceLog.log"}

Can anyone please let me know what is causing the issue ? and what configuration I need to change to get the last_run_metadata_path file created?

Thanks in advance

2

The problem has been solved by changing the jdbc_connection_string as shown below ,

jdbc_connection_string => "jdbc:mariadb://127.0.0.1:3306/database?sessionVariables=sql_mode=ANSI_QUOTES"

Note the "sql_mode=ANSI_QUOTES"
From the MariaDB documentation: https://mariadb.com/kb/en/mariadb/sql-mode/

ANSI_QUOTES: Changes " to be treated as `, the identifier quote character. This may break old MariaDB applications which assume that " is used as a string quote character.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.