Implement concurrency while inserting data to ES using logstash

Elastic Stack Logstash
1

Hi,

I am using below config file for logstash. I have used schedule= 1 min as jdbc scheduler which keep on inserting data to ES in every 1 min. But when number of records are more i am not able to identify weather it is following transaction or not . it should follow locking of resource data. How to implement locking kind of thing it in this case.

input {
jdbc {
jdbc_driver_library => "demo\sqljdbc_6.4\enu\mssql-jdbc-6.4.0.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://L-ff;databaseName=Cultivation;integratedSecurity=true;"
jdbc_user => "manjur.gani"
statement_filepath=>"demo:\SQLQuery12.sql"
schedule => "* * * * *"
tracking_column => "sdate"
last_run_metadata_path => "demo.logstash_jdbc_last_run"
tracking_column_type => "timestamp"
use_column_value => true
type =>farm
}

}
output
{
//code
}

2

I'm not sure what you're asking. What transaction? What locking?

But when number of records are more

More than what?

3

okey. lets say schedule time is 1 min. If my first run took more than 1 min because of heavy data and mean while 2nd run starts as we have given schedule time as 1 min.so the question is how we can restrict this behavior.

4

Okay. Queries will never run concurrently, i.e. if the first query takes longer than one minute to process the second query won't run until the first one's done.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.