Pass the schedule dynamically to logstash config file

[Shanmuka_Chowdary]

I'm Indexing data from mssql using logstash with schedule like below

         jdbc {
            jdbc_driver_library => "C:\Program Files\Microsoft JDBC Driver 4.2 for SQL Server\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
            jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
            jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=databasename;"
            jdbc_user => "user"
            jdbc_password => "password"
            statement => "SELECT * TABLE_NAME > :sql_last_value ORDER BY CREATED_ON"
            tracking_column => "created_on"
            schedule => "*/5 * * * *"
            tracking_column_type => "timestamp"
            use_column_value => true
            jdbc_default_timezone => "EST"
        }

But i want to change the schedule from my web application
Is there a way to pass the dynamic schedule to the logstash.conf file when i want to change.

[Wolfram_Haussig]

Hi,

There are multiple ways to do what you want - depending on your requirements:

1. Use an environment variable in the schedule. Your application would have to stop logstash, set the new value for the environment variable and start Logstash again
2. Use automatic config reloading. Your application would have to change the Logstash pipeline file. Logstash would detect the change and restart the pipeline
3. Use centralized pipeline management. Your app would have to use the Kibana API (experimental) to update the logstash pipeline. Logstash would detect the change and restart the pipeline

Maybe there are better ways to do that. but this is what I am aware of.

Best regards
Wolfram

[Shanmuka_Chowdary]

Hi @Wolfram_Haussig
Thanks for the response and the information you have provided is very much helpful.
I was reading the links you shared. Can you please help on the below points

1. Can you please give an example Where I need to set the environmental variable and How to load that to logstash in windows
   where i need to set this export Schedule="*/1 * * * *"

2. Is there a way to change the logstash pipeline file with java programm.

[Wolfram_Haussig]

1. Environment variables
   If you have the following config:

jdbc {
            jdbc_driver_library => "C:\Program Files\Microsoft JDBC Driver 4.2 for SQL Server\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
            jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
            jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=databasename;"
            jdbc_user => "user"
            jdbc_password => "password"
            statement => "SELECT * TABLE_NAME > :sql_last_value ORDER BY CREATED_ON"
            tracking_column => "created_on"
            schedule => "${JDBC_SCHEDULE}"
            tracking_column_type => "timestamp"
            use_column_value => true
            jdbc_default_timezone => "EST"
        }

You can start Logstash using the following commands(Linux/Unix):

set JDBC_SCHEDULE="*/5 * * * *"
./logstash.bat

Your program can do that using(I use Java as example, untested):

public static void main(String[] args) throws Exception {
        ProcessBuilder pb = new ProcessBuilder("cmd", "/c","start", "logstash.bat");
        Map<String, String> env = pb.environment();
        env.put("JDBC_SCHEDULE", "*/5 * * * *");

        Process p = pb.start();
        String output = loadStream(p.getInputStream());
        String error = loadStream(p.getErrorStream());
        int rc = p.waitFor();
        System.out.println("Process ended with rc=" + rc);
        System.out.println("\nStandard Output:\n");
        System.out.println(output);
        System.out.println("\nStandard Error:\n");
        System.out.println(error);
    }

2. LogStash pipeline file
   I am not aware of a library to parse the logstash file but you could do:

• check the GitHub repo how Logstash parses them
• use regular expressions to find and replace the values in the pipeline config

[Shanmuka_Chowdary]

The Java program works fine, Thank you very much.
Before starting the logstash I always need to kill the old process right? Can we do that with Java/ with script.

Since I'm working on windows i couldn't verify Linux/Unix command. It would be great and helpful, if you can provide windows command, So I'll skip Java and write a power shell script

[Wolfram_Haussig]

@Shanmuka_Chowdary

Before starting the logstash I always need to kill the old process right? Can we do that with Java/ with script.

Although Java does not support killing processes natively, you can use windows tools to do that(start them the same way as the logstash example above):

• Get the process using the tasklist commandline tool
• Kill the process using taskkill commandline tool

Powershell has the following corresponding elements:

• Get-Process
• Stop-Process