Get latest log file

Elastic Stack Logstash
1

I am trying to get logs from a SQL Server DB into ElasticSearch using LogStash. All the setup works fine and I am also able to see the logs in ES.
As the DB is updated periodically, so I want to get the latest logs on an hourly basis from it without getting all the previous logs. Is there a way to do this in LogStash, i.e., getting logs based on a time-range?

Any help is really appreciated.

Thanks!!

2

Logstash tracks this for you automatically. You probably only need to make a small adjustment to your SQL query. See the State section in the documentation and the example further down in the docs that uses the sql_last_value parameter.

3

Thanks for quick response! looks like this did the job.

4

@magnusbaeck: I'm trying to schedule a logstash job for every 2 mins but it is not working. The scheduler does not work at all. It would be really helpful if you could review the below code and tell me what I'm doing wrong.

input {
    jdbc {
    # SQLServer jdbc connection string to our database
    jdbc_connection_string => "****"

    #DB credentials
    jdbc_user => "****"
    jdbc_password => "****"


    # The path to our downloaded jdbc driver
    jdbc_driver_library => "path/sqljdbc/enu/sqljdbc41.jar"

    # The name of the driver class for SQL Server
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"

    # SQL query to execute
    statement => "SELECT Id,
            Status,
            Completed
    FROM Table
    WHERE Completed > :sql_last_value"

    #schedule updates
    schedule => "2 * * * *"
}
}
5

The schedule 2 * * * * means "run every hour, two minutes past the hour", i.e. 01:02, 02:02, 03:02, ... Use */2 * * * * instead.

6

Thanks @magnusbaeck. You are awesome!! :slight_smile:

7

@magnusbaeck: another quick question - if I use this schedule for logstash and then close my ssh access to the server, will this process be still running in the background? If not, is there a way to do something like this?

Thanks in advance!!

8

if I use this schedule for logstash and then close my ssh access to the server, will this process be still running in the background?

No, it'll get a SIGHUP signal and shut down when you close your connection.

If not, is there a way to do something like this?

Run Logstash as a daemon. The Debian and RPM packages provide scripts and configuration files for this. Consult your operating system documentation for details.