Require Last 15 minutes from timestamp

[rubhamra (heyanB)]

I have REST API call and I need to pull records which were updated every last 15 minutes.

I can pull last 24 records but I need just updated with in last 15 minutes based on the date field.

This is the ruby code

ruby {
    code => "event.set('[yesterday]', (Time.now - 86400).strftime('%Y-%m-%dT%H:%M:%S.%N'))"
    }

logstash config file (http plugin)

http {
        id => "ch-id"
        verb => "GET"       
        url => "http://localhost:8080/api/arsys/v1.0/entry/task%20list?q=Changed Date > %{yesterday}"   
        headers => {
            "Authorization" => "%{[@metadata][request][token]}"
            "X-AR-Client-Type" => '34'
        }
        request_timeout => 600
        socket_timeout => 600
        body_format => "json"
        target_body => "[response][body]"
        target_headers => "[response][headers]"
        periodic_flush => true           
    }

I need to pull records that were updated with in last 15 minutes based on field Changed Date

Thanks

[Badger]

Can't you just change your ruby code to subtract 900 from Time.now instead of 86400?

[rubhamra (heyanB)]

Thanks Badger, It was just I couldn't think of that but It working as you said, but this logic is not still good for me.

As you can below see I am calling REST API based on passing 'Changed Date' as query parameter to get yesterday data and once you have the all the data from yesterday you can run Logstash every 2 minutes grabbing latest data from last 15 minutes.

but if Logstash stopped or Elasticsearch stopped I need to adjust the time to get all the data since the system is stopped.

I need help on how I can store the last 'Changed Date and time' and compare it next time with current time, to grab data from last 'Changed Date' which somehow I need to store somewhere.

It's Like what you do with JDBC input plugin and you can store the last sql time in .sql_last_value.

Let me know is that possible,

Thanks

http {
        id => "ch-id"
        verb => "GET"       
        url => "http://localhost:8080/api/arsys/v1.0/entry/task%20list?q=Changed Date > %{yesterday}"   
        headers => {
            "Authorization" => "%{[@metadata][request][token]}"
            "X-AR-Client-Type" => '34'
        }
        request_timeout => 600
        socket_timeout => 600
        body_format => "json"
        target_body => "[response][body]"
        target_headers => "[response][headers]"
        periodic_flush => true           
    }