Can we make the query to query only the last 5 min only and make it to run every 5 min?

[Ahmed_Mouawad]

Hello Dears,

I am using logstash pipeline to query from Elasticsearch. I want to query only the last 5 min only and make it to run every 5 min. How to make it?

Logstash.conf:

input {
   elasticsearch {
     hosts => ["##########:9200"]
     ca_file => "/etc/logstash/ca.crt"
     ssl => true
     index => "###"
     user => "####"
     password => "####"
     query => '{"query":{"match_all": {} }}'
}

Any help.

[can.ozdemir]

logstash input api has scheduling option you can find it here;

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

if you are not familiar with crontab, the option you are looking for is: "*/5 * * * *"

about querying on documents from 5 minutes from now, you can use a bool query, first filter your documents to get documents from 5 minutes then match_all.

[Alex_Marquardt]

You will likely want to include a range in your query to request the last 5 min of documents. See Range query | Elasticsearch Guide [7.16] | Elastic.

For building complex queries, you may also be interested in reviewing Boolean query | Elasticsearch Guide [7.16] | Elastic

[Ahmed_Mouawad]

@can.ozdemir

is: "*/5 * * * *"

How to set up in my conf??

[can.ozdemir]

Please read this link

[Ahmed_Mouawad]

like this for run query every 5 min

input {
   elasticsearch {
     hosts => ["##########:9200"]
     ca_file => "/etc/logstash/ca.crt"
     ssl => true
     index => "###"
     user => "####"
     password => "####"
     query => '{"query":{"match_all": {} }}'
     schedule => "*/5 * * * *"
}

[/quote]

[Ahmed_Mouawad]

input {
Elasticsearch {
hosts => ["##########:9200"]
ca_file => "/etc/logstash/ca.crt"
ssl => true
index => "###"
user => "####"
password => "####"
query => '{"query":{ "bool": {"filter":[{"range":{"@timestamp":{"gte":"now-5m/m", "lte":"now/m"}}}],"match_all": {} }}}'
schedule => "*/5 * * * *"
}

[can.ozdemir]

great job!

[Ahmed_Mouawad]

I got this error:

!!!!

[Badger]

You need a space between each entry: */5 * * * *

[Ahmed_Mouawad]

Thanks @Badger , I got this error:

[Badger]

The scheduler is working but there is a problem with your query. I do not run Elasticsearch so I cannot help.

[can.ozdemir]

You need to start with a "must" array and put match_all json object in it after "filter" array is closed.