# Agg last 15 minutes returning just last 2 minutes
**URL:** https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978
**Category:** Elasticsearch
**Created:** [April 27, 2020, 2:50pm UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978 "2020-04-27T14:50:27Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 27, 2020, 2:50pm UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/1 "2020-04-27T14:50:27Z")
```auto
GET forensics/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now"
}
}
}
]
}
},
"aggs": {
"agg1": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s"
}
}
}
}
```
why it returns only data of two last minutes?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 27, 2020, 5:47pm UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/2 "2020-04-27T17:47:09Z")
If you are speaking of the `hits` part of the response that's expected as only 10 hits are returned by default.
But the `aggs` part should give more data I think.
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 28, 2020, 9:03am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/3 "2020-04-28T09:03:57Z")
Im talking about Aggs as I wrote in the title and it doesn`t give me all data just last couple minutes
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 28, 2020, 10:04am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/4 "2020-04-28T10:04:39Z")
Could you share the full output?
BTW are you sure you want to generate 2.880 buckets?
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 8:20am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/6 "2020-04-30T08:20:37Z")
Im showing you edited query the above Code is the output this
```auto
GET forensics/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now"
}
}
}
]
}
},
"aggs": {
"agg1": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s"
},
"aggs":{
"sub_agg1":{
"terms":{
"field": "action"
}
}
}
}
}
}
```
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 8:27am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/7 "2020-04-30T08:27:44Z")
btw 2.880 buckets what ?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 30, 2020, 8:30am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/8 "2020-04-30T08:30:14Z")
Could you share the entire output, not only the aggs part? If too big, share on [gist.github.com](http://gist.github.com) and add the link here.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 30, 2020, 8:30am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/9 "2020-04-30T08:30:59Z")
2880 = 3600 seconds per hour \* 24 hours for a day / 30 seconds
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 8:42am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/11 "2020-04-30T08:42:01Z")
My original "time query " is different, I used here a a simple timeframe to simplify things
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 30, 2020, 8:43am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/12 "2020-04-30T08:43:39Z")
```auto
GET forensics/_search
{
"size": 1,
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now"
}
}
}
]
},
"sort" : [
{ "@timestamp" : {"order" : "asc"}}
]
}
}
```
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 8:46am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/13 "2020-04-30T08:46:52Z")
> [@dadoonet](#):
>
> "sort" : [{ "@timestamp" : {"order" : "asc"}}]
sorry I didnt notice my is
```auto
sort: [
{
"@timestamp": {"order": "desc"},
"_id": {"order": "desc"},
}
]
```
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 8:56am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/14 "2020-04-30T08:56:01Z")
using the new query you gave unfortunately didn`t change much response is the same
still getting just couple minutes
here is the full query with some more properties:
```auto
GET forensics/_search
{
"size": 0,
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now"
}
}
}
]
}
},
"sort": [
{
"@timestamp": {"order": "desc"},
"_id": {"order": "desc"}
}
],
"aggs": {
"agg1": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s"
},
"aggs":{
"sub_agg1":{
"terms":{
"field": "action"
}
}
}
}
}
}
```
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [April 30, 2020, 9:03am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/15 "2020-04-30T09:03:17Z")
I think the objective here is to know if there is data in the past.
For this reason @dadoonet is suggesting to set a sorting by `@timestamp` ascending, so we can see the timestamp of the oldest hit you have in the time range.
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 9:12am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/16 "2020-04-30T09:12:03Z")
he didnt gave any instructions what did he want me to do ?
post the response here ?
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [April 30, 2020, 9:24am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/17 "2020-04-30T09:24:37Z")
Run the following 2 queries (no modifications) and post the full response.
```auto
GET forensics/_search
{
"size": 1,
"sort": [
{
"@timestamp": {
"order": "asc"
}
}
],
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now"
}
}
}
]
}
},
"aggs": {
"stat": {
"stats": {
"field": "@timestamp"
}
}
}
}
```
```auto
GET forensics/_search
{
"size": 0,
"aggs": {
"stat": {
"stats": {
"field": "@timestamp"
}
}
}
}
```
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 9:34am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/18 "2020-04-30T09:34:01Z")
first Q:
```auto
{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 12,
"successful" : 12,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 999,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "forensics-2020.04.30-007132",
"_type" : "_doc",
"_id" : "gntsynEBABC9EwCFjvEw",
"_score" : null,
"_source" : {
"attack_sev" : "High",
"eventMap" : {
"map" : [
{
"key" : "ParamName",
"value" : "cmd"
},
{
"key" : "ParamValue",
"value" : "cmd.exe"
},
{
"key" : "ParamType",
"value" : "ParamTypeURI"
},
{
"key" : "Zone",
"value" : "Parameters"
}
]
},
"eventSignature" : "7592",
"rule" : "7592",
"@timestamp" : "2020-04-30T09:30:37.177Z",
"action" : "REPORT",
"trafficTransId" : "34ebfb0c-77b4-4d90-a648-9adfdc5cfb3b",
"waas_tag" : "app-httpbin",
"related" : "",
"dynamic" : "",
"web_servers" : "Any",
"waas_profile" : "radware/waas-sample-app-httpbin-profile",
"policyClsId" : "classifier1",
"attack" : "URL Access Violation",
"policyVersionHash" : "4c3877395b510e1bf4636c355a36de4ad9552b9cd4945a322add51a60bd00ae1",
"trafficUri" : "/s",
"threat" : "Access Control",
"eventId" : "by_pattern",
"sourceHostname" : "waas-sample-app-httpbin-deployment-5f58dc8c9-b299b",
"policyProtectionId" : "protection1",
"trafficMethod" : "GET",
"tags" : [
"_geoip_lookup_failure"
],
"description" : "Signature engine intercepted a malicious request, which includes a blocked pattern. Description: There was an attempt to retrieve Windows Applications file",
"title" : "Pattern Violation Detected",
"eventModule" : "Known Attacks - Signature Engine",
"policyName" : "httpbinPolicy"
},
"sort" : [
1588239037177
]
}
]
},
"aggregations" : {
"stat" : {
"count" : 999,
"min" : 1.588239037177E12,
"max" : 1.588239103142E12,
"avg" : 1.5882390699646445E12,
"sum" : 1.58665083089468E15,
"min_as_string" : "2020-04-30T09:30:37.177Z",
"max_as_string" : "2020-04-30T09:31:43.142Z",
"avg_as_string" : "2020-04-30T09:31:09.964Z",
"sum_as_string" : "+52248-12-18T05:54:54.680Z"
}
}
}
```
secong Q:
```auto
{
"took" : 34,
"timed_out" : false,
"_shards" : {
"total" : 12,
"successful" : 12,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1050,
"relation" : "eq"
},
"max_score" : null,
"hits" : []
},
"aggregations" : {
"stat" : {
"count" : 1050,
"min" : 1.588239077157E12,
"max" : 1.588239146136E12,
"avg" : 1.5882391116414248E12,
"sum" : 1.667651067223496E15,
"min_as_string" : "2020-04-30T09:31:17.157Z",
"max_as_string" : "2020-04-30T09:32:26.136Z",
"avg_as_string" : "2020-04-30T09:31:51.641Z",
"sum_as_string" : "+54815-10-04T23:33:43.496Z"
}
}
}
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 30, 2020, 9:35am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/19 "2020-04-30T09:35:54Z")
I did here: [Agg last 15 minutes returning just last 2 minutes](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/12)
But indeed I did not say specifically that you should not modify the request I sent and share the output. Thanks @Luca_Belluccini for being more precise.
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [April 30, 2020, 9:42am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/20 "2020-04-30T09:42:44Z")
Given your requests, we can see:
- The oldest document when we apply the time filter `now-1d/d to now` is `2020-04-30T09:30:37.177Z`
- The most recent document when we apply the time filter `now-1d/d to now` is `2020-04-30T09:31:43.142Z`
- Without filters, you have only 1050 documents in the `forensics` alias and the oldest event is `2020-04-30T09:30:37.177Z`, while the most recent one is `2020-04-30T09:31:43.142Z`
In the response I can spot you seem to use ILM and the rollover policy seems quite too frequent as you have `forensics-2020.04.30-007132` (7132!!!) rollovers!
Maybe the ILM policy is configured to delete data too frequently, so you do no more have data past this date?
---
### Author:  [@Elad\_Ben-aderet](https://discuss.elastic.co/u/Elad_Ben-aderet)
#### Post date: [April 30, 2020, 11:32am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/21 "2020-04-30T11:32:21Z")
As it turns out I`m working on a development cluster and its set to delete more then 2 minutes data.
TY @Luca_Belluccini and @dadoonet for helping me figure this out.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 28, 2020, 11:32am UTC](https://discuss.elastic.co/t/agg-last-15-minutes-returning-just-last-2-minutes/229978/22 "2020-05-28T11:32:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.