# Retrieve more than 10000 records for search request in Watcher
**URL:** https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [April 8, 2020, 9:32am UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091 "2020-04-08T09:32:42Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@shubhamshd99](https://discuss.elastic.co/u/shubhamshd99)
#### Post date: [April 8, 2020, 9:32am UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091/1 "2020-04-08T09:32:42Z")
Hi Guys,
I am trying to retrieve records more than 10k through search request in Watcher, but due to its max-limit, it is rejecting all the records over and above 10k number. There is already a discussion
[here](https://discuss.elastic.co/t/pulling-more-than-10000-records-from-elasticsearch-query/181000/2) on how Scroll API could be used for elasticsearch query, but how do we use it in watcher???
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [April 8, 2020, 1:23pm UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091/2 "2020-04-08T13:23:45Z")
Hey,
watcher does not support searching more than those max documents. Maybe you can talk a little bit about your use-case and we can check, if there is a chance to express this in a different manner.
Thanks!
--Alex
---
### Author:  [@shubhamshd99](https://discuss.elastic.co/u/shubhamshd99)
#### Post date: [April 9, 2020, 6:28am UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091/3 "2020-04-09T06:28:45Z")
Hi Alex,
Actually I have tried to implement Splunk **dedup** through watcher, please find below the configuration for the same.
So basically, I have used aggregation to get unique field values(sourcetype, in this case) and along with that there is a query to retrieve actual hits. ( **And here is the actual problem, some time total hits of the query exceed 10k and when I try to access these rejected hits in transform I get a index out of bound error as they have been chopped off** )
Finally in transform section, for every unique value of the field obtained through aggregation, I find a matching document in hits.hits.(If aggregation would have also returned other fields along with the sourcetype field, I wouldn't have to use transform!!! )
This deduplicated hits are then emailed through send email action, which has been removed as per organization policy.
```auto
{
"trigger": {
"schedule": {
"interval" : "5000m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash-wps*"
],
"rest_total_hits_as_int": true,
"body": {
"size" : 10000,
"query": {
"bool": {
"must": [
{
"term": {
"app": "wps_bcl_my"
}
},
{
"range": {
"@timestamp": {
"gte": "now-15m"
}
}
}
]
}
},
"aggregations" : {
"unique_sources" : {
"terms" : {
"field" : "sourcetype",
"size" : 100
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"transform":{
"script":
"""
def doc_count = ctx.payload.hits.total;
def unique_hits = [];
def unique_sources = ctx.payload.aggregations.unique_sources.buckets.stream().map(hit->hit.key).collect(Collectors.toList());
for(def source : unique_sources){
for(def i=0; i< doc_count; i++){
if(source==ctx.payload.hits.hits[i]._source.sourcetype){
unique_hits.add(ctx.payload.hits.hits[i]);
break;
}
}
}
return unique_hits;
"""
}
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 9, 2020, 6:40am UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091/4 "2020-04-09T06:40:49Z")
Have you considered trying to reduce or even [eliminate duplicates at indexing time](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) instead so you do not need to do this at all?
Another option might be to eliminate most duplicates in your ingest pipeline even before you send them to Elasticsearch.
---
### Author:  [@shubhamshd99](https://discuss.elastic.co/u/shubhamshd99)
#### Post date: [April 9, 2020, 8:14am UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091/5 "2020-04-09T08:14:34Z")
Hi Christian,
As per our requirement, we can't deduplicate during ingestion or indexing, it is required only for alerting!!!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 7, 2020, 8:14am UTC](https://discuss.elastic.co/t/retrieve-more-than-10000-records-for-search-request-in-watcher/227091/6 "2020-05-07T08:14:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.