# Get oldest / newest document in \*beat **URL:** \ **Category:** Elasticsearch\ **Created:** [December 17, 2019, 9:14pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229 "2019-12-17T21:14:27Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\ **Post date:** [December 17, 2019, 9:14pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/1 "2019-12-17T21:14:27Z")
Greetings. I need to fetch the oldest or newest document saved into a beat. Someone asked this before and this was the suggested course of action: > [@"How to get the oldest document of all indexes matching an index pattern](https://discuss.elastic.co/t/how-to-get-the-oldest-document-of-all-indexes-matching-an-index-pattern/68542): > > Hello, is there a solution to get this information? This approach will NOT work for metricbeat data because you cannot snag over 10000 records at a time and metricbeat produces almost 9000 in one day. Any suggestions? Thank you! ---
**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\ **Post date:** [December 17, 2019, 9:21pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/2 "2019-12-17T21:21:59Z")
One of the indices I'm looking at has 57,150,950 documents. That's just one of them. : / ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 18, 2019, 8:54am UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/3 "2019-12-18T08:54:22Z")
What is the problem of sorting by timestamp and only return a single document? Execute two searches, one with sort asc and the other with sort desc. Would that work or is there an issue with that approach that I might have missed in your post? ---
**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\ **Post date:** [December 18, 2019, 4:48pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/4 "2019-12-18T16:48:11Z")
There are way too many documents. I'm querying with Python - which has a maximum document size of 10k. With metricbeat data I can go back about a day but that's it. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 19, 2019, 9:06am UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/5 "2019-12-19T09:06:08Z")
we are now talking about something different, it seems. Your initial ask was to retrieve only a single document from what I read. If you need to get more than 10k documents, than use a scroll search. ---
**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\ **Post date:** [December 19, 2019, 3:30pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/6 "2019-12-19T15:30:41Z")
Okay... I need to get the oldest of 15 million records. I don't think I can do it. ---
**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\ **Post date:** [December 19, 2019, 3:45pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/7 "2019-12-19T15:45:03Z")
For instance, if I do this query, I can go back a few days. But that's all. ``` resr = esr.search(index="metricbeat-*", size=documentSize, body={ "query": { 'bool': { 'must': [ {"term": {'fields': thisId}}, {"exists": {"field": "host"}} ] } }, "sort": [ { "@timestamp": { "order": "desc" } } ] }) return resr['hits']['hits'][0]['_source'] ``` I'll look into the scroll functionality and see if I can get it to work. Thank you for the suggestion. ---
**Author:** ![EricJohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericjohnson/32/53592_2.png) [@EricJohnson](https://discuss.elastic.co/u/EricJohnson)\ **Post date:** [December 20, 2019, 8:57pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/8 "2019-12-20T20:57:00Z")
In order to get around this problem I am changing my approach. Of course it created a new issue... But here's the follow up question: > [@Aggregate metricbeat 'host' info by ID & Unique host configuration?](https://discuss.elastic.co/t/aggregate-metricbeat-host-info-by-id-unique-host-configuration/212709): > > Hi there. I'm trying to aggregate host configuration data out of metricbeat. I need to look at the documents for a single day, bucketing by ID, and return the host info. What I'm trying to do is watch for configuration changes on a daily basis. If I use this code: "query": { 'bool': { 'must': [{"exists": {"field": "host"}}, {'term': {'@timestamp': date}}] } }, "aggs": { "by\_id": { "terms": {"field": "fields.id"} } } I get an array out containing our fields.id … Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2020, 8:57pm UTC](https://discuss.elastic.co/t/get-oldest-newest-document-in-beat/212229/9 "2020-01-17T20:57:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.