# Getting the first and last entry(by timestamp) for each filename

**URL:** <https://discuss.elastic.co/t/getting-the-first-and-last-entry-by-timestamp-for-each-filename/114871>\
**Category:** Elasticsearch\
**Created:** [January 10, 2018, 12:42pm UTC](https://discuss.elastic.co/t/getting-the-first-and-last-entry-by-timestamp-for-each-filename/114871 "2018-01-10T12:42:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![phytopius](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phytopius/32/22916_2.png) [@phytopius](https://discuss.elastic.co/u/phytopius)\
**Post date:** [January 10, 2018, 12:42pm UTC](https://discuss.elastic.co/t/getting-the-first-and-last-entry-by-timestamp-for-each-filename/114871/1 "2018-01-10T12:42:56Z")

</div>

I have an index with around 140 milllion documents stored.

The fields in general look like this: systemname, filename, timestamp, message, version, etc,

What I want to get is the first and last document (based on the timestamp) per filename.

What I am doing right now is:

- Query the whole index
- Define window based on filename
- Sort ascending
- Add rownumber
- Add max\_rownumber
- Filter all entries where rownumber = 1 or rownumber = max\_rownumber

This is really really slow.

My second option was to sort it ascending add rownumber find number 1 and do the same sorted descending. This is as slow if not even slower than the first approach.

Code is as follows:  
val ascFileWindow = Window.partitionBy($"systemname", $"filename").orderBy($"timestamp".asc)  
val ds2\_first = ds\_filtered  
.withColumn("rn", row\_number().over(ascFileWindow))  
.withColumn("max\_rn", max($"rn").over(ascFileWindow.rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)))  
.withColumn("lf\_flag", when($"rn" === 1 || $"rn" === $"max\_rn", 1).otherwise(lit(0)))  
.where($"lf\_flag" === 1)

ES Version: 6.0.0  
ES Spark Connector: elasticsearch-spark-20 6.0.0  
Scala Version 2.11.8  
Spark Version: 2.2

If someone could help me with a solution that would be really nice. I am out of Options.  
Not being able to query aggregations seems to become a dealbreaker for elasticsearch but I don't want to switch away 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2018, 12:43pm UTC](https://discuss.elastic.co/t/getting-the-first-and-last-entry-by-timestamp-for-each-filename/114871/2 "2018-02-07T12:43:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.