# Is there a way to implement a keep last N docs per id in a specific index?

**URL:** https://discuss.elastic.co/t/is-there-a-way-to-implement-a-keep-last-n-docs-per-id-in-a-specific-index/217249
**Category:** Elasticsearch
**Created:** [January 30, 2020, 5:15pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-implement-a-keep-last-n-docs-per-id-in-a-specific-index/217249 "2020-01-30T17:15:41Z")
**Posts on this page:** 1
**Showing post:** 3

<div class="post-metadata">

### Author: ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [January 31, 2020, 7:53am UTC](https://discuss.elastic.co/t/is-there-a-way-to-implement-a-keep-last-n-docs-per-id-in-a-specific-index/217249/3 "2020-01-31T07:53:04Z")

</div>

First, there are no "weird questions". It is an interesting usecase.

This can indeed be done with transforms. For this case you create a transform `group_by` customer id and collapse the purchases as list. I described an example in an [advent calendar post](https://discuss.elastic.co/t/dec-5th-2019-de-weihnachtswunsche-zusammenfassen-mit-transforms/209946), unfortunately german language. Let me take it from there, you can start with a scripted metric like this:

```auto
"all_purchases": {
  "scripted_metric": {
    "init_script": "state.docs = []",
    "map_script": "state.docs.add(new HashMap(params['_source']))",
    "combine_script": "return state.docs",
    "reduce_script": "def docs = []; for (s in states) {for (d in s) { docs.add(d);}}return docs"
  }
}

```

This would create a list of all purchases:

```auto
"all_purchases" : [
  {
    "order_id" : 42,
    "date" : "2019-12-01T10:00:00Z",
...
  },
  {
    "order_id" : 99,
    "date" : "2019-12-02T12:00:00Z",
...
  },

```

To get the top-`n` 2 possibilities come to my mind: "at runtime" or "as post processing".

_Post processing_: Sort the list afterwards by order date and cut it at `n`. You can do this as part of the reduce script or you write the output of the transform into a pipeline and use a [script processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/script-processor.html).

_At runtime_: Instead of a list use a sorted map, a [TreeMap](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-util.html#painless-api-reference-shared-TreeMap) and you map `order_date` to the order object. For insert you could only add to the tree map if it either not reached `n` or the [first key](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/SortedMap.html#firstKey()) - which is the "lowest" key - is lower than the key you are looking at. Afterwards you trim it to `n`. Memory-wise this is the most efficient way as this will not keep all orders in memory. Memory consumption might not be a problem for an e-commerce usecase but for IOT it could.

I hope I gave you an idea and would be happy if you share your result.

---

_[View the full topic](https://discuss.elastic.co/t/is-there-a-way-to-implement-a-keep-last-n-docs-per-id-in-a-specific-index/217249)._