# Write Preprocessor to Sum records

**URL:** <https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379>\
**Category:** Elasticsearch\
**Created:** [November 28, 2016, 4:58pm UTC](https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379 "2016-11-28T16:58:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Ainslie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_ainslie/32/55031_2.png) [@Paul\_Ainslie](https://discuss.elastic.co/u/Paul_Ainslie)\
**Post date:** [November 28, 2016, 4:58pm UTC](https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379/1 "2016-11-28T16:58:52Z")

</div>

Would it be possible to write a preprocessor in ES 5.0 which: a) counts our incoming nginx access records; and b) summarizes the byte count on each record and accumulates these two values for each unique session id over 5 minute intervals and then writes that out every 5 minutes for each session id? This could later be changed to longer intervals, but the principle is still the same.

I know everyone's going to say, 'just use aggregations in ES to report on the same thing'. Well, we currently are, but we have a requirement for more summarized data and faster query speeds when summarizing over a period of, say 2 months or 1 year.

Paul

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [November 28, 2016, 7:07pm UTC](https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379/2 "2016-11-28T19:07:54Z")

</div>

See logstash [https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) or entity centric indexing [https://m.youtube.com/watch?v=yBf7oeJKH2Y](https://m.youtube.com/watch?v=yBf7oeJKH2Y)

---

<div class="post-metadata">

**Author:** ![Paul\_Ainslie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_ainslie/32/55031_2.png) [@Paul\_Ainslie](https://discuss.elastic.co/u/Paul_Ainslie)\
**Post date:** [November 29, 2016, 3:08pm UTC](https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379/3 "2016-11-29T15:08:19Z")

</div>

Thanks Mark - so if I'm correct, that's given me two options.  
a) Aggregate real time, at source using the aggregate Logstash plugin.  
b) Create a script on Elasticsearch doing something similar to your entity centrix indexing example. In this case, would you suggest I use Groove or the new Painless language?

Paul

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [November 29, 2016, 3:17pm UTC](https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379/4 "2016-11-29T15:17:08Z")

</div>

> [@Paul\_Ainslie](#):
>
> that's given me two options.

Yep.

> [@Paul\_Ainslie](#):
>
> would you suggest I use Groove or the new Painless language?

Good question 🙂 I've not tried converting my example Groovy script into Painless yet.  
At [this point](https://youtu.be/yBf7oeJKH2Y?t=14m21s) in the video I show the outline of an update script. The things it needs to do typically are:

1. Load saved JSON arrays into a Map or a Set to represent info you want to maintain (e.g. a unique list of products a person has bought over time)
2. A "for" loop to iterate over the new events and update the data in 1)
3. Serialize the collections back to plain old JSON arrays ready for storage.

I'd be interested to hear how easy it is to do in Painless if you attempt this translation.  
My example Groovy script is at [http://bit.ly/entcent](http://bit.ly/entcent)

Cheers  
Mark

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 27, 2016, 3:17pm UTC](https://discuss.elastic.co/t/write-preprocessor-to-sum-records/67379/5 "2016-12-27T15:17:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.