# How to make sum(field)-sum(field2) and get a single result for all documents

**URL:** <https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979>\
**Category:** Logstash\
**Created:** [June 10, 2019, 11:33am UTC](https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979 "2019-06-10T11:33:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aymen\_Ben\_Moussa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aymen_ben_moussa/32/44513_2.png) [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\
**Post date:** [June 10, 2019, 11:33am UTC](https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979/1 "2019-06-10T11:33:39Z")

</div>

I have a mongodb collection that contains two fields, `amount` and `type`, in `type` i have either `cash-in` , `cash-out` or `transfer`. I want to add an index to elasticsearch with a new field which contains `(the sum of all cash-ins in each document) - (sum of all cash-outs in each document)` . ps: every line pushed either is a cash-in a transaction or a cash-out Everytime I add a new document the field should aggregate itself.  
is there something like this

```
> if type="cash-in"
> mutate {
> add_field => {
> "cash-in" => "%{amount}"
> }
> } 
> if type="cash-out"
> mutate {
> add_field => {
> "cash-out" => "%{amount}"
> }
> } 
> 
> mutate {
> add_field => {
> "total" => "sum%{cash-in}-sum%{cash-out}"
> }
> }
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 10, 2019, 4:57pm UTC](https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979/2 "2019-06-10T16:57:52Z")

</div>

It could be done with an aggregate filter

```
    mutate { add_field => { "[@metadata][task]" => "constant" } }
    aggregate {
        task_id => "%{[@metadata][task]}"
        code => '
            map["total"] ||=0
            t = event.get("type")
            if t == "cash-in"
                map["total"] += event.get("amount")
            elsif t == "cash-out"
                map["total"] -= event.get("amount")
            end
            event.set("total", map["total"])
        '
    }

```

That requires '--pipeline.workers 1' so it does not scale, and it makes assumptions about event ordering that I don't think are guaranteed by logstash (every aggregate does that).

---

<div class="post-metadata">

**Author:** ![Aymen\_Ben\_Moussa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aymen_ben_moussa/32/44513_2.png) [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\
**Post date:** [June 11, 2019, 9:00am UTC](https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979/3 "2019-06-11T09:00:34Z")

</div>

Thank you @Badger i have three questions though:  
-can you please explain to me the first three lines.  
-do `map["total"] ||=0` creates the`total` field automatically and assign it ? because i don't have that field created.  
-I'm a newbie on this so please bear with me, what i've understood from your last couple of lines (out of code) is that i need to set --pipeline.workers 1 along with my command to launch the logstash pipeline because of concurrency issues ? but in a long term probably data will increase and i need to use more cpu nah ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 11, 2019, 9:57am UTC](https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979/4 "2019-06-11T09:57:24Z")

</div>

An aggregate filter requires a task\_id so that it can aggregate lines that are associated with the same event. In this case you need to aggregate every line together, so we use a constant value for the task\_id. That constant value is in the [@metadata] object so that it does not get indexed.

'map["total"] ||=0' is a ruby idiom that says to sets map["total"] to zero if it does not exist. It only has any effect the first time through the filter.

You have to use '--pipeline.workers 1' when using an aggregate filter. It does not scale. It does not matter how many CPUs you have, it can only use 1.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 9, 2019, 9:57am UTC](https://discuss.elastic.co/t/how-to-make-sum-field-sum-field2-and-get-a-single-result-for-all-documents/184979/5 "2019-07-09T09:57:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.