# Enrich pipeline - how to get sum of enriched values from array

**URL:** <https://discuss.elastic.co/t/enrich-pipeline-how-to-get-sum-of-enriched-values-from-array/350604>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [January 8, 2024, 3:49pm UTC](https://discuss.elastic.co/t/enrich-pipeline-how-to-get-sum-of-enriched-values-from-array/350604 "2024-01-08T15:49:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pataposha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pataposha/32/122428_2.png) [@pataposha](https://discuss.elastic.co/u/pataposha)\
**Post date:** [January 8, 2024, 3:49pm UTC](https://discuss.elastic.co/t/enrich-pipeline-how-to-get-sum-of-enriched-values-from-array/350604/1 "2024-01-08T15:49:25Z")

</div>

Hi!  
I'm trying to apply "enrich pipeline" to my data to be able to filter/sort my main index with new fields. These fields are originally stored in a separate index.

Let's say I have two indices:  
`index1` - main index where main documents are stored (recipient index);  
`index2` - additional index with fields I want to copy/enrich into index1 (donor index);

Document from `index1` may have one-to-many relation to `index2`

Example doc from `index1` :

```auto
PUT /index1/_doc/111
{
  "id": "111",
  "foo": "bar"
}

```

Example docs from `index2`:

```auto
PUT /index2/_doc/child-111
{
  "id": "child-111",
  "views": 4,
  "parents": [
    {
      "parent_id": "111",
      "foo": "bar"
    },
    {
      "parent_id": "333",
      "foo": "bar"
    }
    ]
}

PUT /index2/_doc/child-222
{
  "id": "child-222",
  "views": 10,
  "parents": [
    {
      "parent_id": "111",
      "foo": "bar"
    },
    {
      "parent_id": "333",
      "foo": "bar"
    }
    ]
}

```

I want to match documents from `index2` by value inside array of objects:`parents.parent_id` with value of `id` form `index1`. And write sum of matched `views` into main doc from `index1`.

My enrich policy:

```auto
GET /_enrich/policy/test-enrich-summ
{
  "policies": [
    {
      "config": {
        "match": {
          "name": "test-enrich-summ",
          "indices": [
            "index2"
          ],
          "match_field": "parents.parent_id",
          "enrich_fields": [
            "views"
          ]
        }
      }
    }
  ]
}

```

My ingest pipeline:

```auto
[
  {
    "enrich": {
      "field": "id",
      "policy_name": "test-enrich-summ",
      "target_field": "enrich_data",
      "max_matches": 121,
      "ignore_missing": true
    }
  }
]

```

After executing enrich policy I do update docs from `index1` by:

```auto
POST /index1/_update_by_query?pipeline=test-summ-pipeline

```

Resulting doc from `index`:

```auto
GET /index1/_doc/111
{
  "_index": "index1",
  "_id": "111",
  "_version": 19,
  "_seq_no": 18,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "enrich_data": [
      {
        "views": 10,
        "parents": [
          {
            "parent_id": "111"
          },
          {
            "parent_id": "333"
          }
        ]
      },
      {
        "views": 4,
        "parents": [
          {
            "parent_id": "111"
          },
          {
            "parent_id": "222"
          }
        ]
      }
    ],
    "foo": "bar",
    "id": "111"
  }
}

```

It correctly saved combined matched data into `enrich_data` field. But I don't need array of objects there, `parents` and separate `views` fileds are not needed (perhaps only for debug). I want to get a sum of `views`, like some `enrich_data.total_views` field. How can accomplish this?

---

<div class="post-metadata">

**Author:** ![pataposha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pataposha/32/122428_2.png) [@pataposha](https://discuss.elastic.co/u/pataposha)\
**Post date:** [January 8, 2024, 6:00pm UTC](https://discuss.elastic.co/t/enrich-pipeline-how-to-get-sum-of-enriched-values-from-array/350604/2 "2024-01-08T18:00:40Z")

</div>

Answering my own question: you can combine multiple processors into a single pipeline.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/6/e64b2dd2bfb6889e15ca5a69d580eb0d47996ba1.jpeg)

I added 2 addition processors:

1. Script processor to sum `views ` into `total_views`
2. Remove `enrich_data ` field from main doc.

My final pipeline processors:

```auto
[
  {
    "enrich": {
      "field": "id",
      "policy_name": "test-enrich-summ",
      "target_field": "enrich_data",
      "max_matches": 121,
      "ignore_missing": true
    }
  },
  {
    "script": {
      "source": "int total_views = 0;\nfor (item in ctx['enrich_data']) {\n total_views += item['views'];\n }\nctx['total_views'] = total_views;"
    }
  },
  {
    "remove": {
      "field": "enrich_data",
      "ignore_missing": true,
      "ignore_failure": true
    }
  }
]

```

P.S.  
If there is more "elastic" way to accomplish task - please let me know)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2024, 6:01pm UTC](https://discuss.elastic.co/t/enrich-pipeline-how-to-get-sum-of-enriched-values-from-array/350604/3 "2024-02-05T18:01:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.