# Opensearch transform splitting array values to new events

**URL:** <https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575>\
**Category:** Elasticsearch\
**Created:** [August 6, 2022, 5:33am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575 "2022-08-06T05:33:50Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akshay\_Kulkarni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_kulkarni/32/101037_2.png) [@Akshay\_Kulkarni](https://discuss.elastic.co/u/Akshay_Kulkarni)\
**Post date:** [August 6, 2022, 5:33am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/1 "2022-08-06T05:33:50Z")

</div>

Hi,

I'm transforming following event. But the values inside of array are splitting into the new events.

```auto
      {
        "_index": "heal_collated_txn_health_2022.05",
        "_type": "_doc",
        "_id": "LAUpboIBh6CUatILrsN3",
        "_score": 1,
        "_source": {
          "timeInGMT": 0,
          "kpiId": 0,
          "compInstanceIdentifier": "d0352b7d-0484-4714-bbc8-eb67cbb7be70",
          "agentIdentifier": "ComponentAgent-171",
          "kpiIdentifier": "PACKETS_DROPPED",
          "categoryIdentifier": "Network Utilization",
          "applicationIdentifier": null,
          "serviceIdentifiers": [
            "Supervisor_Controller Service",
            "Event_Detector Service",
            "UI_Service",
            "Redis",
            "CC_Service"
          ],
          "clusterIdentifiers": [
            "a5c57ef5-4018-41b8-b727-27c8f8376c0e"
          ],
          "collectionInterval": 60,
          "value": "0.0",
          "kpiType": "Core",
          "groupAttribute": "ALL",
          "groupIdentifier": null,
          "watcherValue": null,
          "errorCode": null,
          "clusterOperation": null,
          "aggLevelInMins": 1,
          "error": false,
          "kpiGroup": false,
          "discovery": false,
          "maintenanceExcluded": false,
          "@timestamp": "2022-05-01T01:32:00.000Z"
        }

```

Following is the transform job configuration.

```auto
curl -u admin:admin -XPUT "http://XXX.XXX.XX.XXX9201/_plugins/_transform/my-array-job-2" -H 'Content-type: application/json' -d'
{
    "transform": {
        "schedule": {
            "interval": {
                "start_time": 1659705000000,
                "period": 1,
                "unit": "Minutes"
            }
        },
        "metadata_id": null,
        "updated_at": 1659456180000,
        "enabled": true,
        "enabled_at": 1659457620000,
        "description": "",
        "source_index": "collated_txn_health_2022.05",
        "data_selection_query": {
            "match_all": {
                "boost": 1
            }
          },
        "target_index": "transform_collated_txn_health_2022.05",
        "page_size": 1000,
        "groups": [
            {
                "date_histogram": {
                    "fixed_interval": "1m",
                    "source_field": "@timestamp",
                    "target_field": "@timestamp",
                    "timezone": "Asia/Calcutta"
                }
            },
            {
                "terms": {
                    "source_field": "clusterIdentifiers",
                    "target_field": "clusterIdentifiers"
                }
            },
            {
                "terms": {
                    "source_field": "serviceIdentifiers",
                    "target_field": "serviceIdentifiers"
                }
            },
            {
                "terms": {
                    "source_field": "compInstanceIdentifier",
                    "target_field": "compInstanceIdentifier"
                }
            },
            {
                "terms": {
                    "source_field": "agentIdentifier",
                    "target_field": "agentIdentifier"
                }
            }
        ],
        "aggregations": {
            "count_@timestamp": {
                "value_count": {
                    "field": "@timestamp"
                }
            }
        }
    }
}'

```

Following are the events from the transform index.

```auto
      {
        "_index": "transform_heal_collated_txn_health_2022.05",
        "_type": "_doc",
        "_id": "ybK0McQ9NZrt9xdo9iWKbA",
        "_score": 1,
        "_source": {
          "transform._id": "my-array-job-2",
          "transform._doc_count": 2,
          "@timestamp": 1651365120000,
          "clusterIdentifiers": "a5c57ef5-4018-41b8-b727-27c8f8376c0e",
          "serviceIdentifiers": "Redis",
          "compInstanceIdentifier": "a5c57ef5-4018-41b8-b727-27c8f8376c0e",
          "agentIdentifier": "ComponentAgent-170",
          "count_@timestamp": 2
        }
      },
      {
        "_index": "transform_heal_collated_txn_health_2022.05",
        "_type": "_doc",
        "_id": "Wf-4KwnFaYuw9bL-V-9WEQ",
        "_score": 1,
        "_source": {
          "transform._id": "my-array-job-2",
          "transform._doc_count": 2,
          "@timestamp": 1651365120000,
          "clusterIdentifiers": "a5c57ef5-4018-41b8-b727-27c8f8376c0e",
          "serviceIdentifiers": "Redis_Server Service",
          "compInstanceIdentifier": "a5c57ef5-4018-41b8-b727-27c8f8376c0e",
          "agentIdentifier": "ComponentAgent-170",
          "count_@timestamp": 2
        }

```

It would be a great help if somebody suggest me with solution.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 6, 2022, 5:33am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/2 "2022-08-06T05:33:50Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 7, 2022, 11:05pm UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/3 "2022-08-07T23:05:38Z")

</div>

You will need to ask aws, their implementation of this is entirely different to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![Akshay\_Kulkarni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_kulkarni/32/101037_2.png) [@Akshay\_Kulkarni](https://discuss.elastic.co/u/Akshay_Kulkarni)\
**Post date:** [August 8, 2022, 9:04am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/4 "2022-08-08T09:04:40Z")

</div>

Thanks @warkolm for the reply.

Opensearch community is that so active as elastic. Have posted the same, but they have kept my post in pending state.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 8, 2022, 10:24am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/6 "2022-08-08T10:24:16Z")

</div>

So you should switch to the official elasticsearch. 😉

---

<div class="post-metadata">

**Author:** ![Akshay\_Kulkarni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_kulkarni/32/101037_2.png) [@Akshay\_Kulkarni](https://discuss.elastic.co/u/Akshay_Kulkarni)\
**Post date:** [August 8, 2022, 10:27am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/7 "2022-08-08T10:27:22Z")

</div>

@ [dadoonet](https://discuss.elastic.co/u/dadoonet)

Im using Elasticsearch since last 2 years, 😃

Its time to explore new things.

By the way I have solved above issue with painless script, will post after few testing.

Thanks.

---

<div class="post-metadata">

**Author:** ![Akshay\_Kulkarni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_kulkarni/32/101037_2.png) [@Akshay\_Kulkarni](https://discuss.elastic.co/u/Akshay_Kulkarni)\
**Post date:** [August 8, 2022, 11:33am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/8 "2022-08-08T11:33:47Z")

</div>

Have solved the issue with following painless script. Which help to transform array fields in opensearch.

```auto
PUT _plugins/_transform/my-array-job-2
{
  
  "transform": {
        "schedule": {
            "interval": {
                "start_time": 1659705000000,
                "period": 1,
                "unit": "Minutes"
            }
        },
        "metadata_id": null,
        "updated_at": 1659456180000,
        "enabled": true,
        "enabled_at": 1659457620000,
        "description": "",
        "source_index": "heal_collated_txn_heal_health_2022.05_reindex",
        "target_index": "transform_heal_collated_txn_heal_health_2022.05",
        "page_size": 1000,
        "groups": [
            {
                "date_histogram": {
                    "fixed_interval": "1m",
                    "source_field": "@timestamp",
                    "target_field": "@timestamp",
                    "timezone": "Asia/Calcutta"
                }
            },    
            {
                "terms": {
                    "source_field": "kpiIdentifier",
                    "target_field": "kpiIdentifier"
                }
            },
            {
                "terms": {
                    "source_field": "clusterIdentifiers",
                    "target_field": "clusterIdentifiers"
                }
            }            
        ],
    "aggregations": { 
      "count_@timestamp": {
          "value_count": {
           "field": "@timestamp"
         }
       },
      "count_agentIdentifier": {
          "value_count": {
              "field": "agentIdentifier"
          }
      },
      "sum_value": {
          "sum": {
              "field": "value"
          }
      },
      "max_value": {
          "max": {
              "field": "value"
          }
      },
      "avg_value": {
          "avg": {
              "field": "value"
          }
      },
      "count_value": {
          "value_count": {
              "field": "value"
          }
      },
      "percentiles_value": {
          "percentiles": {
              "field": "value",
             "percents": [
                  95
              ],
              "keyed": true,
              "tdigest": {
                  "compression": 100
              }
          }
      },
      "serviceIdentifiers": {
        "scripted_metric": {
          "init_script": "state.docs = []", 
          "map_script": """ 
            Map span = [
              'url':doc['serviceIdentifiers']
            ];
            state.docs.add(span)
          """,
          "combine_script": "return state.docs;", 
          "reduce_script": """ 
            def all_docs = [];
            for (s in states) {
              for (span in s) {
                all_docs.add(span);
              }
            }
            def size = all_docs.size();
            def serviceIdentifiers_1 = all_docs[0]['url'];
            def ret = new HashMap();
            ret['serviceIdentifiers'] = serviceIdentifiers_1;
            return ret;
          """
        }
      }
    }
  }
}

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d0086f673763e0a33160d51fbaf20d5467769cb2.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2022, 11:34am UTC](https://discuss.elastic.co/t/opensearch-transform-splitting-array-values-to-new-events/311575/10 "2022-09-05T11:34:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.