Merging documents based on matched fields values

Elastic Stack Elasticsearch
2

This can be achieved by grouping docs using job_name and build_number and than aggregate the docs with a scripted_metric. A perfect fit for that is a composite aggregation. job_name and build_number are the sources, the scripted_metric that merges the fields your aggregation.

However, you want to visualize on top of that, so I think you want to have the merged documents in an index. That's where transform comes into play. It basically runs a composite aggregation, takes the results and writes the output into a new index. You can set it up to run continuously, so it will consume your new incoming data (you need a timestamp for that, but I guess you have that). In transform job_name and build_number would be your group_by.

Regarding the scripted_metric aggregation, please have a look here. The examples are not transform specific, but work in aggregations.

Here is a more specific example, I found:

"scripted_metric": {
          "init_script": "state.join = new HashMap()",
          "map_script": "String[] fields = new String[] {'name', 'best_friend', 'hobby'}; for (e in fields) { if (doc.containsKey(e)) {state.join.put(e, doc[e])}}",
          "combine_script": "return state.join",
          "reduce_script": "String[] fields = new String[] {'name', 'best_friend', 'hobby'}; Map j=new HashMap(); for (s in states) {for (e in fields) { if (s.containsKey(e)) {j.put(e, s[e].get(0))}}} return j;"
        }
1 Like