Transforming Indexes

Elastic Stack Elasticsearch
1

Transforming Indexes is considering the timestamp field under "_source" while comparing the data. How to exclude this field while comparison?

Below is the query I am using to transform the data from two indexes:

PUT /_transform/compare_indexes?pretty
{
"source" : {
"index" : [
"index2",
"index5"
],
"query" : {
"match_all" : { }
}
},
"dest" : {
"index" : "compare_indexes"
},
"pivot": {
"group_by": {
"name" : {
"terms" : {
"field" : "Name"
}
}

},
"aggregations" : {
  "compare" : { 
    "scripted_metric" : {
      "map_script" : "state.doc = new HashMap(params['_source'])", 
      "combine_script" : "return state", 
      "reduce_script" : """ 
        if (states.size() !=2 ) {
          return "document_mismatch"
        }
        else if (states.get(0).equals(states.get(1)))
        {
          return "match"
        }
        else {
          return "data_mismatch"
        }
        """
    }
  }
}

},
"frequency": "1m",
"sync": {
"time": {
"field": "@timestamp",
"delay": "1h"
}
}
}

2

You may remove some keys from the HashMap. Something like:

"map_script" : "state.doc = new HashMap(params['_source']);state.doc.remove('@timestamp');",

In addition, please edit the post to preformat the whole script using </> button.

3

Thanks Tomo I will try that and let you know.
Can you also help me like how to get the field which is not matching in the second if clause?
Thanks in Advance.

4

I suppose you have to check by iterating keys and values of the hashmap.

5

Thanks Tomo will let you know.

1 Like
6

This is working like a charm,
Thanks Tomo.
Any idea how can I make this transform continuous?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.