Copy multiple fields from Transform to entity

[xbeumala (Xavi)]

Hi all,

I have an analytics system and for every user action I log an event. The document contains event specific fields but also session level properties.

I've created a transform to create a user centric entity. on the transform I want to copy several values from the last event document to the transform document.

I've got the code to find the last Document

    "latest_doc": {
        "scripted_metric": {
          "init_script": "state.timestamp_latest = 0L; state.last_doc = ''", 
          "map_script": """ 
            def current_date = doc['ts'].getValue().toInstant().toEpochMilli();
            if (current_date > state.timestamp_latest)
            {state.timestamp_latest = current_date;
            state.last_doc = new HashMap(params['_source']);}
          """,
          "combine_script": "return state", 
          "reduce_script": """ 
            def last_doc = '';
            def timestamp_latest = 0L;
            for (s in states) {if (s.timestamp_latest > (timestamp_latest))
            {timestamp_latest = s.timestamp_latest; last_doc = s.last_doc;}}
            return last_doc
          """
        }
      }

The problem is that I need 4-6 properties from the last document. Is there a recomended way so I don't have to duplicate 5-6 times the code above?

many thanks

[Hendrik_Muhs]

The code you posted copies the full document, that means if I understand correctly the properties you are interested in are part of that.

As you only want to have those, you can get rid of the other properties in an ingest pipeline using a remove processor. The ones of interest can be moved/renamed using rename.

Other alternatives I can think of:

A stored scripts which can be specified by id in scripted metric.

A scripted metric that only copies the fields of interest:

        "scripted_metric": {
          "init_script": "state.join = new HashMap()",
          "map_script": "String[] fields = new String[] {'a', 'b', 'c'}; for (e in fields) { if (doc.containsKey(e)) {state.join.put(e, doc[e])}}",
          "combine_script": "return state.join",
          "reduce_script": "String[] fields = new String[] {'a', 'b', 'c'}; Map j=new HashMap(); for (s in states) {for (e in fields) { if (s.containsKey(e)) {j.put(e, s[e].get(0))}}} return j;"
        }
      }

Hope that helps!