Does transform job's groupBy order matter?

Ashfur · January 7, 2022, 9:14pm

When we specify multiple groupBys in a pivot transform job, does the order matter? For example, let's say we have a TermGroupBy by userId and a DateHistogramGroupBy by timestamp. Does it matter which is declared first?

In the Elasticsearch Java sdk, when we specify multiple groupby, the configuration is stored in an unordered HashMap, which implies that the order does not matter.

github.com

elastic/elasticsearch/blob/353117244e661653aa49dcdb67d149245c7be464/client/rest-high-level/src/main/java/org/elasticsearch/client/transform/transforms/pivot/GroupConfig.java#L175

    
      
          @Override
          public String toString() {
              return Strings.toString(this, true, true);
          }
          
          
public static Builder builder() {
              return new Builder();
          }
          
          
public static class Builder {
              private final Map<String, SingleGroupSource> groups = new HashMap<>();
          
          
    /**
               * Add a new grouping to the builder
               * @param name The name of the resulting grouped field
               * @param group The type of grouping referenced
               * @return The {@link Builder} with a new grouping entry added
               */
              public Builder groupBy(String name, SingleGroupSource group) {
                  groups.put(name, group);
                  return this;

I would just like to get some clarity on this.

Thanks.

Hendrik_Muhs · January 10, 2022, 8:01am

Thanks, for the info. This looks like a bug in the client to me. If you are using http directly the behavior is different: The configuration gets stored using the order you specified in the request. If you look into source you pasted you see that the parsing code uses a linked hashmap to preserve order. The builder should do the same, I will file a bug.

TL/DR

Order can make a difference as documented in the "Transform at scale" guide. Since 7.15 transform internally re-orders groupings to auto-tune for performance. However if 2 groupings have the same order type (e.g. 2 term groupings on a non-runtime field), we respect the given order.

system · February 7, 2022, 8:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.