# ES Hadoop push down aggregations or not?

**URL:** https://discuss.elastic.co/t/es-hadoop-push-down-aggregations-or-not/277514
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [July 1, 2021, 6:24am UTC](https://discuss.elastic.co/t/es-hadoop-push-down-aggregations-or-not/277514 "2021-07-01T06:24:07Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)
#### Post date: [July 1, 2021, 6:24am UTC](https://discuss.elastic.co/t/es-hadoop-push-down-aggregations-or-not/277514/1 "2021-07-01T06:24:07Z")

</div>

I am on ELK 7.13.0 and I have a super very slow query:

```auto
events_df = es_reader.load("priam_unified_host-{0}/unified-host".format(date))

counts_df = events_df.where('EventID == 4688').groupBy(col("LogHost"),col("ProcessName")).count()
counts_df.explain(extended=True)

```

This is the execution plan:

```auto

== Physical Plan ==
*(2) HashAggregate(keys=[LogHost#1347, ProcessName#1354], functions=[count(1)], output=[LogHost#1347, ProcessName#1354, count#1408L])
+- Exchange hashpartitioning(LogHost#1347, ProcessName#1354, 200), ENSURE_REQUIREMENTS, [id=#258]
   +- *(1) HashAggregate(keys=[LogHost#1347, ProcessName#1354], functions=[partial_count(1)], output=[LogHost#1347, ProcessName#1354, count#1413L])
      +- *(1) Project [LogHost#1347, ProcessName#1354]
         +- *(1) Filter (isnotnull(EventID#1345L) AND (EventID#1345L = 4688))
            +- *(1) Scan ElasticsearchRelation(Map(es.inferschema -> false, es.net.http.auth.user -> elastic, es.net.ssl.cert.allow.self.signed -> true, es.net.http.auth.pass -> 123456, es.read.field.as.array.include -> tags, es.resource -> priam_unified_host-2021-05-03/unified-host, es.nodes -> elasticsearch:9200, es.net.ssl -> true),org.apache.spark.sql.SQLContext@515d109c,None) [LogHost#1347,ProcessName#1354,EventID#1345L] PushedFilters: [IsNotNull(EventID), EqualTo(EventID,4688)], ReadSchema: struct<LogHost:string,ProcessName:string,EventID:bigint>

```

Does it mean that it is effectively scanning the index (via scrolling???) and not leveraging the Aggregates constructs in ES?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 29, 2021, 6:25am UTC](https://discuss.elastic.co/t/es-hadoop-push-down-aggregations-or-not/277514/2 "2021-07-29T06:25:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

### Author: ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [December 8, 2021, 2:09pm UTC](https://discuss.elastic.co/t/es-hadoop-push-down-aggregations-or-not/277514/3 "2021-12-08T14:09:10Z")

</div>

You are correct. Es-hadoop supports pushing down most filter clauses but not aggregations/group bys. This functionality has just recently been added to spark and has not made it into es-hadoop yet -- [Support Spark's Datasource V2 API · Issue #1801 · elastic/elasticsearch-hadoop · GitHub](https://github.com/elastic/elasticsearch-hadoop/issues/1801).