# Progress on Hive "Push Down Filtering" **URL:** \ **Category:** Elasticsearch\ **Created:** [December 4, 2014, 5:41am UTC](https://discuss.elastic.co/t/progress-on-hive-push-down-filtering/21076 "2014-12-04T05:41:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![James\_Andrew\_Smith](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@James\_Andrew\_Smith](https://discuss.elastic.co/u/James_Andrew_Smith)\ **Post date:** [December 4, 2014, 5:41am UTC](https://discuss.elastic.co/t/progress-on-hive-push-down-filtering/21076/1 "2014-12-04T05:41:36Z")
Has there been any progress on the "Push Down Filtering" mentioned by Costin? ( > **[Tableau to ElasticSearch (Querying ElasticSearch with Hive SQL and Hadoop)](http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542)** > > A Step-by-Step / How-To Guide for: Tableau -\> Hive SQL -\> Hadoop -\> Map/Reduce -\> ES. With no Full House references in sight! ) Right now I am working around this by creating a lot of specific table mappings to maintain performance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 4, 2014, 9:04am UTC](https://discuss.elastic.co/t/progress-on-hive-push-down-filtering/21076/2 "2014-12-04T09:04:02Z")
Hi, There are two aspects when dealing with large tables. 1. Projection The table mapping/definition is necessary as it indicates what information is needed - a small mapping excludes a lot of unnecessary data. 1. Push Down filtering Unfortunately there hasn't been much happening on this front since the functionality is fairly restricted and not really pluggable especially when dealing with non-HDFS resources. The ORC support has improved things a bit however it's still early days... Cheers, On 12/4/14 7:41 AM, James Andrew-Smith wrote: > Has there been any progress on the "Push Down Filtering" mentioned by Costin? > ([Connecting Tableau to ElasticSearch (READ: How to query ElasticSearch with Hive SQL and Hadoop) | Ryan Robitaille](http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542)) > > Right now I am working around this by creating a lot of specific table mappings to maintain performance. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54802382.5060704%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/54802382.5060704%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![James\_Andrew\_Smith](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@James\_Andrew\_Smith](https://discuss.elastic.co/u/James_Andrew_Smith)\ **Post date:** [December 4, 2014, 10:06am UTC](https://discuss.elastic.co/t/progress-on-hive-push-down-filtering/21076/3 "2014-12-04T10:06:13Z")
Hi Costin, Thank you for the rapid response - just wanted to say I appreciate the Hadoop install works so easily just as advertised. Shame about the push down filter but this is what I expected. I'll focus on keeping the projection as lightweight as possible - on that note - I started another thread ( [Redirecting to Google Groups](https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg)) about using aggregation queries (which I am a huge fan of) via Hive. Is this in possible/in the pipeline? Cheers James On Thursday, 4 December 2014 20:04:17 UTC+11, Costin Leau wrote: > Hi, > > There are two aspects when dealing with large tables. > > 1. Projection > > The table mapping/definition is necessary as it indicates what information > is needed - a small mapping excludes a lot of > unnecessary data. > > 1. Push Down filtering > > Unfortunately there hasn't been much happening on this front since the > functionality is fairly restricted and not really > pluggable especially when > dealing with non-HDFS resources. The ORC support has improved things a bit > however it's still early days... > > Cheers, > > On 12/4/14 7:41 AM, James Andrew-Smith wrote: > > > Has there been any progress on the "Push Down Filtering" mentioned by > > Costin? > > ( > > [Connecting Tableau to ElasticSearch (READ: How to query ElasticSearch with Hive SQL and Hadoop) | Ryan Robitaille](http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542)) > > > Right now I am working around this by creating a lot of specific table > > mappings to maintain performance. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer)\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 4, 2014, 12:28pm UTC](https://discuss.elastic.co/t/progress-on-hive-push-down-filtering/21076/4 "2014-12-04T12:28:08Z")
Sorry I missed the other thread; I'll respond here. Yes, that's in the pipeline - see issue #276. As you pointed out with push down this could potentially be done automatically... Cheers, P.S. Thanks for the kind words. If you encounter issues/bug or have suggestions, please keep the feedback coming. On 12/4/14 12:06 PM, James Andrew-Smith wrote: > Hi Costin, > > Thank you for the rapid response - just wanted to say I appreciate the Hadoop install works so easily just as advertised. > > Shame about the push down filter but this is what I expected. > > I'll focus on keeping the projection as lightweight as possible - on that note - I started another thread > ([Redirecting to Google Groups](https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg) > [https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg](https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg)) about using aggregation queries (which I am a > huge fan of) via Hive. Is this in possible/in the pipeline? > > Cheers > James > > On Thursday, 4 December 2014 20:04:17 UTC+11, Costin Leau wrote: > > ``` > Hi, > > There are two aspects when dealing with large tables. > > 1. Projection > > The table mapping/definition is necessary as it indicates what information is needed - a small mapping excludes a > lot of > unnecessary data. > > 2. Push Down filtering > > Unfortunately there hasn't been much happening on this front since the functionality is fairly restricted and not > really > pluggable especially when > dealing with non-HDFS resources. The ORC support has improved things a bit however it's still early days... > > Cheers, > > On 12/4/14 7:41 AM, James Andrew-Smith wrote: > > Has there been any progress on the "Push Down Filtering" mentioned by Costin? > > (http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542 > ) > > > > > > > Right now I am working around this by creating a lot of specific table mappings to maintain performance. > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to > >elasticsearc...@googlegroups.com >. > > To view this discussion on the web visit > >https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com > > > >. > > > For more options, visithttps://groups.google.com/d/optout . > > -- > Costin > > ``` > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54805358.5020208%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/54805358.5020208%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:45am UTC](https://discuss.elastic.co/t/progress-on-hive-push-down-filtering/21076/5 "2017-07-06T00:45:59Z")