Progress on Hive "Push Down Filtering"

Has there been any progress on the "Push Down Filtering" mentioned by
Costin? (


)

Right now I am working around this by creating a lot of specific table
mappings to maintain performance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

There are two aspects when dealing with large tables.

  1. Projection

The table mapping/definition is necessary as it indicates what information is needed - a small mapping excludes a lot of
unnecessary data.

  1. Push Down filtering

Unfortunately there hasn't been much happening on this front since the functionality is fairly restricted and not really
pluggable especially when
dealing with non-HDFS resources. The ORC support has improved things a bit however it's still early days...

Cheers,

On 12/4/14 7:41 AM, James Andrew-Smith wrote:

Has there been any progress on the "Push Down Filtering" mentioned by Costin?
(http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542)

Right now I am working around this by creating a lot of specific table mappings to maintain performance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54802382.5060704%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi Costin,

Thank you for the rapid response - just wanted to say I appreciate the
Hadoop install works so easily just as advertised.

Shame about the push down filter but this is what I expected.

I'll focus on keeping the projection as lightweight as possible - on that
note - I started another thread (
https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg) about
using aggregation queries (which I am a huge fan of) via Hive. Is this in
possible/in the pipeline?

Cheers
James

On Thursday, 4 December 2014 20:04:17 UTC+11, Costin Leau wrote:

Hi,

There are two aspects when dealing with large tables.

  1. Projection

The table mapping/definition is necessary as it indicates what information
is needed - a small mapping excludes a lot of
unnecessary data.

  1. Push Down filtering

Unfortunately there hasn't been much happening on this front since the
functionality is fairly restricted and not really
pluggable especially when
dealing with non-HDFS resources. The ORC support has improved things a bit
however it's still early days...

Cheers,

On 12/4/14 7:41 AM, James Andrew-Smith wrote:

Has there been any progress on the "Push Down Filtering" mentioned by
Costin?
(
http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542)

Right now I am working around this by creating a lot of specific table
mappings to maintain performance.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com

<
https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sorry I missed the other thread; I'll respond here.

Yes, that's in the pipeline - see issue #276.

As you pointed out with push down this could potentially be done automatically...

Cheers,

P.S. Thanks for the kind words. If you encounter issues/bug or have suggestions, please keep the feedback coming.

On 12/4/14 12:06 PM, James Andrew-Smith wrote:

Hi Costin,

Thank you for the rapid response - just wanted to say I appreciate the Hadoop install works so easily just as advertised.

Shame about the push down filter but this is what I expected.

I'll focus on keeping the projection as lightweight as possible - on that note - I started another thread
(https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg
https://groups.google.com/forum/m/#!topic/elasticsearch/-3Lbdw5Wigg) about using aggregation queries (which I am a
huge fan of) via Hive. Is this in possible/in the pipeline?

Cheers
James

On Thursday, 4 December 2014 20:04:17 UTC+11, Costin Leau wrote:

Hi,

There are two aspects when dealing with large tables.

1. Projection

The table mapping/definition is necessary as it indicates what information is needed - a small mapping excludes a
lot of
unnecessary data.

2. Push Down filtering

Unfortunately there hasn't been much happening on this front since the functionality is fairly restricted and not
really
pluggable especially when
dealing with non-HDFS resources. The ORC support has improved things a bit however it's still early days...

Cheers,


On 12/4/14 7:41 AM, James Andrew-Smith wrote:
> Has there been any progress on the "Push Down Filtering" mentioned by Costin?
> (http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542
<http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/#comment-1169375542>)

>
>
> Right now I am working around this by creating a lot of specific table mappings to maintain performance.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
>elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com>
> <https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/bd8151a0-af89-4617-9efe-aa738e70862c%40googlegroups.com?utm_medium=email&utm_source=footer>>.

> For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/612d65f5-90f9-4ac5-93ca-5cf8fe6fb03f%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54805358.5020208%40gmail.com.
For more options, visit https://groups.google.com/d/optout.