Memory for Facet query with filter

Kartavya · July 5, 2012, 11:09am

Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70 days.
I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal

kimchy · July 5, 2012, 10:42pm

A facet query (or a facet filter) with an additional filter do not load any
field to memory, since they do not "run" on a specific field.

On Thu, Jul 5, 2012 at 1:09 PM, Pulkit Agrawal pulkitdotcom@gmail.comwrote:

Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70 days.
I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal

Kartavya · July 23, 2012, 6:33am

Hi Shay,

Thanks for your prompt reply.

I am pasting facet query with filters on date.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND
Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt":
{"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}}}}

This query loads the whole index irrespective of date filter range.

Please have a look on the query and let me know what's wrong with that.

Thanks,
Pulkit Agrawal

On Fri, Jul 6, 2012 at 4:12 AM, Shay Banon kimchy@gmail.com wrote:

A facet query (or a facet filter) with an additional filter do not load
any field to memory, since they do not "run" on a specific field.

On Thu, Jul 5, 2012 at 1:09 PM, Pulkit Agrawal pulkitdotcom@gmail.comwrote:

Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70
days. I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal

dadoonet · July 23, 2012, 6:36am

Apply a filter on the facet also !
It will load only what you need to see.

HTH
David
Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 08:33, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Hi Shay,

Thanks for your prompt reply.

I am pasting facet query with filters on date.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt": {"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}}}}

This query loads the whole index irrespective of date filter range.

Please have a look on the query and let me know what's wrong with that.

Thanks,
Pulkit Agrawal

On Fri, Jul 6, 2012 at 4:12 AM, Shay Banon kimchy@gmail.com wrote:
A facet query (or a facet filter) with an additional filter do not load any field to memory, since they do not "run" on a specific field.

On Thu, Jul 5, 2012 at 1:09 PM, Pulkit Agrawal pulkitdotcom@gmail.com wrote:
Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70 days. I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal

Kartavya · July 23, 2012, 6:53am

Hi David,

You mean a query like below will fix the problem.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND
Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt":
{"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}},
"facet_filter": {"range": {

      "from": "2012-7-8T00:00:00",
      "to": "2012-7-19T23:59:59"

    }

}
}}

Please review the query and let me know if it is proper or not.

Thanks,
Pulkit Agrawal

On Mon, Jul 23, 2012 at 12:06 PM, David Pilato david@pilato.fr wrote:

Apply a filter on the facet also !
It will load only what you need to see.

HTH
David
Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 08:33, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Hi Shay,

Thanks for your prompt reply.

I am pasting facet query with filters on date.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt": {"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}}}}

This query loads the whole index irrespective of date filter range.

Please have a look on the query and let me know what's wrong with that.

Thanks,
Pulkit Agrawal

On Fri, Jul 6, 2012 at 4:12 AM, Shay Banon kimchy@gmail.com wrote:

A facet query (or a facet filter) with an additional filter do not load any field to memory, since they do not "run" on a specific field.

On Thu, Jul 5, 2012 at 1:09 PM, Pulkit Agrawal pulkitdotcom@gmail.com wrote:

Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70 days. I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal

dadoonet · July 23, 2012, 7:42am

Yes but with the same filter. Something like:

"facets":{
"MsgData":{
"terms":{ "field":"MsgData", "size":20 }
},
"facet_filter":{
"range":{

"CreatedAt":{
"from":"2012-7-8T00:00:00",
"to":"2012-7-19T23:59:59"
}
}
}
}

See Elasticsearch Platform — Find real-time answers at scale | Elastic
http://www.elasticsearch.org/guide/reference/api/search/facets/index.html

David.

Le 23 juillet 2012 à 08:53, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Hi David,

You mean a query like below will fix the problem.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND
Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt":
{"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}},
"facet_filter": {"range": {
      "from": "2012-7-8T00:00:00",
      "to": "2012-7-19T23:59:59"

    }
}
}}

Please review the query and let me know if it is proper or not.

Thanks,
Pulkit Agrawal

On Mon, Jul 23, 2012 at 12:06 PM, David Pilato david@pilato.fr wrote:

Apply a filter on the facet also !
It will load only what you need to see.

HTH
David
Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 08:33, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Hi Shay,

Thanks for your prompt reply.

I am pasting facet query with filters on date.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND
Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt":
{"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}}}}

This query loads the whole index irrespective of date filter range.

Please have a look on the query and let me know what's wrong with that.

Thanks,
Pulkit Agrawal

On Fri, Jul 6, 2012 at 4:12 AM, Shay Banon kimchy@gmail.com wrote:

A facet query (or a facet filter) with an additional filter do not load any
field to memory, since they do not "run" on a specific field.

On Thu, Jul 5, 2012 at 1:09 PM, Pulkit Agrawal pulkitdotcom@gmail.com
wrote:

Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70 days.
I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

Kartavya · July 23, 2012, 9:25am

David as per your suggestion I drafted this query

curl -XGET 'http://localhost:9200/new_index/new_indext/_search' -d '{
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": "Hero"
}
}
],
"minimum_number_should_match": 1
}},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 20
}
},
"facet_filter": {"range": {"CreatedAt": {"from":
"2012-7-8T00:00:00","to": "2012-7-19T23:59:59"}}}
}

}'

But it returns FacetPhaseExecutionException[Facet [facet_filter]: no
ranges defined for range facet]; }]","status":500}

What's wrong with this query?

Thanks

On Mon, Jul 23, 2012 at 1:12 PM, David Pilato david@pilato.fr wrote:

Yes but with the same filter. Something like:

"facets":{
"MsgData":{
"terms":{ "field":"MsgData", "size":20 }
},
"facet_filter":{
"range":{

"CreatedAt":{
"from":"2012-7-8T00:00:00",
"to":"2012-7-19T23:59:59"
}
}
}
}

See
Elasticsearch Platform — Find real-time answers at scale | Elastic

David.

Le 23 juillet 2012 à 08:53, Pulkit Agrawal pulkitdotcom@gmail.com a écrit
:
Hi David,

You mean a query like below will fix the problem.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND
Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt":
{"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}},
"facet_filter": {"range": {
      "from": "2012-7-8T00:00:00",
      "to": "2012-7-19T23:59:59"

    }
}
}}

Please review the query and let me know if it is proper or not.

Thanks,
Pulkit Agrawal

On Mon, Jul 23, 2012 at 12:06 PM, David Pilato david@pilato.fr wrote:

Apply a filter on the facet also !
It will load only what you need to see.

HTH
David
Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 08:33, Pulkit Agrawal pulkitdotcom@gmail.com a
écrit :

Hi Shay,

Thanks for your prompt reply.

I am pasting facet query with filters on date.

{"query":
{"filtered":
{"query":
{"bool":
{"should":[{"field":{"MsgData":"(royal AND
Marriage)"}}],"minimum_number_should_match":1} }
,"filter":{"range":{"CreatedAt":
{"from":"2012-7-8T00:00:00","to":"2012-7-19T23:59:59"}}}},
"facets":{"MsgData":{"terms":{"field":"MsgData","size":20}}}}

This query loads the whole index irrespective of date filter range.

Please have a look on the query and let me know what's wrong with that.

Thanks,
Pulkit Agrawal

On Fri, Jul 6, 2012 at 4:12 AM, Shay Banon kimchy@gmail.com wrote:

A facet query (or a facet filter) with an additional filter do not load
any field to memory, since they do not "run" on a specific field.

On Thu, Jul 5, 2012 at 1:09 PM, Pulkit Agrawal pulkitdotcom@gmail.com
wrote:

Hi

I am running facet query with date range filter.

It consume the same memory whether I am running it for 7 days, or 70
days. I know facet loads the index in memory but What if I am using filter.
Does it load complete index irrespective of filtered data?

Any help would be highly appreciated.

Regards,
Pulkit Agrawal
--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

Clinton_Gormley · July 23, 2012, 9:29am

Hi Pukit

    "facets": {
        "MsgData": {
            "terms": {
                "field": "MsgData",
                "size": 20
            }
        },
        "facet_filter": {"range": {"CreatedAt": {"from":

"2012-7-8T00:00:00","to": "2012-7-19T23:59:59"}}}
}

}'

Your facet_filter should be at the same level as 'terms', not as
'MsgData'. See

clint

Kartavya · July 23, 2012, 9:32am

Thanks a lot clint.

It is working now.

Regards,
Pulkit Agrawal

On Mon, Jul 23, 2012 at 2:59 PM, Clinton Gormley clint@traveljury.com wrote:

Hi Pukit
    "facets": {
        "MsgData": {
            "terms": {
                "field": "MsgData",
                "size": 20
            }
        },
        "facet_filter": {"range": {"CreatedAt": {"from":
"2012-7-8T00:00:00","to": "2012-7-19T23:59:59"}}}
}

}'
Your facet_filter should be at the same level as 'terms', not as
'MsgData'. See
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

Kartavya · July 23, 2012, 1:47pm

Hi

Now, I have two facet queries

Query 1:
curl -XGET 'http://localhost:9200/new_index/new_indext/_search' -d '{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": "google"
}
}
],
"minimum_number_should_match": 1
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-6-12T00:00:00",
"to": "2012-7-23T23:59:59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100
}
}
}
}'

Query 2:

curl -XGET 'http://localhost:9200/new_index/new_indext/_search' -d
'{"query": {"bool": {"should": [
{
"field": {
"MsgData": "google"
}
}
],
"minimum_number_should_match": 1
}},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 20
},
"facet_filter": {"range": {"CreatedAt": {"from":
"2012-6-12T00:00:00","to": "2012-7-23T23:59:59"}}}
}
}

}'

Are they differ in performance and memory usage? Which one would be
fast and using less heap space?

Thanks,
Pulkit Agrawal

On Mon, Jul 23, 2012 at 3:02 PM, Pulkit Agrawal pulkitdotcom@gmail.com wrote:

Thanks a lot clint.

It is working now.

Regards,
Pulkit Agrawal

On Mon, Jul 23, 2012 at 2:59 PM, Clinton Gormley clint@traveljury.com wrote:
Hi Pukit
    "facets": {
        "MsgData": {
            "terms": {
                "field": "MsgData",
                "size": 20
            }
        },
        "facet_filter": {"range": {"CreatedAt": {"from":
"2012-7-8T00:00:00","to": "2012-7-19T23:59:59"}}}
}

}'
Your facet_filter should be at the same level as 'terms', not as
'MsgData'. See
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint