How do I get Elasticsearch max_clause_count to take effect?


(jbattle) #1

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settings with this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin

Termsの上限数について
(Karussell) #2

The setting should go in the elasticsearch config (there where the
analyzers go), not in the index settings.

And: are you really need more than 1024 clauses (can be bad in terms
of RAM)? Why not using a boolean filter or a terms filter? Or are you
hitting the limit because of some query expansion of a different
query?

Peter.


(Shay Banon) #3

It can only be set on the config file, not on index creation settings. This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbattle@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settings with this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin

(jbattle) #4

Peter,

Thanks for your response. I am using a terms filter where I'm
filtering against thousands of accounts. It appears this is why I was
hitting the 1024 limit.

  • justin

On Dec 13, 3:23 am, Karussell tableyourt...@googlemail.com wrote:

The setting should go in the elasticsearch config (there where the
analyzers go), not in the index settings.

And: are you really need more than 1024 clauses (can be bad in terms
of RAM)? Why not using a boolean filter or a terms filter? Or are you
hitting the limit because of some query expansion of a different
query?

Peter.


(jbattle) #5

Shay,

Thank you for your response. I have done this and am no longer bound
to the 1024 limit.

Much Thanks,

  • justin

On Dec 13, 6:27 am, Shay Banon kim...@gmail.com wrote:

It can only be set on the config file, not on index creation settings. This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbat...@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settingswith this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin

(Shay Banon) #6

Terms filter is not bounded by the max clause count, only the bool query.

On Tue, Dec 13, 2011 at 8:34 PM, jbattle jbattle@gmail.com wrote:

Shay,

Thank you for your response. I have done this and am no longer bound
to the 1024 limit.

Much Thanks,

  • justin

On Dec 13, 6:27 am, Shay Banon kim...@gmail.com wrote:

It can only be set on the config file, not on index creation settings.
This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbat...@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settingswith this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin

(icenine18) #7

Is the setting being referred to in the elasticsearch yaml file?:

query:
bool:
max_clause_count: 10240

I've increased the value of this setting on my cluster, but am still
receiving the clause limit error using the Java API for certain
queries. thanks.

On Dec 13, 5:31 pm, Shay Banon kim...@gmail.com wrote:

Terms filter is not bounded by the max clause count, only the bool query.

On Tue, Dec 13, 2011 at 8:34 PM, jbattle jbat...@gmail.com wrote:

Shay,

Thank you for your response. I have done this and am no longer bound
to the 1024 limit.

Much Thanks,

  • justin

On Dec 13, 6:27 am, Shay Banon kim...@gmail.com wrote:

It can only be set on the config file, not on index creation settings.
This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbat...@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settingswiththis line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin

(Karussell) #8

Is the setting being referred to in the elasticsearch yaml file?:

Yes either yaml or json. I think it should be

index:
query:
bool:
max_clause_count: 10240

Peter.

On 14 Dez., 00:43, icenine18 bwitha...@yahoo.com wrote:

Is the setting being referred to in the elasticsearch yaml file?:

query:
bool:
max_clause_count: 10240

I've increased the value of this setting on my cluster, but am still
receiving the clause limit error using the Java API for certain
queries. thanks.


(narinder.izap) #9

Following is my query that I am using :

{
"query": {
"bool": {
"must": [
{
"text": {
"searchable.title": {
"type": "phrase_prefix",
"query": "a",
"analyzer": "simple"
}
}
}
]
}
}
}

Its being crashed when I am querying single characters as "a","b","c","v"
etc. but not all character, like "z" is giving the response. and the
exception in the former cases is :

Query Failed [Failed to execute main query]]; nested: TooManyClauses[maxClauseCount is set to 1024]; }]

So now I am trying to set the value of max_clause_count in the index as
follows:

index.query.bool.max_clause_count : 10240

but the index is failing to start if the setting is there in the yml file
and starts if I remove the setting.

So what can be the solution
??


(Clinton Gormley) #10

On Wed, 2012-01-11 at 02:57 -0800, Narinder Kaur wrote:

Following is my query that I am using :

{
"query": {
"bool": {
"must": [
{
"text": {
"searchable.title": {
"type": "phrase_prefix",
"query": "a",
"analyzer": "simple"
}
}
}
]
}
}
}

Don't do this.

phrase_prefix or prefix queries don't perform well when there are lots
of matching terms.

They have to:

  • load all terms beginning with $prefix
  • construct a query like term_1 OR term_2 OR .....

So yes, if you say "give me everything beginning with 'a'" then you are
going to have a LOT of terms, which will kill your memory and perform
very badly.

That's why there is a limit to the number of terms/clauses. It's there
to protect you.

The way to do this more efficiently is to analyse your text with ngrams
or edge-ngrams.

See this email for more:
http://elasticsearch-users.115913.n3.nabble.com/help-needed-with-the-query-tt3177477.html#a3178856

Its being crashed when I am querying single characters as
"a","b","c","v" etc. but not all character, like "z" is giving the
response. and the exception in the former cases is :

Query Failed [Failed to execute main query]]; nested: TooManyClauses[maxClauseCount is set to 1024]; }]

So now I am trying to set the value of max_clause_count in the index
as follows:

index.query.bool.max_clause_count : 10240

but the index is failing to start if the setting is there in the yml
file and starts if I remove the setting.

So what can be the solution
??


(Anirudh Modi) #11

Isn't there a way to continue to search even if the clause exceed, like instead of throwing an error, cant we just take the 1024 expanded clauses and continue with our search?
I am also facing the same issue, so wanted to know if we can somehow live with this is error?


(Christian Dahlqvist) #12

This is a very, very old thread. Please start a new one for you question.


(Petey Pab Pro) #13

Did you repost this question (about whether you can just continue the search with 1024 phrases instead of getting an error)? I'm curious as well..


(system) #14