How do I get Elasticsearch max_clause_count to take effect?

Elastic Stack Elasticsearch
1

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settings with this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin
2

The setting should go in the elasticsearch config (there where the
analyzers go), not in the index settings.

And: are you really need more than 1024 clauses (can be bad in terms
of RAM)? Why not using a boolean filter or a terms filter? Or are you
hitting the limit because of some query expansion of a different
query?

Peter.

3

It can only be set on the config file, not on index creation settings. This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbattle@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settings with this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin
4

Peter,

Thanks for your response. I am using a terms filter where I'm
filtering against thousands of accounts. It appears this is why I was
hitting the 1024 limit.

  • justin

On Dec 13, 3:23 am, Karussell tableyourt...@googlemail.com wrote:

The setting should go in the elasticsearch config (there where the
analyzers go), not in the index settings.

And: are you really need more than 1024 clauses (can be bad in terms
of RAM)? Why not using a boolean filter or a terms filter? Or are you
hitting the limit because of some query expansion of a different
query?

Peter.

5

Shay,

Thank you for your response. I have done this and am no longer bound
to the 1024 limit.

Much Thanks,

  • justin

On Dec 13, 6:27 am, Shay Banon kim...@gmail.com wrote:

It can only be set on the config file, not on index creation settings. This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbat...@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settingswith this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin
6

Terms filter is not bounded by the max clause count, only the bool query.

On Tue, Dec 13, 2011 at 8:34 PM, jbattle jbattle@gmail.com wrote:

Shay,

Thank you for your response. I have done this and am no longer bound
to the 1024 limit.

Much Thanks,

  • justin

On Dec 13, 6:27 am, Shay Banon kim...@gmail.com wrote:

It can only be set on the config file, not on index creation settings.
This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbat...@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settingswith this line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin
7

Is the setting being referred to in the elasticsearch yaml file?:

query:
bool:
max_clause_count: 10240

I've increased the value of this setting on my cluster, but am still
receiving the clause limit error using the Java API for certain
queries. thanks.

On Dec 13, 5:31 pm, Shay Banon kim...@gmail.com wrote:

Terms filter is not bounded by the max clause count, only the bool query.

On Tue, Dec 13, 2011 at 8:34 PM, jbattle jbat...@gmail.com wrote:

Shay,

Thank you for your response. I have done this and am no longer bound
to the 1024 limit.

Much Thanks,

  • justin

On Dec 13, 6:27 am, Shay Banon kim...@gmail.com wrote:

It can only be set on the config file, not on index creation settings.
This
is because its a static setting in Lucene.

On Tue, Dec 13, 2011 at 1:19 AM, jbattle jbat...@gmail.com wrote:

I am currently setting the max_clause_count at the creation of my
index as follows:

"index":{
"type" :"new",
"bulk_size":"100000",
"bulk_timeout":"1000ms",
"query":{
"bool":{
"max_clause_count": 1000000
}
}
}

The setting is refelected when I check my index settings at
http://localhost:9200/myindex/_settingswiththis line:
index.query.bool.max_clause_count: "1000000"

Despite this, when I run a query, I get the below error stating that
the max_clause_count is the default 1024:
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

Any insight to why despite the setting showing the max_clause_count as
set to 1000000, that Elasticsearch is still throwing that error?

Thanks

  • justin
8

Is the setting being referred to in the elasticsearch yaml file?:

Yes either yaml or json. I think it should be

index:
query:
bool:
max_clause_count: 10240

Peter.

On 14 Dez., 00:43, icenine18 bwitha...@yahoo.com wrote:

Is the setting being referred to in the elasticsearch yaml file?:

query:
bool:
max_clause_count: 10240

I've increased the value of this setting on my cluster, but am still
receiving the clause limit error using the Java API for certain
queries. thanks.

9

Following is my query that I am using :

{
"query": {
"bool": {
"must": [
{
"text": {
"searchable.title": {
"type": "phrase_prefix",
"query": "a",
"analyzer": "simple"
}
}
}
]
}
}
}

Its being crashed when I am querying single characters as "a","b","c","v"
etc. but not all character, like "z" is giving the response. and the
exception in the former cases is :

Query Failed [Failed to execute main query]]; nested: TooManyClauses[maxClauseCount is set to 1024]; }]

So now I am trying to set the value of max_clause_count in the index as
follows:

index.query.bool.max_clause_count : 10240

but the index is failing to start if the setting is there in the yml file
and starts if I remove the setting.

So what can be the solution
??

10

On Wed, 2012-01-11 at 02:57 -0800, Narinder Kaur wrote:

Following is my query that I am using :

{
"query": {
"bool": {
"must": [
{
"text": {
"searchable.title": {
"type": "phrase_prefix",
"query": "a",
"analyzer": "simple"
}
}
}
]
}
}
}

Don't do this.

phrase_prefix or prefix queries don't perform well when there are lots
of matching terms.

They have to:

  • load all terms beginning with $prefix
  • construct a query like term_1 OR term_2 OR .....

So yes, if you say "give me everything beginning with 'a'" then you are
going to have a LOT of terms, which will kill your memory and perform
very badly.

That's why there is a limit to the number of terms/clauses. It's there
to protect you.

The way to do this more efficiently is to analyse your text with ngrams
or edge-ngrams.

See this email for more:
http://elasticsearch-users.115913.n3.nabble.com/help-needed-with-the-query-tt3177477.html#a3178856

Its being crashed when I am querying single characters as
"a","b","c","v" etc. but not all character, like "z" is giving the
response. and the exception in the former cases is :

Query Failed [Failed to execute main query]]; nested: TooManyClauses[maxClauseCount is set to 1024]; }]

So now I am trying to set the value of max_clause_count in the index
as follows:

index.query.bool.max_clause_count : 10240

but the index is failing to start if the setting is there in the yml
file and starts if I remove the setting.

So what can be the solution
??

11

Isn't there a way to continue to search even if the clause exceed, like instead of throwing an error, cant we just take the 1024 expanded clauses and continue with our search?
I am also facing the same issue, so wanted to know if we can somehow live with this is error?

12

This is a very, very old thread. Please start a new one for you question.

13

Did you repost this question (about whether you can just continue the search with 1024 phrases instead of getting an error)? I'm curious as well..