Is there a way to adjust the parameter http.max_header_size (doc) from Elastic cloud?
Context:
I am using Elasticsearch through App Search.
When I perform a search request with header exceeding 8kb, I receive an error 431"Request header fields too large".
My issue is very similar to this ticket from 2014. The proposed solution is to adjust elasticsearch parameter http.max_header_size.
However I am using Elastic Cloud to configure the deployment, and it does not allow changing this parameter. I have tried to change it through adjusting user settings, but it only supports limited list of parameters, which does not include http.max_header_size.
Also, the documentation states that http.max_header_size defaults to 16kb, but what I observe is that it starts failing at 8kb.
This is a blocker for us, because we use the Header Authorization to embed visibility rules, and the search does not work at all for some users because the header is too large.
The limit you're hitting is not in Elasticsearch, but in App Search. Elasticsearch has a max_header_size of 16kb, but App Search has 8kb limit.
There is no possibility of configuration on the App Search side for raising this limit as of now.
Is there something that could be done to reduce the headers size from the client? Would an intermediate proxy work as a solution to overcome this problem?
Thank you for your prompt and precise answer! I'm glad we have an explanation for this issue.
There is no possibility of configuration on the App Search side for raising this limit as of now.
Is there something that could be done to reduce the headers size from the client? Would an intermediate proxy work as a solution to overcome this problem?
We use the signed search keys strategy described in the following article:
This allows us to tailor visible content to the user that performs the search (by embedding some filters in the API key).
We'd have to reconsider the visibility rules and I don't think it is feasble in the short term unfortunately.
If filters are what is causing the API key to be more than 8KB in size, you might consider different approaches:
Include simplified filter fields in your docs (instead of using a combination of complex filters, use additional fields in the docs that correspond to these conditions)
Use separate engines for the docs according to their permissions, and use engines as part of different public search keys.
Use separate engines for the docs according to their permissions, and use engines as part of different public search keys.
We do use different engines actually, so we can't really use this further
Include simplified filter fields in your docs (instead of using a combination of complex filters, use additional fields in the docs that correspond to these conditions)
I'll see if we can decrease the amount of filter values to be able to reduce the header size then
My team and I are working on this embedded filter issue. We were wondering, is there a plan on Elastic's side to make the App Search 8KB max header size configurable at some point? (Like the max header size in Elastic is 16KB by default but can be modified).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.