Url too long, too many indices

Elastic Stack Elasticsearch
1

Hi,

We have a situation where we'll end up maintaining a large-ish set of indices in ES, and where we want to query on sets of these indices (chosen by users). Right now, we're seeing an issue where the search URL reaches 6.000 characters, and we're then getting an error on that. Removing unecessary parts of our index names has brought us down to 4.500 characters, but it's still above the 4.096 character limit -- and as we'll get more indices in the future, this is a temporary solution at best.

I've found that we can ditch the indices in the Query string altogether, and go with a Terms Query on the virtual "_index" field. This'll work - but ... is it good?

Is there any adverse effect to doing this? Any optimizations we might miss when querying a single index f.ex. .. ?

Should I put the indices in the URL when I can (up to say.. 3.000 characters), and then in the body / Terms Query above that? ..

Error presented:
An HTTP line is larger than 4096 bytes

Mike.

2

Are you able to use wildcarding or index aliases? They might not apply to your case and you might be fine with the Terms Query _index field, but these have simplified index selection/specification in my case.

https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-index.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

3

In addition to @dss suggestion for using aliases, is it possible to condense index names through wildcard matches?

4

In our case, the index names come from uploaded binaries which we use to process data. For simplicity, we name them a similar thing to the binaries' names -- we could in principle name them our internal tracking id's, they're shorter, but it would complicate troubleshooting and more.

Our internal id's are 16 characters in length, and for my example above I had 156 indexes totaling ~6.500 characters in lenght (average: 41 characters), so using that we could go up to ~240 indexes before we hit the limit again.

I'm not sure aliases would help us (users may select any combination of indices.. We could create aliases on the fly -- but it's my impression that that would be even worse, it's a cluster-operation, right?).

Expansion might also not help. If we had two indices named AA and AB, and the user searches in "AA", we don't want to risk hitting AB by searching in "A*".

Mike.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.