Hi, I've got an application that uses the same basic index strategy as logstash (one index per day), but I have data that covers several years, which means I have thousands of indexes. Now, I want to run an aggregation across these years of data, but that quickly causes overflow in the default 1000 element search queue.
I realize that an ideal solution would be to change the index strategy to have fewer indexes rather than a very large number of small ones, but for various reasons, that's not easy to do with this application.
The most expedient solution appears to be increasing the search queue size, but I'm having a hard time finding any concrete description of what consequences this might have? I understand that some search operations may take longer if they have to wait for a while in a longer queue. Are there other things I should be concerned about? Is the memory consumed by requests waiting in the search queue anything to worry about (particularly given that there are gigabytes of segment data already filling up the heap)?
Another possible solution is to perform a series of smaller aggregations from my application and then combine the results myself, but that feels wasteful as it duplicates a task that ES is already quite good at.
So, any guidance or words of wisdom?
Thanks in advance...