Es rejected execution exception on AWS

Elastic Stack Elasticsearch
1

Hi,

I have a one node ES cluster on AWS (t2.small.elasticsearch) and I am receiving an es_rejected_execution_exception on the Grafana side. It occurs when I do couple of refresh on the dashboard (or sometimes when two people opens the same dashboard). (having around 20 widgets, so 20 concurrent queries, usually really simple queries like the last item in the last hour or count of entries grouped by hour).

Here is my setup:
I have 3 main indexes with daily rotate.
It is running now for couple of weeks and because of the rotation I have around 39 indices in total.
According to the _settings API, I have these amounts for each of the indices:

  • number_of_shards: 5
  • number_of_replicas: 0

This sums up around ~191 shards. Each index contains around 1-6MB of data and the one node instance has around 66K searchable documents.

What could be the issue here or how to optimize this?
I believe the shard and index count do matter. Should I push in an index template with one shard and rotate only weekly? I am trying to guess here after reading around some topics.

Any idea or help would be really appreciated.

Many thanks,
TZ

2

t2 instances are not very suitable at all for Elasticsearch as they have limited CPU allocation and easily get throttled. I would recommend upgrading to an m4/m5 type instance instead.

Yes, that is probably a good idea. See this blog post for further details.

I would also recommend considering our Elastic Cloud Elasticsearch Service as this supports hot/warm architectures.

3

Thanks, I will consider this.

I have checked the whole data size and for these couple of weeks we only have around 100MB. Do you think it is better to rotate monthly?

In that case I can use the reindex api as I have a pattern like index-2018-12-18 so easy to reindex to index-2018-12. But once I do this it will be a little harder to go back to weekly right?

If I do this I will have just really a couple indices with couple of shards and couple of 100MB. According to the documentations and posts I have read it is better then having lots of smaller shards.

4

Where have you read that? If you look at some of the resources I linked to, the advice is basically the opposite.

5

Sorry that was a "than" and not "then". So having lots of smaller shards is not too good, this is what I read.

If this is the case do you think monthly is better in my case? Considering that I would like to keep this t2 instance for now just to prevent this only one dashboard having trouble.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.