Uneven size per shards

Elastic Stack Elasticsearch
1

Hello!

I saw some previous posts about uneven shard distribution but I was unable to get information that I need.
I have an index which has 2 Shards and 1 Replication:

index-name 1 r STARTED 1510504 546.6mb 10.10.1.34 node-name
index-name 1 r STARTED 1510504 458.4mb 10.10.1.229 node-name
index-name 1 p STARTED 1510504 502.9mb 10.10.1.224 node-name
index-name 0 p STARTED 374273705 51.4gb 10.105.1.34 node-name
index-name 0 r STARTED 374273705 52.6gb 10.10.1.139 node-name
index-name 0 r STARTED 374273705 51.4gb 10.10.1.57 node-name

Based from the API explanation, there's no reason to do a rebalance but as we see the docs and size difference is significant. I don't see any important output in the deciders that might affect the result. Below are some result of explanation:

  • "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
  • "can_remain_on_current_node" : "yes",
  • "can_rebalance_cluster" : "yes",
  • "can_rebalance_to_other_node" : "no",

Here are some other notable settings:

  • The current index settings mostly use the default settings for cluster allocation.
  • The same index has a balance shard distribution in dev and staging environment. Only the production has this status.
  • We're doing a deletion of docs in this index.
  • We're using custom _id for each docs.

Kindly let me know if there's anymore information needed.
Thank you!

2

are you using routing? or parent/child or the join type? There must be a reason why one shard only has 500MB and the other has 100x the size.

1 Like
3

Hi Alexander,

We're only using the default routing provided by Elasticsearch, and not a custom one. Below is the cluster settings config that we have.

{
      "persistent" : { },
      "transient" : {
        "cluster" : {
          "routing" : {
            "allocation" : {
              "enable" : "all",
              "exclude" : {
                "_ip" : "10.5.1.109"
              }
            }
          }
        }
      }
    }

The IP which is excluded doesn't exist anymore.

4

sorry for being unclear. This was not the routing I am referring to. Are you using the routing parameter when indexing data?

If not, how do your ids look like?

5

We're not using any routing parameter when indexing data.

Here's some sample of IDs in that index.

"586ab92846e0fb0001f74910",
"59376fa49948f20001aec204",
"59376f81a5ca4f0001774bf1",
"59376fa9a5ca4f00017760ea",
"5896b3c046e0fb00013a7537",
"5895b69e46e0fb0001a0ea88",
"5a0e121aa752c50001d0676f",
"5a0e009e71bf640001b42140"

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.