Incices are created with replicashards on single node cluster

Elastic Stack Elasticsearch
1

Hi!

I have found myself with a problem that I don't know how to fix, and I don't know exactly why this happens.

What I have right now
Elasticsearch 6.4.1
Single node cluster
Indices: ~1000
Documents: ~500 000 000.

What happend
I was getting ready to perform an upgrade and all I did was to restart the server. At this point we had about 1.2 billion documents. After the server restarted, the cluster entered some kind of recovery-mode and performed what I believe to be (after some searching on Google) a checksum-check on every single document.. This progressed very slowly and after letting it sit for 7 full days, it had only reached about 700 000 000 documents. What was worse is that it seemed to slow down.

So I did another restart, same issue.

This is were the problem started. In an attempt to fix this issue, I changed some clustersettings to speed up the cluster upstart (like num of concurrent recoveries, bandwidth and such). However, one setting I changed caused and issue that I don't know how to fix (and I don't know what I did). To fix the slow cluster start I had to simply delete most of my document. No big deal.

What happens now is that every time a new index is created, it is created with replica-shards? I've never had them before.. And I only have 1 node at this time. So I don't know why is does this?

This means that every day when I get in to work, the cluster-status i Yellow and I need to perform the following command to fix the issue. At least, until the next time a new index is created..

Fix:

curl -XPUT localhost:9200/_settings -H 'Content-Type: application/json' -d'{ "index": { "number_of_replicas": 0 } }'

After I do that, the cluster is back to green. How do I change my cluster back to where it does not try to create replications-shards?

This is some relevant data from today, before performing the "fix":

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4875,
"active_shards" : 4875,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 45,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 99.08536585365853
}

curl -s -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

org.web.metrics-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.web.metrics-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.web.metrics-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.web.metrics-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.web.metrics-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.lb.performance-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.lb.performance-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.lb.performance-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.lb.performance-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.lb.performance-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.web.errors-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.web.errors-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.web.errors-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.web.errors-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.web.errors-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.security.login-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.security.login-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.security.login-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.security.login-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.security.login-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.lb.syslog-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.lb.syslog-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.lb.syslog-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.lb.syslog-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.lb.syslog-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.servers.vpn-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.servers.vpn-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.servers.vpn-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.servers.vpn-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.servers.vpn-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.web.iis-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.web.iis-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.web.iis-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.web.iis-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.web.iis-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.obp.iis-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.obp.iis-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.obp.iis-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.obp.iis-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.obp.iis-2019.12.18 0 r UNASSIGNED INDEX_CREATED
org.servers.systeminfo-2019.12.18 1 r UNASSIGNED INDEX_CREATED
org.servers.systeminfo-2019.12.18 3 r UNASSIGNED INDEX_CREATED
org.servers.systeminfo-2019.12.18 4 r UNASSIGNED INDEX_CREATED
org.servers.systeminfo-2019.12.18 2 r UNASSIGNED INDEX_CREATED
org.servers.systeminfo-2019.12.18 0 r UNASSIGNED INDEX_CREATED

curl -s -XGET 'localhost:9200/_cat/recovery?v&pretty' | wc -l

4876

curl -s -XGET localhost:9200/_cluster/allocation/explain?pretty

{
"index" : "org.web.metrics-2019.12.18",
"shard" : 1,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "INDEX_CREATED",
"at" : "2019-12-18T03:00:03.196Z",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "nvesxBuQT3OpFi7s4UA7sg",
"node_name" : "seswelog01",
"transport_address" : "172.16.33.195:9300",
"node_attributes" : {
"ml.machine_memory" : "135109414912",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[org.web.metrics-2019.12.18][1], node[nvesxBuQT3OpFi7s4UA7sg], [P], s[STARTED], a[id=Wj9Zm00tTHujWUGqmfwyLg]]"
}
]
}
]
}

curl -s -XGET localhost:9200/_cluster/settings?pretty

{
"persistent" : {
"cluster" : {
"routing" : {
"allocation" : {
"node_concurrent_recoveries" : "5",
"enable" : "all",
"node_initial_primaries_recoveries" : "8"
}
}
},
"indices" : {
"recovery" : {
"max_bytes_per_sec" : "200mb"
}
},
"xpack" : {
"monitoring" : {
"collection" : {
"enabled" : "true"
}
}
}
},
"transient" : { }
}

2

I suspect this is far too many indices for a 1-node cluster. According to an official blog post,

A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. This will generally help the cluster stay in good health.

With your 1000+ indices you have at least a 1000 primary shards, while at most - if you max out with a 30 GB Java Heap - you should have 600. This large number of shards will create a massive cluster state and slow down every cluster operation.

Ideally, shards should be 20-50 GB in size, so you may want to try to reduce the number of indices to get fewer and larger shards. That will be reduce the cluster state and be more efficient for your cluster.

You need to create an Index template in your cluster where you set the "number_of_replicas": 0, otherwise a new index will be created with the default 1 replica shard. Just make sure that the index_patterns field matches all the index names you create.

3

Thank you for taking time to answer me!

Yes, there are a few too many indices according to the norm, however the server is a beast and we have had no performance-issues, other than that day when I restarted with 1.2B documents. Having said that, I have already bought another server and will have 2 nodes in the cluster after I have performed the upgrade I originally set out to do. I do have the maximum heap-size.

You need to create an Index template in your cluster where you set the "number_of_replicas": 0 , otherwise a new index will be created with the default 1 replica shard. Just make sure that the index_patterns field matches all the index names you create.

Really! I have seen this info in other forums, but I didn't think that would apply to me since, to my knowledge, I haven't done any changes to any templates. But I surely might have, during my panicmode..

I'll look into it and come back with my result.

Thanks!

4

To check all the installed index templates for any number_of_replicas setting you can run a command like this:

curl -XGET "localhost:9200/_template/*?pretty"

If a template is missing number_of_replicas it defaults to 1.

5

This is not the default behaviour, but it is possible if you set index.shard.check_on_startup. If so, the solution is not to use that setting.

Also to echo what @Bernt_Rostad says, you have far too many shards. 1.2 billion documents doesn't sound like an especially large dataset -- if split evenly over 4800 shards that works out at a tiny 250k docs per shard.

6

curl -XGET "localhost:9200/_template/*?pretty"

Well.. There seems to be zero templates. Every time that I have added a new log to ES, I have just created an index pattern in Kibana and nothing else. Do I need to create a template as well? That would not explain why it has worked fine before though?

Let's say that I have missed that a template always needs to be created as well as an index pattern in Kibana, would this be OK? Our setup is all standard AFAIK:

curl -X PUT "localhost:9200/_template/org.web.metrics?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns" : ["org.web.metrics-*"],
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
}
}
'

And repeat for all logs that are created for with replicas. Correct?

Thank you.

7

This is not the default behaviour

So I've read as well, but I don't think I ever changed it.

Also to echo what @Bernt_Rostad says, you have far too many shards. 1.2 billion documents doesn't sound like an especially large dataset -- if split evenly over 4800 shards that works out at a tiny 250k docs per shard.

I do apologize, I'm realizing now that I do not understand how shards work nor where to set them up. I will try to read about it again and see if I can change the number of shards to a lower number. According to the monitoring in Kibana, we use about 310 GB of diskspace, which would be about 65 MB per shard, which is much lower that 20-25 GB.

8

Index patterns in Kibana has nothing to do with index creation and since there are no index templates in your cluster I would expect all your indices to be created with the default number of primary and replica shards.

In Elasticsearch version 6.4, which you use, the default index setting is 5 primary and 1 replica shard. Both are wrong in your case, you probably only need one primary shard per index and no replicas, so you need to override these defaults. By the way, in 7.x the default number of primary shards is 1.

If you want to control the default index settings, such as the number of primary and replica shards, yes.

I agree. That baffles me too. Perhaps there was an index template there before which got deleted? Other than that I have no idea.

You only need to create the index template once, it will be reused every time a new index is created with a name matching its index_patterns.

I'm not sure if I understand.

An index template is only used when a new index is created, it will not affect any existing indices.

In order to change the existing number of primary shards in your indices you will either have to reindex each old index to a new one (or reindex many old into one new) with the desired number of primary shards. Or you could try the shrink API, which I'm not familiar with.

Good luck!

9

Oh, I figured that I would have to make a template for each single logtype, since it requires an index pattern..

So, what you are suggesting, is something along this:

curl -X PUT "localhost:9200/_template/mydefaults?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns" : ["*"],
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
}
}
'

Which would apply for all newly created indices since the pattern is "*". The old ones being wrong isn't a huge deal, worst case I'll just let them expire or delete them.

10

Yes, that should match all new indices. But you may want to add an order field, e.g.

curl -X PUT "localhost:9200/_template/mydefaults" -H 'Content-Type: application/json' -d'
{
    "index_patterns" : ["*"],
    "order": 0,
    "settings" : {
         "number_of_shards" : 1,
         "number_of_replicas" : 0
    }
}

The reason I mention order is that you can apply more than one template for index creation, a bit like inheritance where the lowest order templates define the defaults and the higher order the more specialized values. in such cases the order is crucial. In my clusters I commonly have up to four orders of templates, with an order 0 template with the defaults and then more specialized templates to override some of these defaults for certain indices.

Let's say you later realize that you need a special template for some new indices, perhaps they need 2 primary shards. Then you can add a second index template with that setting but a more precise index_patterns that only matches those special indices. But in order for Elasticsearch to apply this second template after the first, it needs to have a higher order value. If not, the default template could end up being applied after the more specialized one, and then you won't get the 2 primary shards setting after all.

11

Great explanations, thank you! I have now created the template and reset the cluster back to Green. Tonight at 12 o' clock all new indices will be created and hopefully with correct settings. I will let you know either way!

Looking at my templates now, I see the one we created:

curl -s -XGET "localhost:9200/_template/*?pretty" | jq -r .

image
image447×956 94.6 KB

Will get back to you tomorrow, thank you!

12

Hi!

This morning all indices were created correctly and the cluster is back to Green!

Thank you!

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.