Decrease the number of shards

Elastic Stack Elasticsearch
1

Hi,

I need help with changing the number of shards. I got basically default settings, ELK stack 5.4.

Already had some indexes, but had 5 shards. Would like to change to two.
I deleted existing indexes (not the system indices) and would like to change the number of shards before i add new indices.

Tried like that (How do you change the default number of shards?) but no success.

and also like that:
curl -XPUT 'localhost:9200/_all/_settings?preserve_existing=true' -d '{

"index.number_of_shards" : "2"
}'

Also no success, i think. -> ,"reason":"Can't update non dynamic settings [[index.number_of_shards]] for open indices. Does this mean that new indices will have the changed setting?

Probaby doing something trivial wrong, but currently i'm kind of stuck. :confused:

Hope for help.

Thanks!
M.

2 Likes
2

Hi!
decreasing the number of primary shards is only possible using the Shrink API
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/indices-shrink-index.html

And even this has some downsides.

You should therefore try to get the correct number of shards at index creation.
For example you can use index tempaltes to define how many shards an index should have.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

If you already have an index and don't plan on creating a new one anytime soon the reindex API can be helpful to migrate indices.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

4 Likes
3

Is it possible to update a index template?
I tried just to add

  "settings": {
"number_of_shards": 1

},

To existing logstash template, but removed everything else. Got a backup of template data, tried to overwrite but got an error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_e_o_f_exception",
      "reason" : "Unexpected end-of-input: expected close marker for Object (start marker at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@617ebc28; line: 2, column: 1])\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@617ebc28; line: 76, column: 1724]"
    }
  },
  "status" : 400
}

Not sure what i'm doing wrong...

4

Hi again.
there is no way to update templates.
I would suggest to run a GET and the update it manually and then PUT it again.

Let me know if that works for you. If you have another different question don't hesitate to make a new post :slight_smile:

1 Like
5

I'm somewhere here:

curl -XGET 'localhost:9200/_template/logstash?pretty'
{
  "logstash" : {
"order" : 0,
"template" : "logstash-*",
"settings" : {
  "index" : {
    "number_of_shards" : "2"
  }
},
"mappings" : { },
"aliases" : { }
  }
}

But my indices:

curl -XGET 'localhost:9200/smsc-*/_settings?pretty'

  "smsc-2017.05.01" : {
    "settings" : {
      "index" : {
        "creation_date" : "1495180973719",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "67XydLUlQkqlm1KhxBFFrg",
        "version" : {
          "created" : "5040099"
        },
        "provided_name" : "smsc-2017.05.01"
      }
    }
  }

So it didn't have any effect. Or is it cause provided name is different?

Thank you!

6

You can mention the shards limits while creating the index.

PUT my_index
{
   "settings": {
      "number_of_replicas": 0,
      "number_of_shards": 2,
      "refresh_interval": "30s"
  },
  "mappings": {
    "logs": {
      "_source": {
        "enabled": true
      },
      "_all": {
        "enabled": true
      },
      "properties": {
          "name" : {
               "type" : "keyword"
       }
}

Once you created the index we cannot change the shard size directly. We have to use Shrink API

7

Oh, sorry.
Your exception was related to a JSON parse error.
This is the minimum that you need for a template

{
  "name" : {
    "order" : 0,
    "template" : "indexname-*",
    -----other stuff goes here----- (settings, mappings, aliases)
}
8

hmm...tried like that

$ curl -XPUT 'localhost:9200/_template/smsc/?pretty' -H 'Content-Type: application/json' -d'
> {
>   "smsc" : {
>     "order" : 0,
>     "template" : "smsc-*",
>     "settings" : {
>       "index" : {
>         "number_of_shards" : "2"
>       }
>     },
>     "mappings" : { },
>     "aliases" : { }
>   }
> }
> '
{
  "error" : {
    "root_cause" : [
      {
        "type" : "action_request_validation_exception",
        "reason" : "Validation Failed: 1: template is missing;"
      }
    ],
    "type" : "action_request_validation_exception",
    "reason" : "Validation Failed: 1: template is missing;"
  },
  "status" : 400
}
9

Just to avoid confusion: The index template is only used when you create a new index. This will not change the number of shards for existing indices. You will have to use the shrink API, which Luca suggested, for that (or do a reindex, but that's even worse for reducing the number of shards).

And if you have 5 shards right now, you can only shrink down to 1 shard, since shrinking is always a factor. Prime numbers don't give you many options there unfortunately.

1 Like
10

No problem with creating new indexes. Got backup of all data so i can repharse it.

So new index with one shard. Was any if my previous attemps correct?

Thanks!

11

If you have set the template up correctly and the pattern matches, creating a new index will apply the pattern. I'm a bit confused by the question. Have you tried it out?

12

Was too quick with my question. Created a new template, creating new index and number of shards 1.
Works.

Thank you guys for help!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.