Changing the shard size

Elastic Stack Elasticsearch
1

Hi,
I have Aws elastic search cluster with two nodes. The data is being pushed from fluentd to elastic search. By default all the indexs have 5 primary shards and 1 replica. And i want to resize the no of primary shards to 2 for the new indexes. "I know i can use shrunk api for older indices"

Where can i do this. Indices are created by fluentd itself.

2

I don't think you can reduce a 5 shards index to 2 shards. You would need to reindex instead in a new index.
But you can reduce to 1 shard I think. Here is a typical script to do that on an index INDEX in a cluster which has a node with name NODE1:

# Read only and move primaries to one single node and no replica
PUT /INDEX/_settings
{
  "settings": {
    "index.number_of_replicas": 0,
    "index.routing.allocation.require._name": "NODE1", 
    "index.blocks.write": true
  }
}

# Check shards allocation
GET /_cat/shards/INDEX*?v&h=index,shard,prirep,state,docs,node

# Shrink the index
POST /INDEX/_shrink/INDEX_shrunk
{
  "settings": {
    "index.routing.allocation.require._name": null,
    "index.blocks.write": null,
    "index.number_of_replicas": 1,
    "index.number_of_shards": 1 
  }
}

Check the fluentd documentation. May be you need to change an index template...

BTW did you look at Cloud by Elastic, also available if needed from AWS Marketplace, Azure Marketplace and Google Cloud Marketplace?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next :slight_smile: ...

3

Hey Thank you for the reply

I dont want to change shard count for previous indexes . I only want to chnage the or restrict primary shard count to 2 for the new indexes which are going to come in. Is there a way to do that from elastic side. Or it should be done from fluentd end? Note: Fluentd is creating the index as the data coming into it.

4

You do that through an index template that matches the index names created by fluentd.

5

How can i change the default value of shards in the cluster settings?

6

You need to use index templates.

1 Like
7

I used this index template

PUT _template/template_1
{
"index_patterns": ["te*", "bar*"],
"settings": {
"number_of_shards": 1
},
"mappings": {
"_source": {
"enabled": false
},
"properties": {
"host_name": {
"type": "keyword"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z yyyy"
}
}
}
}

Now my indexes are not showing all the fields. should i also add fields to this index template.

If thats the case i cant some fields come in dynamically sometimes.

8

Why did you disable the source field?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.