High index size causing performance issue

Elastic Stack Elasticsearch
1

Hello,

I have a java application that is writting data directly to an Elasticsearch index named as candidate having 1 primary and 1replica and this is now causing issue.Need to increase no of primary shard. 300gb candidate index, 5 node.

I am aware create new index candidate-new and then reindex all data with increased shard.

But challenge is downtime is not possible to stop write on current index candidate and also customer need all his data in same named index only again as index name is hardcoded in java code and they don’t want to change.

Post this , needs to maintain along with ilm and rollover to work effectively.

I have a roadmap with downtime, without downtime how can I achieve it?, any suggestion would be helpful.

2

Is the data being written immutable or are you also performing updates and/or deletes?

Is the writing being performed to a hardcoded index name?

Are you running searches against this hardcoded index name?

Is the issue around downtime applicable to reads, writes or both?

Which version of Elasticsearch are you using?

3

Thanks for your time to look into this.

1)Data is immutable
2)write is being performed on hardcoded index
3 )Search is also done on hardcoded index
4)Downtime is applicable for both read and write.(can’t take downtime as java application directly write to candidate index)
5) Version 7.17.7

Same scenario is there for one more index but there data is updated in index.

4

There is no way to achieve what you want to do without downtime. If you want to switch to rollover I suspect you also may need to change your application to read from an alias with a different name.

5

Is the below approach correct if I can manage to block write operations for few minute?

Step 1:new index

PUT /candidate-new
{
"settings": {
"number_of_shards": 8,
"number_of_replicas": 1
},
"mappings": {
... # same mappings as candidate
}
}

Step 2: block write temporary to original index

PUT /candidate/_settings
{
"index.blocks.write": true
}

Step 3: assign alias name to new index, as same index in java code

POST /_aliases
{
"actions": [
{
"add": {
"index": "candidate-new",
"alias": "candidate"
}
}
]
}

Step4: again make it writable

PUT /candidate/_settings
{
"index.blocks.write": false
}

Step5:

POST _reindex
{
"source": {
"index": "candidate"
},
"dest": {
"index": "candidate-new"
}
}

Step6:
Delete candidate

If i can take downtime then how can it be done in better way so that in future same issue don't come, how can I rollover and apply ilm.

6

You can not have an alias with the same name as an index so your approach will not work. I would try the following:

  1. Block write on the candidate index
  2. Use the split index API to create a candidate-new index with the desired number of primary shards. This is the fastest and most efficient way to get an index with more primary shards and all the data of the original index
  3. Once complete, delete the original candidate index
  4. Create a candidate alias linked to the candidate-new index. This will allow the application to read and write through the alias as it points to a single index
  5. Enable traffic

As far as I can tell this is the best way to minimize downtime while not having to make any changes to your application. It will resolve the issue, although I would recommend you make changes to the application (make name of read and write index configurable separately so they can point to different aliases) so you can switch to ILM later on.

Changing to rollover and ILM will require a different procedure but we can talk about that once the changes to your application has been made.