Index Management - Operational Procedures

[fvegaucr (Fernando)]

Hi I was wondering if someone can point me to documentation or information about how to deal with indexes, like regular operations that admins use or try to fix issues, sort of like bestpractices.

I want to understand or how to deal with different issues for example when:

• You have an index with and ILM policy and some field change in the new incoming data and elastic can't index it.

• When you delete an index with and ILM policy ( for example if I have index-00001, index-0002, index-00004) Lets say index-00004 detected that one field is not there any more and generates an exception, how to make that process return to normal.

• how to reindex data.

• How to move old indexes that might have a few fields different that the new dataset to follow the new indexes ILM policies.

• How to add or change a field type old indexes.

• How to deal with failing shards, or unresponsive shards.

• How to fix issues during phase transitions like when you apply ILM policy and indexes are being move from hot to warm

• How to fix issues during phase transitions like when you apply ILM policy and indexes are being move from hot to warm, but then the new data has some field or key changed, and elastic cant index the data due to discrepancy with the mappings.

Thank you very much

Regards,

[warkolm (Mark Walkom)]

I'm not aware of anything that covers all of this to the detail you are after.

@fvegaucr

You have an index with and ILM policy and some field change in the new incoming data and elastic can't index it.

What do you mean by can't index it? Is there an error? Are your mappings set as strict?

@fvegaucr

When you delete an index with and ILM policy ( for example if I have index-00001, index-0002, index-00004) Lets say index-00004 detected that one field is not there any more and generates an exception, how to make that process return to normal.

What do you mean by return to normal? If you're talking about in Kibana then you may need to refresh the index pattern.

@fvegaucr

• how to reindex data.

Into an ILM policy. or something else?

@fvegaucr

How to move old indexes that might have a few fields different that the new dataset to follow the new indexes ILM policies.

You'd need to reindex that. I think it'd make sense to create a new index and then add that to the ILM alias so that it's accessible. This would be a manual process at the moment.

@fvegaucr

• How to add or change a field type old indexes.

This is a reindex process.

@fvegaucr

• How to deal with failing shards, or unresponsive shards.

Depends entirely why they are failing or unresponsive.

@fvegaucr

How to fix issues during phase transitions like when you apply ILM policy and indexes are being move from hot to warm

What issues are you talking about?

@fvegaucr

How to fix issues during phase transitions like when you apply ILM policy and indexes are being move from hot to warm, but then the new data has some field or key changed, and elastic cant index the data due to discrepancy with the mappings.

I think this relates to the first question you asked, so clarifying that would help answer this too

[fvegaucr (Fernando)]

Hi Thank you for your reply, and apologize I wasn't very clear.
So basically Im looking for information that can help with the issues described above, general approach, best practices or how to troubleshoot them, I know for shards depends of different factors, so I will like to know what are the most common ones and how they got fixed.

On regards the data not being index, this is the error that I see
`

[2020-04-16T10:28:16,137][WARN ][logstash.outputs.elasticsearch][turnservers] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"turnservers_atl1-2020.04.09", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x4020732e], :response=>{"index"=>{"_index"=>"turnservers_atl1-2020.04.09", "_type"=>"_doc", "_id"=>"IhEAhHEBUxMK2H4qmwQq", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [overlord_environment] of type [boolean] in document with id 'IhEAhHEBUxMK2H4qmwQq'. Preview of field's value: 'production'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Failed to parse value [production] as only [true] or [false] are allowed."}}}}}
`

So I had fix issues similar to that one by either deleting the index and re-indexing the data. But with ILM I remove the index in question, and then I had a different error that I forgot to save, but was able to fix once I remove the rest of the ilm indexes sequence.

Thanks again