We have a rails application that is using the official elasticsearch gem to connect to elasticsearch.
I did a deploy which simply added one field (number) to the index (by editing the model + adding it to as_indexed_json) and would query by this field specifically in search.
A few minutes after the deploy load spiked very hard on one of the 3 servers in our Elasticsearch cluster. And then errors starting spamming in our log:
Dynamic mappings are not available on the node that holds the primary yet.
Searches started to get very, very slow, and eventually the cluster just failed entirely.
We had to force restart the cluster, and remove the code that added the field/search, and things started to work perfectly fine.
We did NOT do a PUT to update the mappings for the index, I was waiting for the first model index to basically do this.
Any ideas/pointers would be greatly appreciated.