I am trying to run an elasticsearch cluster where each of our customers (~5000 and not likely to change too much) has their own index. Unfortunately, update times are very slow.
Creating or deleting an index takes ~ 5s, and data is only uploaded at 100 docs/s (using the bulk upload). Each index is fairly small ~10K docs each ~1KB so 10MB total. As the data is so small, on each data change we create a new index with the new data and point a consistent alias to the new index, before deleting the old index.
These times are not an issue in themselves - we do not need new data to be immediately available - but it appears that indexes cannot be created/deleted in parallel? Thus we are limited to 6 update jobs per minute. Are their any config changes we could make to increase this? I have tried reducing the consistency to one, and the refresh interval to 2 mins. Below are some features of our cluster.
"number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 4968, "active_shards" : 9936,
Thanks for any suggestions and let me know if there is any other info I could provide