I'm wondering is it normal to create/delete indexes, let's say, once per 10 minutes. Is it memory or cpu consuming? Does elastic allocates storage and if so, how much is time waiting for this storage being available once again for elastic?
There's some overhead, but it's reasonably minimal. When creating an index, the master makes note of it in the Cluster State and publishes the change to all nodes. Individual nodes will allocate a directory for the shards of the index and write a few small files containing metadata. So there's a bit of network overhead, and a bit of IO.
But ES doesn't pre-allocate any large portions of disk or anything.
When deleting, the process is basically in reverse. The master removes it from the cluster state, the shards remove the directories from disk.
So all in all, it's a reasonably light operation.
I'm not sure if it's common practice to do this, but it probably won't be too harmful to performance of your cluster. Why do you need to create/destroy indices so quickly?
There are unit tests in our application, e.g. test for update_by_query which updates document and then searches for it and looks whether it was updated. So if at least two developers will perform this test using one index it will lead to unexpected results e.g. conflicts. So I decided to create index with random name for every test to avoid such kind of problems. But some days after these changes we've got elastic broken because of java heap out of memory. So I thought probably creating and deleting indexes leads to huge overhead
Ah, I see. I'm not sure what was causing the OOM, but it likely wasn't the index creation/destruction. That's exactly how we run a bunch of our integration tests too (the "yaml test runner" that the various client libraries use builds/destroys indices a few hundred times during the test run).
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.