Optimize elastic for creation and deletion of many small indexes

Elastic Stack Elasticsearch
1

We have integration tests for our queries to elastic (to ensure that proper data is returned). Therefore we create several indexes per application branch (with unique name). For each test we create only a fiew record and run query.

Everything works fine on local machine, but on our build environment over time shared elastic instance behaves slowly. Our 200+ tests run for 1 min locally, and 3 min from build environment over network. But sometimes it slows down to 40 min. The slowest part is manual index refresh (so we ensure that all records are available on search).

We use default elastic docker container with 4 gb of heap for jvm.

What would you suggest to look at to overcome elastic slow downs? We reuse index per test run, but create a new index per branch since mappings are different and may not be compatible

2

Are you just running a single node? there's a lot of network traffic updating cluster state when you create/delete indices

Try with more heap for elastic?

Do you have ssd?

Delete any unnecessary indices

More memory for docker so filesystem cache kicks in better

I don't think there is a magic setting that can improve things really

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.