I am almost done with the migration of out old ES 2.4 to the new ES6.6
So the time has come to perform some real-world task.
I have a java job which fetches data from ES and stores it in a file.
So the same job is now migrated to use the high level REST client. Other than that, there no difference in the code (the queries are 100% the same).
The job works...
it is more than 10 times slower compared to the same job against the old cluster.
There are some differences between the clusters as follows:
OLD : 34 nodes(4 cores per node), NEW 10 (8 cores per node) nodes
OLD : uses less powerful hardware compared to the new one.
Both clusters use SSD drives
The new cluster runs inside docker containers, one container per node. Nothing else runs on the servers.
The indices in the new cluster have less shards (11 in the old, 6 in the new). So i have larger shards. There is no shard larger then 40GB atm.
The data in both clusters is 100% the same (25 Bio docs)
My test was against a single index with (~800 Mio docs, 224 GB with the replication of 1).
During the test, there is not I/O wait more than 0.3-0.4 so i think the problem is not IO related.
The CPU however is heavily utilized( above 80%)
The elasticsearch.yml is the default.
Am i missing some important configuration that needs to be done in the elasticsearch.yml ?
What could be the reason for such a huge difference in the speed ?
Can the REST client introduce such a huge difference?