We are upgrading our Elasticsearch 5.6 clusters to Elasticsearch 7.6.
We're detecting that the search performance is worst in Elasticsearch 7.6 than 5.6.
We are testing empty clusters with the same installation process (ubuntu 18.04 + deb package with systemd) for the two versions of Elasticsearch.
The cluster has 6 data nodes and 1 dedicated master node. (1 cpu, 2 cores and 8 GB ram).
We are populating the clusters with 3000 indices with one shard and at least one replica (about 6500 shards). 22 M of documents and 82 GB of data.
The only modified settings is the mapping that we changed some deprecated tokens ngram instead of nGram, edge_ngram instead of edge_nGram and word_delimiter_graph instead of word_delimiter (due a change in Lucene that break word_delimiter after synonyms filter).
The same data, the same indices, the same search test with the same frequency and we have a 8-10ms of average performance loss in Elasticsearch 7.
After review more the processes we detect that Elasticsearch 7 takes a 1 or 2 ms more time in search phase and the rest in the fetch phase and the cpu usage is more high in Elasticsearch 7.
We are playing with different jvm sizes (2000mb, 2500mb, 3000mb and 3800mb), with different java versions and with different changes over the cluster, with and without ARS, some buffers changes, etc...
We don't know why the same hardware and the same data/settings with the same queries over the clusters have worst performance in Elasticsearch 7.
Could someone help us?