Analyzing profiling results

[alexs1]

Hello,
I'm migrating a cluster from an old one, and for some reason, after reindexing by best practices (than I know), performance are slower in the new cluster.

the first has 500GB- 5 shards and 1 replica
the new cluster is using ILM with rollover each 90 days/44gb. Each rolled over index has 1 shard with a replica and replica.

I was expecting the new index will be faster since 5 shards where skipped, but it's the opposite

The new nodes got 58gb, 1.8 tb, and 50% is free

attached a profile results for the following example:

query:

---

GET index/_search
{
  "profile": "true",
  "size": 0,
  "query": {
    "range": {
      "ingest_date": {
        "gte": "now-48h"
      }
    }
  },
  "aggs": {
    "NAME": {
      "date_histogram": {
        "field": "ingest_timestamp",
        "interval": "hour",
        "min_doc_count": 1
      }
    }
  }
}

---

the ingest_date has the following format : "yyyy-MM-dd", and the ingest_timestamp: "epoch_second"

old index profile result

new index profile result:

what am I missing?
thank you!

[Christian_Dahlqvist]

What is the specification of the old cluster compared to the new? Are they running on the same type of hardware? Which versions of Elasticsearch are you using? Can you show the output of the cat indices API for the queried indices?

[alexs1]

After more investigation, splitting the shards to 20gb instead of 50gb fixed the issue.
I still wonder why the default setup in ilm is 50 gb

Thanks for your time!

[alexs1]

@Christian_Dahlqvist unfortunately I was too hurry to think it's resolved.

I moved from aws managed service (6 nodes r5.2xlarge.elasticsearch no dedicated master with 7.4) to Elastic cloud on aws with 3 hot nodes (58 gb), 3 warm nodes, coordinating nodes, dedicated masters) version 7.12.
Entire data is 5 tb.

Everything seems to work slower.. any query on any index.

I've tried restoring an index from aws, without changing it, and ran the same query on both, aws always wins by far. (It can be 1 sec vs 8 sec for example)

Any recommendations will be appreciated .

[dadoonet (David Pilato)]

@alexs1

I moved from aws managed service (6 nodes r5.2xlarge.elasticsearch no dedicated master with 7.4) to Elastic cloud on aws with 3 hot nodes (58 gb), 3 warm nodes, coordinating nodes, dedicated masters) version 7.12.
Entire data is 5 tb.

Everything seems to work slower.. any query on any index.

I'd recommend contacting the Elastic Cloud support team. Please see Specialized Support from Designated Elastic Engineers | Elastic on how to raise a support ticket for Elastic Cloud

Note also that you are comparing 7.4 with 7.12, with a different architecture (specifically warm nodes). So a lot of things can happen.
To compare things, you need to check first that you are only hitting indices which are located on the hot nodes. Calling the warm nodes might be slower IMHO.