ElasticSearch 2.3.0 Performance

I am using elastic search 2.3.0 hosted on AWS Cloud.

I have recently made code changes for using the bulk indexing option provided by elastic.

When i have tested this change on my PERF env , i was happy with the performance achieved.
But , when i pushed this code to PROD env , the performance was very poor compared to the performance observed in PERF environment.

  • Both PERF and PROD environments have the same hardware configuration (m4.2xlarge EC2 Server)
  • Data Volume in PERF is about 170 Million Docs Stored in a single index , whereas data volume in PROD is about 800 Million Docs Stored in a single index.

Rest all the configurations in elasticsearch.yml are same in both PERF and PROD environments

I am unable to figure out the poor performance. Please provide me with some pointers.


Are you running the AWS ES service?

The main difference seems to be the amount of data in each cluster. As you have a single index, are you assigning your own document IDs and/or performing document updates? Does the shard size differ between the environments?

Shard size is same in both the environments. We are assigning our own document id in both the environments ..

If the shard size is the same, does that mean that you have 4-5 times more shards in the production environment as the number of documents is considerable larger? How many shards do you have? What is the average shard size?

I meant to say number of shards are same.

Both in PERF and PROD , we have 50 Shards for the index.

In Production each shard size is now currently at 6 GB.

I found this out using the command GET /_cat/shards?v

When you say that performance is poor, are you referring to indexing and/or query performance? What performance are you seeing?

Both Indexing and Querying as well.

Actually Querying is good on PROD as well , when there is no Indexing happening.

When there is indexing running in parallel , querying is slow in response.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.