Elasticsearch Query Performance while reading and writting in parallel

Elastic Stack Elasticsearch
1

Hi All,

We have setup 10 nodes ES cluster each has (16 cores and 32 GB RAM and 2
disk). We are using separate disk for each node.

We have around 24 indices (10 shards each), size of index is 10 GB.

Case1:
We are able to fetch 10000 records in 9 secs if indexing is not running in
parallel with query.

Case2:
When both indexing and query are running in parallel, then same query takes
70 secs.

Thanks,
Ankit Jain
iLabs

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hello Ankit,

In case2, I assume you have slower query performance coming from 2
directions:

  • indexing creates additional load
  • indexing triggers merges, which invalidate caches

If there's a lot of load generated by indexing, I don't see a good option
to fix the issue, other than adding more/bigger nodes. Maybe increase the
refresh_interval. Also, 240 total shards sounds a little too much for 10
nodes.

If your indexing is relatively light, you might counter the cache
invalidation effect by using
warmershttp://www.elasticsearch.org/guide/reference/api/admin-indices-warmers/
.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Fri, Apr 26, 2013 at 11:06 AM, Ankit Jain ankitjaincs06@gmail.comwrote:

Hi All,

We have setup 10 nodes ES cluster each has (16 cores and 32 GB RAM and 2
disk). We are using separate disk for each node.

We have around 24 indices (10 shards each), size of index is 10 GB.

Case1:
We are able to fetch 10000 records in 9 secs if indexing is not running in
parallel with query.

Case2:
When both indexing and query are running in parallel, then same query
takes 70 secs.

Thanks,
Ankit Jain
iLabs

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hello Ankit,

For changing the refresh_interval, I suggest you use the Indices Update
Settings API:

Best regards,
Radu

On Fri, Apr 26, 2013 at 8:50 PM, Ankit Jain ankitjaincs06@gmail.com wrote:

Hello Radu,

Can you share the configuration details that i need to tweak to get the
near real time performance.

Thanks & Regards,
Ankit Jain
iLabs

On Friday, 26 April 2013 18:31:15 UTC+5:30, Radu Gheorghe wrote:

Hello Ankit,

In case2, I assume you have slower query performance coming from 2
directions:

  • indexing creates additional load
  • indexing triggers merges, which invalidate caches

If there's a lot of load generated by indexing, I don't see a good option
to fix the issue, other than adding more/bigger nodes. Maybe increase the
refresh_interval. Also, 240 total shards sounds a little too much for 10
nodes.

If your indexing is relatively light, you might counter the cache
invalidation effect by using warmershttp://www.elasticsearch.org/guide/reference/api/admin-indices-warmers/
.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Fri, Apr 26, 2013 at 11:06 AM, Ankit Jain ankitj...@gmail.com wrote:

Hi All,

We have setup 10 nodes ES cluster each has (16 cores and 32 GB RAM and 2
disk). We are using separate disk for each node.

We have around 24 indices (10 shards each), size of index is 10 GB.

Case1:
We are able to fetch 10000 records in 9 secs if indexing is not running
in parallel with query.

Case2:
When both indexing and query are running in parallel, then same query
takes 70 secs.

Thanks,
Ankit Jain
iLabs

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.