Performance degrade while querying in multiple indices

Elastic Stack Elasticsearch
1

Hi All,

We are creating indices on hour basis and we are firing queries on 2 day
data. The number of records expected to hit is less than 100. But, the
queries are taking unpredictable time (1 sec to 1 min).

Is queries will execute in all 48 indices in parallel or sequential?
What will be the main reason for performance hit?

Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

parallel.

Let's say you are using default settings, so you have 5 shards per index.
You are running queries on 240 shards (that means 240 Lucene requests).

Why do you need to split your data every hours?
Do you see anything in logs? GC thing for example...

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 juil. 2013 à 10:13, Ankit Jain ankitjaincs06@gmail.com a écrit :

Hi All,

We are creating indices on hour basis and we are firing queries on 2 day data. The number of records expected to hit is less than 100. But, the queries are taking unpredictable time (1 sec to 1 min).

Is queries will execute in all 48 indices in parallel or sequential?
What will be the main reason for performance hit?

Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi All,

Why do you need to split your data every hours?
We are getting 10 million records each hour and also we would like to
provide time based searching.

Also, I am using below code to delete records.

            DeleteByQueryRequestBuilder dbqb =

client.prepareDeleteByQuery(
existingIndices.toArray(new String[existingIndices
.size()])).setQuery(qb);
log.info("Request done :");
DeleteByQueryResponse deleteByQueryResponse = dbqb.execute()
.actionGet();

dbqb.execute().actionGet(); -> This statement takes 1 min to get the
response from ES.

Regards,
Ankit

On Wednesday, 3 July 2013 13:58:11 UTC+5:30, David Pilato wrote:

parallel.

Let's say you are using default settings, so you have 5 shards per index.
You are running queries on 240 shards (that means 240 Lucene requests).

Why do you need to split your data every hours?
Do you see anything in logs? GC thing for example...

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 3 juil. 2013 à 10:13, Ankit Jain <ankitj...@gmail.com <javascript:>> a
écrit :

Hi All,

We are creating indices on hour basis and we are firing queries on 2 day
data. The number of records expected to hit is less than 100. But, the
queries are taking unpredictable time (1 sec to 1 min).

Is queries will execute in all 48 indices in parallel or sequential?
What will be the main reason for performance hit?

Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

You should remove the index itself. Why removing records?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 juil. 2013 à 11:27, Ankit Jain ankitjaincs06@gmail.com a écrit :

Hi All,

Why do you need to split your data every hours?
We are getting 10 million records each hour and also we would like to provide time based searching.

Also, I am using below code to delete records.

            DeleteByQueryRequestBuilder dbqb = client.prepareDeleteByQuery(
                    existingIndices.toArray(new String[existingIndices
                            .size()])).setQuery(qb);
            log.info("Request done :");
            DeleteByQueryResponse deleteByQueryResponse = dbqb.execute()
                    .actionGet();

dbqb.execute().actionGet(); -> This statement takes 1 min to get the response from ES.

Regards,
Ankit

On Wednesday, 3 July 2013 13:58:11 UTC+5:30, David Pilato wrote:
parallel.

Let's say you are using default settings, so you have 5 shards per index.
You are running queries on 240 shards (that means 240 Lucene requests).

Why do you need to split your data every hours?
Do you see anything in logs? GC thing for example...

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 juil. 2013 à 10:13, Ankit Jain ankitj...@gmail.com a écrit :

Hi All,

We are creating indices on hour basis and we are firing queries on 2 day data. The number of records expected to hit is less than 100. But, the queries are taking unpredictable time (1 sec to 1 min).

Is queries will execute in all 48 indices in parallel or sequential?
What will be the main reason for performance hit?

Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks for the input.

On Wednesday, 3 July 2013 15:24:22 UTC+5:30, David Pilato wrote:

You should remove the index itself. Why removing records?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 3 juil. 2013 à 11:27, Ankit Jain <ankitj...@gmail.com <javascript:>> a
écrit :

Hi All,

Why do you need to split your data every hours?
We are getting 10 million records each hour and also we would like to
provide time based searching.

Also, I am using below code to delete records.

            DeleteByQueryRequestBuilder dbqb =

client.prepareDeleteByQuery(
existingIndices.toArray(new String[existingIndices
.size()])).setQuery(qb);
log.info("Request done :");
DeleteByQueryResponse deleteByQueryResponse =
dbqb.execute()
.actionGet();

dbqb.execute().actionGet(); -> This statement takes 1 min to get the
response from ES.

Regards,
Ankit

On Wednesday, 3 July 2013 13:58:11 UTC+5:30, David Pilato wrote:

parallel.

Let's say you are using default settings, so you have 5 shards per index.
You are running queries on 240 shards (that means 240 Lucene requests).

Why do you need to split your data every hours?
Do you see anything in logs? GC thing for example...

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 3 juil. 2013 à 10:13, Ankit Jain ankitj...@gmail.com a écrit :

Hi All,

We are creating indices on hour basis and we are firing queries on 2 day
data. The number of records expected to hit is less than 100. But, the
queries are taking unpredictable time (1 sec to 1 min).

Is queries will execute in all 48 indices in parallel or sequential?
What will be the main reason for performance hit?

Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.