Elasticsearch performance issue with load increasing reponse time is increasing

kondapallinaresh · February 27, 2014, 9:32am

Hi All

We had elasticsearch with 4 Nodes the the Cluster each 8gb memory.
when we are calling the elastic search query from java with 25 threads
first thread is taking 400 msec where as last is taking around 5secs its
more than 10 times.

we had 50 million of data with 5 shards and 0 replicas.

we are monitoring the jvm everything is running less than 25% usage and cpu
usage too is fine.

do we need to have any setting in yml to improve the performance.

here are logs for response time.

The time taken to retrun data serial number search::470
The time taken to retrun data serial number search::471
The time taken to retrun data serial number search::490
The time taken to retrun data serial number search::491
The time taken to retrun data serial number search::501
The time taken to retrun data serial number search::502
The time taken to retrun data serial number search::440
The time taken to retrun data serial number search::441
The time taken to retrun data serial number search::489
The time taken to retrun data serial number search::490
The time taken to retrun data serial number search::461
The time taken to retrun data serial number search::462
The time taken to retrun data serial number search::497
The time taken to retrun data serial number search::498
The time taken to retrun data serial number search::534
The time taken to retrun data serial number search::510
The time taken to retrun data serial number search::535
The time taken to retrun data serial number search::511
The time taken to retrun data serial number search::524
The time taken to retrun data serial number search::525
The time taken to retrun data serial number search::488
The time taken to retrun data serial number search::489
The time taken to retrun data serial number search::602
The time taken to retrun data serial number search::603
The time taken to retrun data serial number search::627
The time taken to retrun data serial number search::628
The time taken to retrun data serial number search::613
The time taken to retrun data serial number search::614
The time taken to retrun data serial number search::668
The time taken to retrun data serial number search::669
The time taken to retrun data serial number search::702
The time taken to retrun data serial number search::704
The time taken to retrun data serial number search::717
The time taken to retrun data serial number search::718
The time taken to retrun data serial number search::689
The time taken to retrun data serial number search::689
The time taken to retrun data serial number search::742
The time taken to retrun data serial number search::743
The time taken to retrun data serial number search::722
The time taken to retrun data serial number search::723
The time taken to retrun data serial number search::1005
The time taken to retrun data serial number search::1018
The time taken to retrun data serial number search::1006
The time taken to retrun data serial number search::1019
The time taken to retrun data serial number search::985
The time taken to retrun data serial number search::986
The time taken to retrun data serial number search::1078
The time taken to retrun data serial number search::1079
The time taken to retrun data serial number search::1563
The time taken to retrun data serial number search::1564
The time taken to retrun data serial number search::1585
The time taken to retrun data serial number search::1586
The time taken to retrun data serial number search::1712
The time taken to retrun data serial number search::1713
The time taken to retrun data serial number search::2047
The time taken to retrun data serial number search::2048
The time taken to retrun data serial number search::2073
The time taken to retrun data serial number search::2074
The time taken to retrun data serial number search::2065
The time taken to retrun data serial number search::2066
The time taken to retrun data serial number search::2875
The time taken to retrun data serial number search::2876
The time taken to retrun data serial number search::3065
The time taken to retrun data serial number search::3065
The time taken to retrun data serial number search::3068
The time taken to retrun data serial number search::3069
The time taken to retrun data serial number search::3070
The time taken to retrun data serial number search::3075
The time taken to retrun data serial number search::3071
The time taken to retrun data serial number search::3076
The time taken to retrun data serial number search::3092
The time taken to retrun data serial number search::3093
The time taken to retrun data serial number search::3529
The time taken to retrun data serial number search::3530
The time taken to retrun data serial number search::3570
The time taken to retrun data serial number search::3570
The time taken to retrun data serial number search::4124
The time taken to retrun data serial number search::4125
The time taken to retrun data serial number search::4152
The time taken to retrun data serial number search::4153
The time taken to retrun data serial number search::4153
The time taken to retrun data serial number search::4154
The time taken to retrun data serial number search::4149
The time taken to retrun data serial number search::4150
The time taken to retrun data serial number search::5165
The time taken to retrun data serial number search::5165
The time taken to retrun data serial number search::5134
The time taken to retrun data serial number search::5135
The time taken to retrun data serial number search::5146
The time taken to retrun data serial number search::5146
The time taken to retrun data serial number search::5291
The time taken to retrun data serial number search::5292
The time taken to retrun data serial number search::5290
The time taken to retrun data serial number search::5291
The time taken to retrun data serial number search::5639
The time taken to retrun data serial number search::5640

Thanks,
Naresh.K

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3ba6f043-e5a6-43e0-8755-1ad96b6de185%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Binh_Ly_2 · February 27, 2014, 2:21pm

While all the threads are running, can you check the search thread pool
stats:

curl "localhost:9200/_nodes/stats/thread_pool?pretty"

It is possible that 1 or more searches are in the queue so they would wait
until the search threadpool has availability again.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9d4bc226-b6f7-4077-8196-6932b0841dba%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · February 27, 2014, 4:38pm

It would help if you can post the query you use, and how the lines you send
correspond to the number of threads, and how you fire off the queries in
parallel.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoELuRqhiiFO5HO-LGB0d78tre-mi2YGLvUJ9VBLmjwwJA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

kondapallinaresh · February 27, 2014, 11:56pm

Here is the query's i am using

first query returns the parent instance id's based on those parent instance
is trigirreng the second query.

SearchResponse response =
elasticClient.prepareSearch("ibdataindex").setTypes("coveredibsa").setQuery(QueryBuilders.termsQuery("SERIAL_NUMBER",
serialNo))

.setSearchType(SearchType.QUERY_AND_FETCH).addField("PARENT_INSTANCE_ID").setFrom(0)
.setSize(20000)
.execute()
.actionGet();
results = response.getHits().hits();
LOGGER.info("The time taken to retrun data serial number search::"+
(System.currentTimeMillis()-time1));

String instanceQueryString = "";
int i=0;
for (SearchHit hit : results) {
instanceQueryString =instanceQueryString+"
"+hit.getFields().get("PARENT_INSTANCE_ID").getValue();

}
List<Map<String,Object>> hits = new ArrayList<Map<String,Object>>();
//qb =
QueryBuilders.queryString("PARENT_INSTANCE_ID:("+instanceQueryString+")");
long time2 = System.currentTimeMillis();

if(!instanceQueryString.equals("")){
response = elasticClient.prepareSearch("ibdataindex")
.setTypes("coveredibsa").setQuery(QueryBuilders.termsQuery("PARENT_INSTANCE_ID",serialNo))
.setSearchType(SearchType.QUERY_AND_FETCH).addFields(coulns.split(","))
.setFrom(0).setSize(10000)
.execute()
.actionGet();

I am using spring task scheduler in which i am passing 250 serial nos as
batch.

On Thursday, February 27, 2014 8:38:49 AM UTC-8, Jörg Prante wrote:

It would help if you can post the query you use, and how the lines you
send correspond to the number of threads, and how you fire off the queries
in parallel.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/85442edf-c452-4c04-92af-34c4fc5fa420%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

webish · May 14, 2014, 4:13am

Did you ever solve this?

On Thursday, February 27, 2014 4:32:29 AM UTC-5, kondapal...@gmail.com
wrote:

Hi All

We had elasticsearch with 4 Nodes the the Cluster each 8gb memory.
when we are calling the Elasticsearch query from java with 25 threads
first thread is taking 400 msec where as last is taking around 5secs its
more than 10 times.

we had 50 million of data with 5 shards and 0 replicas.

we are monitoring the jvm everything is running less than 25% usage and
cpu usage too is fine.

do we need to have any setting in yml to improve the performance.

here are logs for response time.

The time taken to retrun data serial number search::470
The time taken to retrun data serial number search::471
The time taken to retrun data serial number search::490
The time taken to retrun data serial number search::491
The time taken to retrun data serial number search::501
The time taken to retrun data serial number search::502
The time taken to retrun data serial number search::440
The time taken to retrun data serial number search::441
The time taken to retrun data serial number search::489
The time taken to retrun data serial number search::490
The time taken to retrun data serial number search::461
The time taken to retrun data serial number search::462
The time taken to retrun data serial number search::497
The time taken to retrun data serial number search::498
The time taken to retrun data serial number search::534
The time taken to retrun data serial number search::510
The time taken to retrun data serial number search::535
The time taken to retrun data serial number search::511
The time taken to retrun data serial number search::524
The time taken to retrun data serial number search::525
The time taken to retrun data serial number search::488
The time taken to retrun data serial number search::489
The time taken to retrun data serial number search::602
The time taken to retrun data serial number search::603
The time taken to retrun data serial number search::627
The time taken to retrun data serial number search::628
The time taken to retrun data serial number search::613
The time taken to retrun data serial number search::614
The time taken to retrun data serial number search::668
The time taken to retrun data serial number search::669
The time taken to retrun data serial number search::702
The time taken to retrun data serial number search::704
The time taken to retrun data serial number search::717
The time taken to retrun data serial number search::718
The time taken to retrun data serial number search::689
The time taken to retrun data serial number search::689
The time taken to retrun data serial number search::742
The time taken to retrun data serial number search::743
The time taken to retrun data serial number search::722
The time taken to retrun data serial number search::723
The time taken to retrun data serial number search::1005
The time taken to retrun data serial number search::1018
The time taken to retrun data serial number search::1006
The time taken to retrun data serial number search::1019
The time taken to retrun data serial number search::985
The time taken to retrun data serial number search::986
The time taken to retrun data serial number search::1078
The time taken to retrun data serial number search::1079
The time taken to retrun data serial number search::1563
The time taken to retrun data serial number search::1564
The time taken to retrun data serial number search::1585
The time taken to retrun data serial number search::1586
The time taken to retrun data serial number search::1712
The time taken to retrun data serial number search::1713
The time taken to retrun data serial number search::2047
The time taken to retrun data serial number search::2048
The time taken to retrun data serial number search::2073
The time taken to retrun data serial number search::2074
The time taken to retrun data serial number search::2065
The time taken to retrun data serial number search::2066
The time taken to retrun data serial number search::2875
The time taken to retrun data serial number search::2876
The time taken to retrun data serial number search::3065
The time taken to retrun data serial number search::3065
The time taken to retrun data serial number search::3068
The time taken to retrun data serial number search::3069
The time taken to retrun data serial number search::3070
The time taken to retrun data serial number search::3075
The time taken to retrun data serial number search::3071
The time taken to retrun data serial number search::3076
The time taken to retrun data serial number search::3092
The time taken to retrun data serial number search::3093
The time taken to retrun data serial number search::3529
The time taken to retrun data serial number search::3530
The time taken to retrun data serial number search::3570
The time taken to retrun data serial number search::3570
The time taken to retrun data serial number search::4124
The time taken to retrun data serial number search::4125
The time taken to retrun data serial number search::4152
The time taken to retrun data serial number search::4153
The time taken to retrun data serial number search::4153
The time taken to retrun data serial number search::4154
The time taken to retrun data serial number search::4149
The time taken to retrun data serial number search::4150
The time taken to retrun data serial number search::5165
The time taken to retrun data serial number search::5165
The time taken to retrun data serial number search::5134
The time taken to retrun data serial number search::5135
The time taken to retrun data serial number search::5146
The time taken to retrun data serial number search::5146
The time taken to retrun data serial number search::5291
The time taken to retrun data serial number search::5292
The time taken to retrun data serial number search::5290
The time taken to retrun data serial number search::5291
The time taken to retrun data serial number search::5639
The time taken to retrun data serial number search::5640

Thanks,
Naresh.K

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa1d262e-959a-46f7-b6a7-4e9f7559983b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.