ElasticSearch Node.Client Options

Elastic Stack Elasticsearch
1

We have client, data and master node configurations.

And we are seeing lots of OutOfMemory errors on node.client nodes.

One of the thing which we have some doubt is, node.client caches data for
some reason? I am hoping not.

If yes then what are the ways we can control this caching so client does
not cache any data.

And in general what are setting options which we can control through yml
for client nodes (node.client).

And what all operations happens on Node.Client when search with some query.

Regards,
VB.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hey,

client nodes do execute a reduce phase (merging all the results from a
search in order to find out the top-n results), and if you do deep
pagination this can take some memory. There is no data stored on a client
node (as node.data property is always set to false).

Do you have a stack trace of the exception, so one can see where the OOM
happens?

--Alex

On Tue, Jun 3, 2014 at 8:58 PM, VB vishal.batghare@gmail.com wrote:

We have client, data and master node configurations.

And we are seeing lots of OutOfMemory errors on node.client nodes.

One of the thing which we have some doubt is, node.client caches data for
some reason? I am hoping not.

If yes then what are the ways we can control this caching so client does
not cache any data.

And in general what are setting options which we can control through yml
for client nodes (node.client).

And what all operations happens on Node.Client when search with some query.

Regards,
VB.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9_qsrYQvrgk5dnGeyHhAL5e_a7QC%3Dd25DaYbyQwOSafA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks Alex for responding to my post from your busy schedule.

Please find attached screenshot of eclipse memory analyzer

On Monday, 16 June 2014 04:58:19 UTC-7, Alexander Reelsen wrote:

Hey,

client nodes do execute a reduce phase (merging all the results from a
search in order to find out the top-n results), and if you do deep
pagination this can take some memory. There is no data stored on a client
node (as node.data property is always set to false).

Do you have a stack trace of the exception, so one can see where the OOM
happens?

--Alex

On Tue, Jun 3, 2014 at 8:58 PM, VB <vishal....@gmail.com <javascript:>>
wrote:

We have client, data and master node configurations.

And we are seeing lots of OutOfMemory errors on node.client nodes.

One of the thing which we have some doubt is, node.client caches data for
some reason? I am hoping not.

If yes then what are the ways we can control this caching so client does
not cache any data.

And in general what are setting options which we can control through yml
for client nodes (node.client).

And what all operations happens on Node.Client when search with some
query.

Regards,
VB.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6434f582-272b-4131-a6ee-933bc2014f8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

And this stack trace.

[2014-06-04 14:47:12,939][INFO ][cluster.service ] [BUS2F2801F3]
master {new
[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true}, previous
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}}, removed
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-master_failed
([ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true})
[2014-06-04 14:48:03,969][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55503][489] duration [49.6s], collections [1]/[49.9s], total
[49.6s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[51.3mb]->[42.8mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:48:40,256][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55504][490] duration [35.7s], collections [1]/[36.2s], total
[35.7s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[42.8mb]->[58.6mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:49:30,335][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55505][491] duration [49.9s], collections [1]/[50s], total
[49.9s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[58.6mb]->[63.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:49:30,350][INFO ][discovery.zen ] [BUS2F2801F3]
master_left
[[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to ping, tried [3]
times, each with maximum [30s] timeout]
[2014-06-04 14:49:30,865][WARN ][discovery.zen ] [BUS2F2801F3]
not enough master nodes after master left (reason = failed to ping, tried
[3] times, each with maximum [30s] timeout), current nodes:
{[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5_j-Q][inet[/10.76.122.38:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][inet[/10.76.125.48:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300]]{client=true,
data=false,
master=false},[ELS-10.76.120.62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[BUS9364B62][YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][inet[/10.76.125.38:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.214][H8N9nIU0TKyGv_prKyRVCQ][inet[/10.76.124.214:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/10.76.120.87:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][inet[/10.76.122.67:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][inet[/10.76.121.129:9300]]{data=false,
max_local_storage_nodes=1,
master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-_AXw][inet[/10.76.122.20:9300]]{client=true,
data=false,
master=false},[EDI1AE4FD76][UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][inet[/10.76.125.46:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/10.76.120.82:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.34][kPSYm9G8R8i_z2skK_jq1g][inet[/10.76.122.34:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][inet[/10.76.125.43:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/10.76.124.227:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][inet[/10.76.124.225:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.61][_60f636_QsOPIWN0tKyN2A][inet[/10.76.120.61:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/10.76.124.18:9300]]{client=true,
data=false,
master=false},[BUS936E1B3][Vnr_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][inet[/10.76.121.186:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EC098][SvH6eHsHRoyz8_PEig46LA][inet[/10.76.120.83:9300]]{client=true,
data=false,
master=false},[ELS-10.76.122.39][swxUZAjCTCeBZS2-Uo_Bpw][inet[/10.76.122.39:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][inet[/10.76.120.63:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.37][TBw4vykLR_uRT76gt23UUQ][inet[/10.76.122.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og_DdvQ][inet[/10.76.122.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi_iA][inet[/10.76.120.86:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.50][jv7xpDnpTiOo21Kg_JgkVw][inet[/10.76.125.50:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][inet[/10.76.124.223:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/10.76.120.84:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.221][nB65C4WGRiSCntVZoovX_Q][inet[/10.76.124.221:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][inet[/10.76.124.224:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300]]{client=true,
data=false,
master=false},[STG0A4C78E2][ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true,
data=false, master=false},[BUS2F2801F3][al8pR84OSlOIhnpT0JSJHQ][inet[
BUS2F2801F3.SWLAB.RMSCLOUD.NET/10.76.122.123:9300]]{client=true
http://bus2f2801f3.swlab.rmscloud.net/10.76.122.123:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][inet[/10.76.124.218:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][inet[/10.76.125.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/10.76.121.248:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][inet[/10.76.125.49:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][inet[/10.76.122.35:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/10.76.120.85:9300]]{client=true,
data=false,
master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true,
data=false, master=false},}
[2014-06-04 14:50:14,937][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55506][492] duration [44.5s], collections [1]/[44.6s], total
[44.5s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[63.7mb]->[65.9mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:50:14,937][INFO ][cluster.service ] [BUS2F2801F3]
removed
{[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5_j-Q][inet[/10.76.122.38:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][inet[/10.76.125.48:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300]]{client=true,
data=false,
master=false},[ELS-10.76.120.62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[BUS9364B62][YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][inet[/10.76.125.38:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.214][H8N9nIU0TKyGv_prKyRVCQ][inet[/10.76.124.214:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/10.76.120.87:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][inet[/10.76.122.67:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][inet[/10.76.121.129:9300]]{data=false,
max_local_storage_nodes=1,
master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-_AXw][inet[/10.76.122.20:9300]]{client=true,
data=false,
master=false},[EDI1AE4FD76][UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][inet[/10.76.125.46:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/10.76.120.82:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.34][kPSYm9G8R8i_z2skK_jq1g][inet[/10.76.122.34:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][inet[/10.76.125.43:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/10.76.124.227:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][inet[/10.76.124.225:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.61][_60f636_QsOPIWN0tKyN2A][inet[/10.76.120.61:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/10.76.124.18:9300]]{client=true,
data=false,
master=false},[BUS936E1B3][Vnr_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][inet[/10.76.121.186:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EC098][SvH6eHsHRoyz8_PEig46LA][inet[/10.76.120.83:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1,
master=true},[ELS-10.76.122.39][swxUZAjCTCeBZS2-Uo_Bpw][inet[/10.76.122.39:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][inet[/10.76.120.63:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.37][TBw4vykLR_uRT76gt23UUQ][inet[/10.76.122.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og_DdvQ][inet[/10.76.122.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi_iA][inet[/10.76.120.86:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.50][jv7xpDnpTiOo21Kg_JgkVw][inet[/10.76.125.50:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][inet[/10.76.124.223:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/10.76.120.84:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.221][nB65C4WGRiSCntVZoovX_Q][inet[/10.76.124.221:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][inet[/10.76.124.224:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300]]{client=true,
data=false,
master=false},[STG0A4C78E2][ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][inet[/10.76.124.218:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][inet[/10.76.125.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/10.76.121.248:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][inet[/10.76.125.49:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][inet[/10.76.122.35:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/10.76.120.85:9300]]{client=true,
data=false,
master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true,
data=false, master=false},}, reason: zen-disco-master_failed
([ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true})
[2014-06-04 14:52:25,048][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55507][493] duration [43.7s], collections [1]/[43.7s], total
[43.7s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.9mb]->[66.3mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 15:07:08,535][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
[2014-06-04 15:07:08,535][DEBUG][action.search.type ] [BUS2F2801F3]
[1004_exposureindex][49], node[ot8ch3V6TYGKOLYnCY9cYw], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@14495084]
lastShard [true]
org.elasticsearch.transport.SendRequestTransportException:
[ELS-10.76.125.44][inet[/10.76.125.44:9300]][search/phase/query]
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:202)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:173)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:208)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[ELS-10.76.125.44][inet[/10.76.125.44:9300]] Node not connected
at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:859)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:540)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:189)
... 8 more

On Wednesday, 18 June 2014 14:51:56 UTC-7, VB wrote:

Thanks Alex for responding to my post from your busy schedule.

Please find attached screenshot of eclipse memory analyzer

On Monday, 16 June 2014 04:58:19 UTC-7, Alexander Reelsen wrote:

Hey,

client nodes do execute a reduce phase (merging all the results from a
search in order to find out the top-n results), and if you do deep
pagination this can take some memory. There is no data stored on a client
node (as node.data property is always set to false).

Do you have a stack trace of the exception, so one can see where the OOM
happens?

--Alex

On Tue, Jun 3, 2014 at 8:58 PM, VB vishal....@gmail.com wrote:

We have client, data and master node configurations.

And we are seeing lots of OutOfMemory errors on node.client nodes.

One of the thing which we have some doubt is, node.client caches data
for some reason? I am hoping not.

If yes then what are the ways we can control this caching so client does
not cache any data.

And in general what are setting options which we can control through yml
for client nodes (node.client).

And what all operations happens on Node.Client when search with some
query.

Regards,
VB.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7dc96c21-bfc1-4201-9544-cd6db7b02f5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Hey,

a client node with a full 10gb heap and garbage collection does not free
anything, so those objects are still in use (which clearly explains THAT
the OOM happens, but not WHY). Do you have huge searches going on spanning
a lot of shards with deep pagination (all the time). Do you have some sort
of backup mechanism which might be response for this? Anything from a
search perspective which might lead to excessive memory usage?

--Alex

On Fri, Jun 20, 2014 at 12:15 AM, VB vishal.batghare@gmail.com wrote:

And this stack trace.

[2014-06-04 14:47:12,939][INFO ][cluster.service ] [BUS2F2801F3]
master {new [ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true}, previous [ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}}, removed {[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-master_failed
([ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300
]]{data=false, max_local_storage_nodes=1, master=true})
[2014-06-04 14:48:03,969][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55503][489] duration [49.6s], collections [1]/[49.9s], total
[49.6s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [51.3mb]->[42.8mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:48:40,256][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55504][490] duration [35.7s], collections [1]/[36.2s], total
[35.7s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [42.8mb]->[58.6mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:49:30,335][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55505][491] duration [49.9s], collections [1]/[50s], total
[49.9s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [58.6mb]->[63.7mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:49:30,350][INFO ][discovery.zen ] [BUS2F2801F3]
master_left [[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to ping, tried [3]
times, each with maximum [30s] timeout]
[2014-06-04 14:49:30,865][WARN ][discovery.zen ] [BUS2F2801F3]
not enough master nodes after master left (reason = failed to ping, tried
[3] times, each with maximum [30s] timeout), current nodes:
{[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300
]]{max_local_storage_nodes=1, master=false},[ELS-10.76.122.
38][5V8bqkEzTP2TzMukB5_j-Q][inet[/10.76.122.38:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][
inet[/10.76.125.48:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300
]]{client=true, data=false, master=false},[ELS-10.76.120.
62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true,
data=false, max_local_storage_nodes=1, master=false},[BUS9364B62][
YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true,
data=false, master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][
inet[/10.76.125.38:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300
]]{client=true, data=false, master=false},[ELS-10.76.124.
214][H8N9nIU0TKyGv_prKyRVCQ][inet[/10.76.124.214:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/
10.76.120.87:9300]]{client=true, data=false, master=false},[ELS-10.76.125.
40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][
inet[/10.76.122.67:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][
inet[/10.76.121.129:9300]]{data=false, max_local_storage_nodes=1,
master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-_AXw][inet[/10.76.122.20:9300
]]{client=true, data=false, master=false},[EDI1AE4FD76][
UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true,
data=false, master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][
inet[/10.76.125.46:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/
10.76.120.82:9300]]{client=true, data=false, master=false},[ELS-10.76.121.
188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.34][kPSYm9G8R8i_z2skK_jq1g][
inet[/10.76.122.34:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][
inet[/10.76.125.43:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/
10.76.124.227:9300]]{client=true, data=false,
master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][
inet[/10.76.124.225:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.61][_60f636_QsOPIWN0tKyN2A][
inet[/10.76.120.61:9300]]{client=true, data=false,
max_local_storage_nodes=1, master=false},[ELS-10.76.125.
47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/
10.76.124.18:9300]]{client=true, data=false, master=false},[BUS936E1B3][
Vnr_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true,
data=false, master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][
inet[/10.76.121.186:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EC098][SvH6eHsHRoyz8_PEig46LA][inet[/
10.76.120.83:9300]]{client=true, data=false, master=false},[ELS-10.76.122.
39][swxUZAjCTCeBZS2-Uo_Bpw][inet[/10.76.122.39:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][
inet[/10.76.120.63:9300]]{client=true, data=false,
max_local_storage_nodes=1, master=false},[ELS-10.76.125.
45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.37][TBw4vykLR_uRT76gt23UUQ][
inet[/10.76.122.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og_DdvQ][
inet[/10.76.122.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi_iA][inet[/
10.76.120.86:9300]]{client=true, data=false, master=false},[ELS-10.76.125.
50][jv7xpDnpTiOo21Kg_JgkVw][inet[/10.76.125.50:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][
inet[/10.76.124.223:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/
10.76.120.84:9300]]{client=true, data=false, master=false},[ELS-10.76.124.
221][nB65C4WGRiSCntVZoovX_Q][inet[/10.76.124.221:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][
inet[/10.76.124.224:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300
]]{client=true, data=false, master=false},[STG0A4C78E2][
ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true,
data=false, master=false},[BUS2F2801F3][al8pR84OSlOIhnpT0JSJHQ][inet[B US2F2801F3.SWLAB.RMSCLOUD.NET/10.76.122.123:9300]]{client=true
http://bus2f2801f3.swlab.rmscloud.net/10.76.122.123:9300]]{client=true,
data=false, master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][
inet[/10.76.124.218:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][
inet[/10.76.125.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/
10.76.121.248:9300]]{client=true, data=false,
master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][
inet[/10.76.125.49:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][
inet[/10.76.122.35:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/
10.76.120.85:9300]]{client=true, data=false,
master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true,
data=false, master=false},}
[2014-06-04 14:50:14,937][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55506][492] duration [44.5s], collections [1]/[44.6s], total
[44.5s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [63.7mb]->[65.9mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:50:14,937][INFO ][cluster.service ] [BUS2F2801F3]
removed {[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/
10.76.125.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5_j-Q][
inet[/10.76.122.38:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][
inet[/10.76.125.48:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300
]]{client=true, data=false, master=false},[ELS-10.76.120.
62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true,
data=false, max_local_storage_nodes=1, master=false},[BUS9364B62][
YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true,
data=false, master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][
inet[/10.76.125.38:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300
]]{client=true, data=false, master=false},[ELS-10.76.124.
214][H8N9nIU0TKyGv_prKyRVCQ][inet[/10.76.124.214:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/
10.76.120.87:9300]]{client=true, data=false, master=false},[ELS-10.76.125.
40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][
inet[/10.76.122.67:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][
inet[/10.76.121.129:9300]]{data=false, max_local_storage_nodes=1,
master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-_AXw][inet[/10.76.122.20:9300
]]{client=true, data=false, master=false},[EDI1AE4FD76][
UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true,
data=false, master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][
inet[/10.76.125.46:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/
10.76.120.82:9300]]{client=true, data=false, master=false},[ELS-10.76.121.
188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.34][kPSYm9G8R8i_z2skK_jq1g][
inet[/10.76.122.34:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][
inet[/10.76.125.43:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/
10.76.124.227:9300]]{client=true, data=false,
master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][
inet[/10.76.124.225:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.61][_60f636_QsOPIWN0tKyN2A][
inet[/10.76.120.61:9300]]{client=true, data=false,
max_local_storage_nodes=1, master=false},[ELS-10.76.125.
47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/
10.76.124.18:9300]]{client=true, data=false, master=false},[BUS936E1B3][
Vnr_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true,
data=false, master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][
inet[/10.76.121.186:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EC098][SvH6eHsHRoyz8_PEig46LA][inet[/
10.76.120.83:9300]]{client=true, data=false, master=false},[ELS-10.76.121.
131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true},[ELS-10.76.122.
39][swxUZAjCTCeBZS2-Uo_Bpw][inet[/10.76.122.39:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][
inet[/10.76.120.63:9300]]{client=true, data=false,
max_local_storage_nodes=1, master=false},[ELS-10.76.125.
45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.37][TBw4vykLR_uRT76gt23UUQ][
inet[/10.76.122.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og_DdvQ][
inet[/10.76.122.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi_iA][inet[/
10.76.120.86:9300]]{client=true, data=false, master=false},[ELS-10.76.125.
50][jv7xpDnpTiOo21Kg_JgkVw][inet[/10.76.125.50:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][
inet[/10.76.124.223:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/
10.76.120.84:9300]]{client=true, data=false, master=false},[ELS-10.76.124.
221][nB65C4WGRiSCntVZoovX_Q][inet[/10.76.124.221:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][
inet[/10.76.124.224:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300
]]{client=true, data=false, master=false},[STG0A4C78E2][
ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true,
data=false, master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][
inet[/10.76.124.218:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][
inet[/10.76.125.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/
10.76.121.248:9300]]{client=true, data=false,
master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][
inet[/10.76.125.49:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][
inet[/10.76.122.35:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/
10.76.120.85:9300]]{client=true, data=false,
master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true,
data=false, master=false},}, reason: zen-disco-master_failed
([ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true})
[2014-06-04 14:52:25,048][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55507][493] duration [43.7s], collections [1]/[43.7s], total
[43.7s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [65.9mb]->[66.3mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 15:07:08,535][WARN ][netty.channel.socket.nio.AbstractNioSelector]
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2014-06-04 15:07:08,535][DEBUG][action.search.type ] [BUS2F2801F3]
[1004_exposureindex][49], node[ot8ch3V6TYGKOLYnCY9cYw], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@14495084]
lastShard [true]
org.elasticsearch.transport.SendRequestTransportException:
[ELS-10.76.125.44][inet[/10.76.125.44:9300]][search/phase/query]
at org.elasticsearch.transport.TransportService.sendRequest(
TransportService.java:202)
at org.elasticsearch.transport.TransportService.sendRequest(
TransportService.java:173)
at org.elasticsearch.search.action.SearchServiceTransportAction.
sendExecuteQuery(SearchServiceTransportAction.java:208)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchA
ction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchA
ction.java:80)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$
BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$
BaseAsyncAction$4.run(TransportSearchTypeAction.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[ELS-10.76.125.44][inet[/10.76.125.44:9300]] Node not connected
at org.elasticsearch.transport.netty.NettyTransport.
nodeChannel(NettyTransport.java:859)
at org.elasticsearch.transport.netty.NettyTransport.
sendRequest(NettyTransport.java:540)
at org.elasticsearch.transport.TransportService.sendRequest(
TransportService.java:189)
... 8 more

On Wednesday, 18 June 2014 14:51:56 UTC-7, VB wrote:

Thanks Alex for responding to my post from your busy schedule.

Please find attached screenshot of eclipse memory analyzer

On Monday, 16 June 2014 04:58:19 UTC-7, Alexander Reelsen wrote:

Hey,

client nodes do execute a reduce phase (merging all the results from a
search in order to find out the top-n results), and if you do deep
pagination this can take some memory. There is no data stored on a client
node (as node.data property is always set to false).

Do you have a stack trace of the exception, so one can see where the OOM
happens?

--Alex

On Tue, Jun 3, 2014 at 8:58 PM, VB vishal....@gmail.com wrote:

We have client, data and master node configurations.

And we are seeing lots of OutOfMemory errors on node.client nodes.

One of the thing which we have some doubt is, node.client caches data
for some reason? I am hoping not.

If yes then what are the ways we can control this caching so client
does not cache any data.

And in general what are setting options which we can control through
yml for client nodes (node.client).

And what all operations happens on Node.Client when search with some
query.

Regards,
VB.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7dc96c21-bfc1-4201-9544-cd6db7b02f5f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7dc96c21-bfc1-4201-9544-cd6db7b02f5f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_9kNeDKZiGPhcXULkkr_iDDkcbQUdOvP4ezfWTN2_zEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.