Rails - tire - elasticsearch : need to restart elasticsearch server

oto_iashvili · June 27, 2013, 1:37pm

HI

I use rails - tire - elasticsearch, everythink is mainly working very well,
but just from time to time, my server start to be very slow. So I have to
restart elasticsearch service and then everything go fine again.

[image: graph from newrelic]

I have impression that it happens after bulk inserts (around 6000
products). Can it be linked ? Inserts last like 2min max, but still after
server has problem.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

oto_iashvili · June 29, 2013, 12:30am

finally it is not linked to bulk inserts

i have only this line in log
[2013-06-29 01:15:32,767][WARN ][monitor.jvm ] [Jon Spectre]
[gc][ParNew][26438][9941] duration [3.4s], collections [1]/[5.2s], total
[3.4s]/[57.7s], memory [951.6mb]->[713.7mb]/[989.8mb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[241.1mb]->[31mb]/[273mb]}{[Par Survivor Space]
[32.2mb]->[0b]/[34.1mb]}{[CMS Old Gen] [678.3mb]->[682.6mb]/[682.6mb]}{[CMS
Perm Gen] [35mb]->[35mb]/[166mb]}

Does someone understand this ?

thanks

On Thursday, June 27, 2013 3:37:18 PM UTC+2, oto iashvili wrote:

HI

I use rails - tire - elasticsearch, everythink is mainly working very
well, but just from time to time, my server start to be very slow. So I
have to restart elasticsearch service and then everything go fine again.

[image: graph from newrelic]

I have impression that it happens after bulk inserts (around 6000
products). Can it be linked ? Inserts last like 2min max, but still after
server has problem.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · June 29, 2013, 7:13am

Are you using default memory settings?
Is it a production server?
Do you have many users searching on it? Do you use facets or sort?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juin 2013 à 02:30, oto iashvili optimum.dulopin@laposte.net a écrit :

finally it is not linked to bulk inserts

i have only this line in log
[2013-06-29 01:15:32,767][WARN ][monitor.jvm ] [Jon Spectre] [gc][ParNew][26438][9941] duration [3.4s], collections [1]/[5.2s], total [3.4s]/[57.7s], memory [951.6mb]->[713.7mb]/[989.8mb], all_pools {[Code Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space] [241.1mb]->[31mb]/[273mb]}{[Par Survivor Space] [32.2mb]->[0b]/[34.1mb]}{[CMS Old Gen] [678.3mb]->[682.6mb]/[682.6mb]}{[CMS Perm Gen] [35mb]->[35mb]/[166mb]}

Does someone understand this ?

thanks

On Thursday, June 27, 2013 3:37:18 PM UTC+2, oto iashvili wrote:

HI

I use rails - tire - elasticsearch, everythink is mainly working very well, but just from time to time, my server start to be very slow. So I have to restart elasticsearch service and then everything go fine again.

I have impression that it happens after bulk inserts (around 6000 products). Can it be linked ? Inserts last like 2min max, but still after server has problem.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

oto_iashvili · June 29, 2013, 11:33am

Hi,

thanks for answer
Im really new to elasticsearch, so Im not really sure what u mean by
"default memory settings"
here my _cluster/nodes/?jvm

{"ok":true,"cluster_name":"elasticsearch","nodes":{"c9VvXkUfQ7qze_jWp-dPwg":{"name":"Airborne","transport_address":"inet[/xx.xx.xx.xx:9300]","hostname":"xxxxxxx","version":"0.90.0","http_address":"inet[/xx.xx.xx.xx:9200]","jvm":{"pid":12331,"version":"1.7.0_21","vm_name":"OpenJDK
64-Bit Server VM","vm_version":"23.7-b01","vm_vendor":"Oracle
Corporation","start_time":1372496399939,"mem":{"heap_init":"1gb","heap_init_in_bytes":1073741824,"heap_max":"989.8mb","heap_max_in_bytes":1037959168,"non_heap_init":"23.1mb","non_heap_init_in_bytes":24313856,"non_heap_max":"214mb","non_heap_max_in_bytes":224395264,"direct_max":"989.8mb","direct_max_in_bytes":1037959168}}}}}

and my _cluster/nodes/?process

{"ok":true,"cluster_name":"elasticsearch","nodes":{"c9VvXkUfQ7qze_jWp-dPwg":{"name":"Airborne","transport_address":"inet[/xx.xx.xx.xx:9300]","hostname":"xxxxxx","version":"0.90.0","http_address":"inet[/xx.xx.xx.xx:9200]","process":{"refresh_interval":1000,"id":12331,"max_file_descriptors":65535}}}}

yes it is in production server.

Depends what u call "many users". According to newrelic, I have around 1000
requests per minutes, 2/3 are for search.
I dont use facet yet. I plan it for later. I use sort by "date, id" or
"price, date".

thanks again for your help

On Saturday, June 29, 2013 9:13:40 AM UTC+2, David Pilato wrote:

Are you using default memory settings?
Is it a production server?
Do you have many users searching on it? Do you use facets or sort?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juin 2013 à 02:30, oto iashvili <optimum...@laposte.net<javascript:>>
a écrit :

finally it is not linked to bulk inserts

i have only this line in log
[2013-06-29 01:15:32,767][WARN ][monitor.jvm ] [Jon Spectre]
[gc][ParNew][26438][9941] duration [3.4s], collections [1]/[5.2s], total
[3.4s]/[57.7s], memory [951.6mb]->[713.7mb]/[989.8mb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[241.1mb]->[31mb]/[273mb]}{[Par Survivor Space]
[32.2mb]->[0b]/[34.1mb]}{[CMS Old Gen] [678.3mb]->[682.6mb]/[682.6mb]}{[CMS
Perm Gen] [35mb]->[35mb]/[166mb]}

Does someone understand this ?

thanks

On Thursday, June 27, 2013 3:37:18 PM UTC+2, oto iashvili wrote:

HI

I use rails - tire - elasticsearch, everythink is mainly working very
well, but just from time to time, my server start to be very slow. So I
have to restart elasticsearch service and then everything go fine again.

[image: graph from newrelic]

I have impression that it happens after bulk inserts (around 6000
products). Can it be linked ? Inserts last like 2min max, but still after
server has problem.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

oto_iashvili · June 30, 2013, 11:27am

Hi,

I answered u, but someone delete my post "This message has been deleted.".
I dont understant why ?

so i will answer again :
I not expert in in rails so Im not sure what u are calling "default memory
settings"
here my index?jvm
"mem":{"heap_init":"1gb","heap_init_in_bytes":1073741824,"heap_max":"989.8mb","heap_max_in_bytes":1037959168,"non_heap_init":"23.1mb","non_heap_init_in_bytes":24313856,"non_heap_max":"214mb","non_heap_max_in_bytes":224395264,"direct_max":"989.8mb","direct_max_in_bytes":1037959168

and here my index?process
"process":{"refresh_interval":1000,"id":12331,"max_file_descriptors":65535

Yes it is a production server

According to newrelic I have around 1000 requests per minute, 2/3 are for
searching. I dont use facet, but I use sort by [date, id] or [price, id]

thanks for your help

On Saturday, June 29, 2013 9:13:40 AM UTC+2, David Pilato wrote:

Are you using default memory settings?
Is it a production server?
Do you have many users searching on it? Do you use facets or sort?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juin 2013 à 02:30, oto iashvili <optimum...@laposte.net<javascript:>>
a écrit :

finally it is not linked to bulk inserts

i have only this line in log
[2013-06-29 01:15:32,767][WARN ][monitor.jvm ] [Jon Spectre]
[gc][ParNew][26438][9941] duration [3.4s], collections [1]/[5.2s], total
[3.4s]/[57.7s], memory [951.6mb]->[713.7mb]/[989.8mb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[241.1mb]->[31mb]/[273mb]}{[Par Survivor Space]
[32.2mb]->[0b]/[34.1mb]}{[CMS Old Gen] [678.3mb]->[682.6mb]/[682.6mb]}{[CMS
Perm Gen] [35mb]->[35mb]/[166mb]}

Does someone understand this ?

thanks

On Thursday, June 27, 2013 3:37:18 PM UTC+2, oto iashvili wrote:

HI

I use rails - tire - elasticsearch, everythink is mainly working very
well, but just from time to time, my server start to be very slow. So I
have to restart elasticsearch service and then everything go fine again.

[image: graph from newrelic]

I have impression that it happens after bulk inserts (around 6000
products). Can it be linked ? Inserts last like 2min max, but still after
server has problem.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · July 1, 2013, 7:18am

So you need probably to give more memory to your elasticsearch instance.
How many docs do you have in total?

Sorting and facetting use memory. If you have many concurrent users, that means that elasticsearch has to load many data in memory to serve the search.

How much physical memory do you have on your instance? Consider giving half of it to Elasticsearch and the rest for the OS.

HTH

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 30 juin 2013 à 13:27, oto iashvili optimum.dulopin@laposte.net a écrit :

Hi,

I answered u, but someone delete my post "This message has been deleted.". I dont understant why ?

so i will answer again :
I not expert in in rails so Im not sure what u are calling "default memory settings"
here my index?jvm
"mem":{"heap_init":"1gb","heap_init_in_bytes":1073741824,"heap_max":"989.8mb","heap_max_in_bytes":1037959168,"non_heap_init":"23.1mb","non_heap_init_in_bytes":24313856,"non_heap_max":"214mb","non_heap_max_in_bytes":224395264,"direct_max":"989.8mb","direct_max_in_bytes":1037959168

and here my index?process
"process":{"refresh_interval":1000,"id":12331,"max_file_descriptors":65535

Yes it is a production server

According to newrelic I have around 1000 requests per minute, 2/3 are for searching. I dont use facet, but I use sort by [date, id] or [price, id]

thanks for your help

On Saturday, June 29, 2013 9:13:40 AM UTC+2, David Pilato wrote:
Are you using default memory settings?
Is it a production server?
Do you have many users searching on it? Do you use facets or sort?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juin 2013 à 02:30, oto iashvili optimum...@laposte.net a écrit :

finally it is not linked to bulk inserts

i have only this line in log
[2013-06-29 01:15:32,767][WARN ][monitor.jvm ] [Jon Spectre] [gc][ParNew][26438][9941] duration [3.4s], collections [1]/[5.2s], total [3.4s]/[57.7s], memory [951.6mb]->[713.7mb]/[989.8mb], all_pools {[Code Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space] [241.1mb]->[31mb]/[273mb]}{[Par Survivor Space] [32.2mb]->[0b]/[34.1mb]}{[CMS Old Gen] [678.3mb]->[682.6mb]/[682.6mb]}{[CMS Perm Gen] [35mb]->[35mb]/[166mb]}

Does someone understand this ?

thanks

On Thursday, June 27, 2013 3:37:18 PM UTC+2, oto iashvili wrote:
HI

I use rails - tire - elasticsearch, everythink is mainly working very well, but just from time to time, my server start to be very slow. So I have to restart elasticsearch service and then everything go fine again.

I have impression that it happens after bulk inserts (around 6000 products). Can it be linked ? Inserts last like 2min max, but still after server has problem.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

oto_iashvili · July 5, 2013, 11:56pm

hi,
thanks for advices. I have 64go of ram. I give 1Go to elasticsearch heap
size as it is saying this is maximum.

anyway, i found out that problem was raise when users where putting to big
number, as rails converts this to "Infinity", elasticsearch was raising
error. But I dont understand why when such error occurs, why elasticsearch
is using all ressources instead of just letting go.

On Monday, July 1, 2013 9:18:27 AM UTC+2, David Pilato wrote:

So you need probably to give more memory to your elasticsearch instance.
How many docs do you have in total?

Sorting and facetting use memory. If you have many concurrent users, that
means that elasticsearch has to load many data in memory to serve the
search.

How much physical memory do you have on your instance? Consider giving
half of it to Elasticsearch and the rest for the OS.

HTH

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 30 juin 2013 à 13:27, oto iashvili <optimum...@laposte.net<javascript:>>
a écrit :

Hi,

I answered u, but someone delete my post "This message has been deleted.".
I dont understant why ?

so i will answer again :
I not expert in in rails so Im not sure what u are calling "default memory
settings"
here my index?jvm

"mem":{"heap_init":"1gb","heap_init_in_bytes":1073741824,"heap_max":"989.8mb","heap_max_in_bytes":1037959168,"non_heap_init":"23.1mb","non_heap_init_in_bytes":24313856,"non_heap_max":"214mb","non_heap_max_in_bytes":224395264,"direct_max":"989.8mb","direct_max_in_bytes":1037959168

and here my index?process
"process":{"refresh_interval":1000,"id":12331,"max_file_descriptors":65535

Yes it is a production server

According to newrelic I have around 1000 requests per minute, 2/3 are for
searching. I dont use facet, but I use sort by [date, id] or [price, id]

thanks for your help

On Saturday, June 29, 2013 9:13:40 AM UTC+2, David Pilato wrote:

Are you using default memory settings?
Is it a production server?
Do you have many users searching on it? Do you use facets or sort?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juin 2013 à 02:30, oto iashvili optimum...@laposte.net a écrit :

finally it is not linked to bulk inserts

i have only this line in log
[2013-06-29 01:15:32,767][WARN ][monitor.jvm ] [Jon Spectre]
[gc][ParNew][26438][9941] duration [3.4s], collections [1]/[5.2s], total
[3.4s]/[57.7s], memory [951.6mb]->[713.7mb]/[989.8mb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[241.1mb]->[31mb]/[273mb]}{[Par Survivor Space]
[32.2mb]->[0b]/[34.1mb]}{[CMS Old Gen] [678.3mb]->[682.6mb]/[682.6mb]}{[CMS
Perm Gen] [35mb]->[35mb]/[166mb]}

Does someone understand this ?

thanks

On Thursday, June 27, 2013 3:37:18 PM UTC+2, oto iashvili wrote:

HI

I use rails - tire - elasticsearch, everythink is mainly working very
well, but just from time to time, my server start to be very slow. So I
have to restart elasticsearch service and then everything go fine again.

[image: graph from newrelic]

I have impression that it happens after bulk inserts (around 6000
products). Can it be linked ? Inserts last like 2min max, but still after
server has problem.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.