ES for persistent data on single node => performance problems

Elastic Stack Elasticsearch
1

Hey everyone

I'm running ES on a single server through a single node with persistent
data. Following some guides I found online, I added the following to
elasticsearch.yml:

Not refreshing the index

index.refresh_interval: -1

Uses less memory this way (default 1)

index.term_index_divisor: 5

also less memory usage

index.term_index_interval: 256

Nevertheless, the server becomes really sluggish after some time (about a
week). The last things I see in the log are:

[2013-06-16 21:15:23,349][DEBUG][action.search.type ] [Pretty
Persuasions] [186556171] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search context
found for id [186556171]
at org.elasticsearch.search.SearchService.findContext(SearchService.java:
455)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.
java:428)
at org.elasticsearch.search.action.SearchServiceTransportAction.
sendExecuteFetch(SearchServiceTransportAction.java:345)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(
TransportSearchQueryThenFetchAction.java:149)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction$2.run(
TransportSearchQueryThenFetchAction.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:615)
at java.lang.Thread.run(Thread.java:722)
[2013-06-16 21:18:19,913][WARN ][monitor.jvm ] [Pretty
Persuasions] [gc][ParNew][867853][376928] duration [1.2s], collections [1]/[
1.2s], total [1.2s]/[2.6h], memory [854.3mb]->[531.4mb]/[2.9gb], all_pools
{[Code Cache] [7.7mb]->[7.7mb]/[48mb]}{[Par Eden Space] [502.8mb]->[366.7kb
]/[532.5mb]}{[Par Survivor Space] [28.4mb]->[66.5mb]/[66.5mb]}{[CMS Old Gen]
[323mb]->[464.5mb]/[2.3gb]}{[CMS Perm Gen] [31.8mb]->[31.8mb]/[82mb]}
[2013-06-16 21:21:24,234][INFO ][monitor.jvm ] [Pretty
Persuasions] [gc][ParNew][868037][377034] duration [997ms], collections [1
]/[1.1s], total [997ms]/[2.6h], memory [1.1gb]->[1010.8mb]/[2.9gb],all_pools
{[Code Cache] [7.7mb]->[7.7mb]/[48mb]}{[Par Eden Space] [406.3mb]->[943kb]/[
532.5mb]}{[Par Survivor Space] [23.3mb]->[66.5mb]/[66.5mb]}{[CMS Old Gen] [
796.2mb]->[943.4mb]/[2.3gb]}{[CMS Perm Gen] [31.8mb]->[31.8mb]/[82mb]}

When I restart the server, the performance goes up again. Did I make some
mistake in my configuration?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi!

For my single-node Elasticsearch instances (and indeed even for the 3-node
cluster), I don't ever set any of those configurations; the defaults work
fine.

For my Java bulk-loader, I temporarily set the index refresh rate to 120s
(not -1; 120s was recommended somewhere in Shay's documentation), then
bulk-load in chunks of 4096 documents at a time, and then set the refresh
rate back to 1s (the default) when done.

For one-at-a-time updates, even under moderate load, I always leave the
default refresh rate.

This strategy never seems to have any issues such as the one you describe.

Hope this helps!

Brian

On Monday, June 17, 2013 5:08:45 AM UTC-4, panmari wrote:

Hey everyone

I'm running ES on a single server through a single node with persistent
data. Following some guides I found online, I added the following to
elasticsearch.yml:

Not refreshing the index

index.refresh_interval: -1

Uses less memory this way (default 1)

index.term_index_divisor: 5

also less memory usage

index.term_index_interval: 256

Nevertheless, the server becomes really sluggish after some time (about a
week). The last things I see in the log are:

[2013-06-16 21:15:23,349][DEBUG][action.search.type ] [Pretty
Persuasions] [186556171] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search context
found for id [186556171]
at org.elasticsearch.search.SearchService.findContext(SearchService.java:
455)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService
.java:428)
at org.elasticsearch.search.action.SearchServiceTransportAction.
sendExecuteFetch(SearchServiceTransportAction.java:345)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(
TransportSearchQueryThenFetchAction.java:149)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction$2.run(
TransportSearchQueryThenFetchAction.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:615)
at java.lang.Thread.run(Thread.java:722)
[2013-06-16 21:18:19,913][WARN ][monitor.jvm ] [Pretty
Persuasions] [gc][ParNew][867853][376928] duration [1.2s], collections [1
]/[1.2s], total [1.2s]/[2.6h], memory [854.3mb]->[531.4mb]/[2.9gb],all_pools
{[Code Cache] [7.7mb]->[7.7mb]/[48mb]}{[Par Eden Space] [502.8mb]->[
366.7kb]/[532.5mb]}{[Par Survivor Space] [28.4mb]->[66.5mb]/[66.5mb]}{[CMS
Old Gen] [323mb]->[464.5mb]/[2.3gb]}{[CMS Perm Gen] [31.8mb]->[31.8mb]/[
82mb]}
[2013-06-16 21:21:24,234][INFO ][monitor.jvm ] [Pretty
Persuasions] [gc][ParNew][868037][377034] duration [997ms], collections [1
]/[1.1s], total [997ms]/[2.6h], memory [1.1gb]->[1010.8mb]/[2.9gb],all_pools
{[Code Cache] [7.7mb]->[7.7mb]/[48mb]}{[Par Eden Space] [406.3mb]->[943kb
]/[532.5mb]}{[Par Survivor Space] [23.3mb]->[66.5mb]/[66.5mb]}{[CMS Old
Gen] [796.2mb]->[943.4mb]/[2.3gb]}{[CMS Perm Gen] [31.8mb]->[31.8mb]/[82mb
]}

When I restart the server, the performance goes up again. Did I make some
mistake in my configuration?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hey Brian

I'll try vanilla configuration then... How much heap space are you giving
ES? And how much memory do you have available?

On Monday, June 17, 2013 3:53:44 PM UTC+2, InquiringMind wrote:

Hi!

For my single-node Elasticsearch instances (and indeed even for the 3-node
cluster), I don't ever set any of those configurations; the defaults work
fine.

For my Java bulk-loader, I temporarily set the index refresh rate to 120s
(not -1; 120s was recommended somewhere in Shay's documentation), then
bulk-load in chunks of 4096 documents at a time, and then set the refresh
rate back to 1s (the default) when done.

For one-at-a-time updates, even under moderate load, I always leave the
default refresh rate.

This strategy never seems to have any issues such as the one you describe.

Hope this helps!

Brian

On Monday, June 17, 2013 5:08:45 AM UTC-4, panmari wrote:

Hey everyone

I'm running ES on a single server through a single node with persistent
data. Following some guides I found online, I added the following to
elasticsearch.yml:

Not refreshing the index

index.refresh_interval: -1

Uses less memory this way (default 1)

index.term_index_divisor: 5

also less memory usage

index.term_index_interval: 256

Nevertheless, the server becomes really sluggish after some time (about a
week). The last things I see in the log are:

[2013-06-16 21:15:23,349][DEBUG][action.search.type ] [Pretty
Persuasions] [186556171] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search
context found for id [186556171]
at org.elasticsearch.search.SearchService.findContext(SearchService.java
:455)
at org.elasticsearch.search.SearchService.executeFetchPhase(
SearchService.java:428)
at org.elasticsearch.search.action.SearchServiceTransportAction.
sendExecuteFetch(SearchServiceTransportAction.java:345)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(
TransportSearchQueryThenFetchAction.java:149)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction$2.run(
TransportSearchQueryThenFetchAction.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:722)
[2013-06-16 21:18:19,913][WARN ][monitor.jvm ] [Pretty
Persuasions] [gc][ParNew][867853][376928] duration [1.2s], collections [1
]/[1.2s], total [1.2s]/[2.6h], memory [854.3mb]->[531.4mb]/[2.9gb],all_pools
{[Code Cache] [7.7mb]->[7.7mb]/[48mb]}{[Par Eden Space] [502.8mb]->[
366.7kb]/[532.5mb]}{[Par Survivor Space] [28.4mb]->[66.5mb]/[66.5mb]}{[CMS
Old Gen] [323mb]->[464.5mb]/[2.3gb]}{[CMS Perm Gen] [31.8mb]->[31.8mb]/[
82mb]}
[2013-06-16 21:21:24,234][INFO ][monitor.jvm ] [Pretty
Persuasions] [gc][ParNew][868037][377034] duration [997ms], collections [
1]/[1.1s], total [997ms]/[2.6h], memory [1.1gb]->[1010.8mb]/[2.9gb],all_pools
{[Code Cache] [7.7mb]->[7.7mb]/[48mb]}{[Par Eden Space] [406.3mb]->[943kb
]/[532.5mb]}{[Par Survivor Space] [23.3mb]->[66.5mb]/[66.5mb]}{[CMS Old
Gen] [796.2mb]->[943.4mb]/[2.3gb]}{[CMS Perm Gen] [31.8mb]->[31.8mb]/[
82mb]}

When I restart the server, the performance goes up again. Did I make some
mistake in my configuration?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

On my laptop (MacBook i7 quad core, 8 GB of RAM, but single slow disk), I
take all the defaults. On our single-node cluster testing machines (not
testing ES, but testing that uses ES to replace a production server), we
like to keep it as tiny as possible (lots of other things running on one
machine); in this case we start it with the -server option to the JVM, and
then set the memory to 256MB. Seems to work fine.

Brian

On Tuesday, June 18, 2013 10:50:49 AM UTC-4, panmari wrote:

Hey Brian

I'll try vanilla configuration then... How much heap space are you giving
ES? And how much memory do you have available?

On Monday, June 17, 2013 3:53:44 PM UTC+2, InquiringMind wrote:

Hi!

For my single-node Elasticsearch instances (and indeed even for the
3-node cluster), I don't ever set any of those configurations; the defaults
work fine.

For my Java bulk-loader, I temporarily set the index refresh rate to 120s
(not -1; 120s was recommended somewhere in Shay's documentation), then
bulk-load in chunks of 4096 documents at a time, and then set the refresh
rate back to 1s (the default) when done.

For one-at-a-time updates, even under moderate load, I always leave the
default refresh rate.

This strategy never seems to have any issues such as the one you describe.

Hope this helps!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.