SearchContextMissingException: No search context found for id

Elastic Stack Elasticsearch
1

Hello all:

We've been having periodic trouble with the below exception, and with
little luck trying to track it down.

Generally speaking, this seems to happen on parent / child queries.
Initially, we'll get this exception (and many more like it) and then search
will return the results for a time (maybe like a half hour or so) and then
we'll get the errors again.

The incantation for elasticsearch looks like this:

/usr/java/default/bin/java -Xms12g -Xmx12g -Xss256k

-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.path.home=/opt/elasticsearch -cp
:/opt/elasticsearch/lib/elasticsearch-0.20.0.RC1.jar:/opt/elasticsearch/lib/:/opt/elasticsearch/lib/sigar/
org.elasticsearch.bootstrap.Elasticsearch

the yml looks like this:

Unicast Discovery (disable multicast)

#discovery.zen.ping.multicast.enabled: false
#discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Name your cluster here to whatever.

cluster.name: XXXXX-search-qa
node.name: "Router QA"
node.master: true
node.data: false

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["x.compute-1.internal",
"y.ec2.internal", "z.ec2.internal", "a.compute-1.internal",
"b.compute-1.internal",
"c.ec2.internal" ]

#internal ip of this router
#m.ec2.internal
network.bind_host: kk.kkk.ll.llll
http.port: 9500
#field cache configuration
index.cache.field.max_size: 20000
index.cache.field.expire: 10m

This is the traceback:

[2012-12-13 15:16:05,196][DEBUG][action.search.type ] [Router QA]
[1107156] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Master01
QA][inet[/XXXXXXXX:9300]][search/phase/fetch/id]
Caused by: org.elasticsearch.search.SearchContextMissingException: No
search context found for id [1107156]
at
org.elasticsearch.search.SearchService.findContext(SearchService.java:459)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:432)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

--

2

Do these parent/child queries take a long time to complete?
Also do see a lot of Failed to send release search context warnings
in your log files?

Martijn

On 13 December 2012 16:42, James Reynolds eire1130@gmail.com wrote:

Hello all:

We've been having periodic trouble with the below exception, and with little
luck trying to track it down.

Generally speaking, this seems to happen on parent / child queries.
Initially, we'll get this exception (and many more like it) and then search
will return the results for a time (maybe like a half hour or so) and then
we'll get the errors again.

The incantation for elasticsearch looks like this:

/usr/java/default/bin/java -Xms12g -Xmx12g -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.path.home=/opt/elasticsearch -cp
:/opt/elasticsearch/lib/elasticsearch-0.20.0.RC1.jar:/opt/elasticsearch/lib/:/opt/elasticsearch/lib/sigar/
org.elasticsearch.bootstrap.Elasticsearch

the yml looks like this:

Unicast Discovery (disable multicast)

#discovery.zen.ping.multicast.enabled: false
#discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Name your cluster here to whatever.

cluster.name: XXXXX-search-qa
node.name: "Router QA"
node.master: true
node.data: false

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["x.compute-1.internal",
"y.ec2.internal", "z.ec2.internal", "a.compute-1.internal",
"b.compute-1.internal",
"c.ec2.internal" ]

#internal ip of this router
#m.ec2.internal
network.bind_host: kk.kkk.ll.llll
http.port: 9500
#field cache configuration
index.cache.field.max_size: 20000
index.cache.field.expire: 10m

This is the traceback:

[2012-12-13 15:16:05,196][DEBUG][action.search.type ] [Router QA]
[1107156] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Master01
QA][inet[/XXXXXXXX:9300]][search/phase/fetch/id]
Caused by: org.elasticsearch.search.SearchContextMissingException: No
search context found for id [1107156]
at
org.elasticsearch.search.SearchService.findContext(SearchService.java:459)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:432)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

--

--
Met vriendelijke groet,

Martijn van Groningen

--

3

Thank you for the response.

I do not see any "Failed to send release search context" in the logs on any
of the boxes.

Really, the only error I see is the ones I copied below.

The parent/child queries return reasonably fast once they are "warmed", but
if it's been a few minutes since the last query, it can take a few seconds
to return.

On Thursday, December 13, 2012 11:11:18 AM UTC-5, Martijn v Groningen wrote:

Do these parent/child queries take a long time to complete?
Also do see a lot of Failed to send release search context warnings
in your log files?

Martijn

On 13 December 2012 16:42, James Reynolds <eire...@gmail.com <javascript:>>
wrote:

Hello all:

We've been having periodic trouble with the below exception, and with
little
luck trying to track it down.

Generally speaking, this seems to happen on parent / child queries.
Initially, we'll get this exception (and many more like it) and then
search
will return the results for a time (maybe like a half hour or so) and
then
we'll get the errors again.

The incantation for elasticsearch looks like this:

/usr/java/default/bin/java -Xms12g -Xmx12g -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.path.home=/opt/elasticsearch -cp

:/opt/elasticsearch/lib/elasticsearch-0.20.0.RC1.jar:/opt/elasticsearch/lib/:/opt/elasticsearch/lib/sigar/

org.elasticsearch.bootstrap.Elasticsearch

the yml looks like this:

Unicast Discovery (disable multicast)

#discovery.zen.ping.multicast.enabled: false
#discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Name your cluster here to whatever.

cluster.name: XXXXX-search-qa
node.name: "Router QA"
node.master: true
node.data: false

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["x.compute-1.internal",
"y.ec2.internal", "z.ec2.internal", "a.compute-1.internal",
"b.compute-1.internal",
"c.ec2.internal" ]

#internal ip of this router
#m.ec2.internal
network.bind_host: kk.kkk.ll.llll
http.port: 9500
#field cache configuration
index.cache.field.max_size: 20000
index.cache.field.expire: 10m

This is the traceback:

[2012-12-13 15:16:05,196][DEBUG][action.search.type ] [Router QA]
[1107156] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Master01
QA][inet[/XXXXXXXX:9300]][search/phase/fetch/id]
Caused by: org.elasticsearch.search.SearchContextMissingException: No
search context found for id [1107156]
at

org.elasticsearch.search.SearchService.findContext(SearchService.java:459)

    at

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:432)

    at

org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)

    at

org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

    at

org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)

    at

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

--

--
Met vriendelijke groet,

Martijn van Groningen

--

4

Your nodes in your cluster aren't under heavy load or scarce resource
(e.g. low available memory)?

I see that you're running with 0.20.0.RC1. Recently 0.20.1 has been
released. Although I don't have prove that a bug related to this has
been fixed, it might be a good idea to upgrade anyway.

Martijn

On 13 December 2012 17:29, James Reynolds eire1130@gmail.com wrote:

Thank you for the response.

I do not see any "Failed to send release search context" in the logs on any
of the boxes.

Really, the only error I see is the ones I copied below.

The parent/child queries return reasonably fast once they are "warmed", but
if it's been a few minutes since the last query, it can take a few seconds
to return.

On Thursday, December 13, 2012 11:11:18 AM UTC-5, Martijn v Groningen wrote:

Do these parent/child queries take a long time to complete?
Also do see a lot of Failed to send release search context warnings
in your log files?

Martijn

On 13 December 2012 16:42, James Reynolds eire...@gmail.com wrote:

Hello all:

We've been having periodic trouble with the below exception, and with
little
luck trying to track it down.

Generally speaking, this seems to happen on parent / child queries.
Initially, we'll get this exception (and many more like it) and then
search
will return the results for a time (maybe like a half hour or so) and
then
we'll get the errors again.

The incantation for elasticsearch looks like this:

/usr/java/default/bin/java -Xms12g -Xmx12g -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.path.home=/opt/elasticsearch -cp

:/opt/elasticsearch/lib/elasticsearch-0.20.0.RC1.jar:/opt/elasticsearch/lib/:/opt/elasticsearch/lib/sigar/
org.elasticsearch.bootstrap.Elasticsearch

the yml looks like this:

Unicast Discovery (disable multicast)

#discovery.zen.ping.multicast.enabled: false
#discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Name your cluster here to whatever.

cluster.name: XXXXX-search-qa
node.name: "Router QA"
node.master: true
node.data: false

discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["x.compute-1.internal",
"y.ec2.internal", "z.ec2.internal", "a.compute-1.internal",
"b.compute-1.internal",
"c.ec2.internal" ]

#internal ip of this router
#m.ec2.internal
network.bind_host: kk.kkk.ll.llll
http.port: 9500
#field cache configuration
index.cache.field.max_size: 20000
index.cache.field.expire: 10m

This is the traceback:

[2012-12-13 15:16:05,196][DEBUG][action.search.type ] [Router QA]
[1107156] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Master01
QA][inet[/XXXXXXXX:9300]][search/phase/fetch/id]
Caused by: org.elasticsearch.search.SearchContextMissingException: No
search context found for id [1107156]
at

org.elasticsearch.search.SearchService.findContext(SearchService.java:459)
at

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:432)
at

org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)
at

org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)
at

org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
at

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

--

--
Met vriendelijke groet,

Martijn van Groningen

--
Met vriendelijke groet,

Martijn van Groningen

--

5

Hi,

I got same problem. I have three EC2 instance nodes (8GB) with 5 shareds and 1 replica (es 0.20.2). And my queries need too long to respond. I need to wait from 10 to 240 sec. Also I have 4 000 000 records and also i have constantly adding new documents.

Thank you,
Milan