I am noticing very slow (timing out) Get API requests on my 0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id} requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
I am noticing very slow (timing out) Get API requests on my 0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id} requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
Do you have a load balancer infront of your client (or a load balancing
client)? When you say you run it with preference set to _local for 3 times,
and the 3rd one times out, do you always hit the same node? Are you running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my 0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id} requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
I did the get tests directly on each nodes using curl on localhost, so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com wrote:
Do you have a load balancer infront of your client (or a load balancing
client)? When you say you run it with preference set to _local for 3 times,
and the 3rd one times out, do you always hit the same node? Are you running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my 0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id} requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out (which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost, so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com wrote:
Do you have a load balancer infront of your client (or a load balancing
client)? When you say you run it with preference set to _local for 3
times,
and the 3rd one times out, do you always hit the same node? Are you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my 0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id} requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
Unfortunately, I don't have the exception. When I actually did the
tests on each node, I didn't wait for the exception to be thrown and I
interrupted curl after waiting for a while. I also looked in the logs
and could not find anything.
Now I cannot reproduce the problem after rebooting 3 nodes this
weekend (ec2 maintenance events).
Colin
On Sun, Dec 4, 2011 at 2:06 PM, Shay Banon kimchy@gmail.com wrote:
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out (which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost, so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com wrote:
Do you have a load balancer infront of your client (or a load balancing
client)? When you say you run it with preference set to _local for 3
times,
and the 3rd one times out, do you always hit the same node? Are you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my 0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id} requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
Unfortunately, I don't have the exception. When I actually did the
tests on each node, I didn't wait for the exception to be thrown and I
interrupted curl after waiting for a while. I also looked in the logs
and could not find anything.
Now I cannot reproduce the problem after rebooting 3 nodes this
weekend (ec2 maintenance events).
Colin
On Sun, Dec 4, 2011 at 2:06 PM, Shay Banon kimchy@gmail.com wrote:
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out (which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost, so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com wrote:
Do you have a load balancer infront of your client (or a load
balancing
client)? When you say you run it with preference set to _local for 3
times,
and the 3rd one times out, do you always hit the same node? Are you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my
0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id}
requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
Unfortunately, I don't have the exception. When I actually did the
tests on each node, I didn't wait for the exception to be thrown and I
interrupted curl after waiting for a while. I also looked in the logs
and could not find anything.
Now I cannot reproduce the problem after rebooting 3 nodes this
weekend (ec2 maintenance events).
Colin
On Sun, Dec 4, 2011 at 2:06 PM, Shay Banon kimchy@gmail.com wrote:
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out (which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost, so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com wrote:
Do you have a load balancer infront of your client (or a load
balancing
client)? When you say you run it with preference set to _local for 3
times,
and the 3rd one times out, do you always hit the same node? Are you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my
0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id}
requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the
cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
The problem has reappeared with some documents and this time, using
?preference=_primary does not solve it, in fact it consistently fails
using _primary.
So, the lastest behaviour is: systematic 2 fails and 1 success not
using ?preference= or using ?preference=_local, and always fail with
?preference=_primary.
When I say fail, it actually just hangs, and after 10 minutes, my curl
request hasn't generated any log/exception on the local node or on the
master node.
Unfortunately, I don't have the exception. When I actually did the
tests on each node, I didn't wait for the exception to be thrown and I
interrupted curl after waiting for a while. I also looked in the logs
and could not find anything.
Now I cannot reproduce the problem after rebooting 3 nodes this
weekend (ec2 maintenance events).
Colin
On Sun, Dec 4, 2011 at 2:06 PM, Shay Banon kimchy@gmail.com wrote:
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out (which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost, so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com wrote:
Do you have a load balancer infront of your client (or a load
balancing
client)? When you say you run it with preference set to _local for 3
times,
and the 3rd one times out, do you always hit the same node? Are you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my
0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id}
requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the
cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
The problem has reappeared with some documents and this time, using
?preference=_primary does not solve it, in fact it consistently fails
using _primary.
So, the lastest behaviour is: systematic 2 fails and 1 success not
using ?preference= or using ?preference=_local, and always fail with
?preference=_primary.
When I say fail, it actually just hangs, and after 10 minutes, my curl
request hasn't generated any log/exception on the local node or on the
master node.
Unfortunately, I don't have the exception. When I actually did the
tests on each node, I didn't wait for the exception to be thrown and I
interrupted curl after waiting for a while. I also looked in the logs
and could not find anything.
Now I cannot reproduce the problem after rebooting 3 nodes this
weekend (ec2 maintenance events).
Colin
On Sun, Dec 4, 2011 at 2:06 PM, Shay Banon kimchy@gmail.com wrote:
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out (which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost,
so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com
wrote:
Do you have a load balancer infront of your client (or a load
balancing
client)? When you say you run it with preference set to _local
for 3
times,
and the 3rd one times out, do you always hit the same node? Are
you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my
0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id}
requests.
Everything else is flowing, bulk inserts, search requests, etc.
Absolutely no log (at DEBUG level), no failures on the the
cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
The problem has reappeared with some documents and this time, using
?preference=_primary does not solve it, in fact it consistently fails
using _primary.
So, the lastest behaviour is: systematic 2 fails and 1 success not
using ?preference= or using ?preference=_local, and always fail with
?preference=_primary.
When I say fail, it actually just hangs, and after 10 minutes, my curl
request hasn't generated any log/exception on the local node or on the
master node.
Unfortunately, I don't have the exception. When I actually did the
tests on each node, I didn't wait for the exception to be thrown and I
interrupted curl after waiting for a while. I also looked in the logs
and could not find anything.
Now I cannot reproduce the problem after rebooting 3 nodes this
weekend (ec2 maintenance events).
Colin
On Sun, Dec 4, 2011 at 2:06 PM, Shay Banon kimchy@gmail.com wrote:
And when you executed it on a specific node, you ran it 3 times with
preference set to _local, and the third time it would time out
(which
exception do you get)? Does it happen on all nodes (this behavior).
I did the get tests directly on each nodes using curl on localhost,
so
yes, when doing the 3 get sequence, it was always hitting the same
node (localhost).
Colin
On Sun, Dec 4, 2011 at 9:15 AM, Shay Banon kimchy@gmail.com
wrote:
Do you have a load balancer infront of your client (or a load
balancing
client)? When you say you run it with preference set to _local
for 3
times,
and the 3rd one times out, do you always hit the same node? Are
you
running
it with curl directly against the nodes?
I am noticing very slow (timing out) Get API requests on my
0.18.5
cluster. These are simple http://{host}/{index}/{type}/{id}
requests.
Everything else is flowing, bulk inserts, search requests,
etc.
Absolutely no log (at DEBUG level), no failures on the the
cluster.
All nodes load/memory are ok.
In fact, if I do a search using
http://{host}/{index}/{type}/_search?q={id} it is very fast.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.