I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time less
than 100ms, large ec2 instance) requests can a singleton Client object
handle?
2- How heavy Client initialization is (based on answer to the question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
I don't have ES experience in production, but I made some debugging during
development.
As far as I understood each client requires initializing local node,
registering node in cluster, build pool of connection threads (to speak with
other nodes). If it's node with data - initialize shards, rebalance cluster.
So a best option would be to have single node per JVM. If you see that you
lack of connections, increase connection pool of client.
I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time less
than 100ms, large ec2 instance) requests can a singleton Client object
handle?
2- How heavy Client initialization is (based on answer to the question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
Hi Alex,
Thanks for your quick response, I agree with you that it is better to
have one "node" per JVM. However, I was wondering if I could obtain
several "Client" objects from one "node" in case I needed to (I am not
sure if it is even needed, please read on)! Correct me if I am wrong,
so client works just like a facade handing over requests to a pool of
threads managed by the node class?
If it is the case then I need to have one singleton instance of "node"
and just obtain one "Client" from it, and if the load picks up it
means that I have to add a new box, right?
To give you a better picture, I am creating a high level API on top of
ES so I do not want to introduce any performance bottlenecks, and this
is why I try to understand how node-client bundle work.
Thanks,
Ali
I don't have ES experience in production, but I made some debugging during
development.
As far as I understood each client requires initializing local node,
registering node in cluster, build pool of connection threads (to speak with
other nodes). If it's node with data - initialize shards, rebalance cluster.
So a best option would be to have single node per JVM. If you see that you
lack of connections, increase connection pool of client.
I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time less
than 100ms, large ec2 instance) requests can a singleton Client object
handle?
2- How heavy Client initialization is (based on answer to the question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
Hi Alex,
Thanks for your quick response, I agree with you that it is better to
have one "node" per JVM. However, I was wondering if I could obtain
several "Client" objects from one "node" in case I needed to (I am not
sure if it is even needed, please read on)! Correct me if I am wrong,
so client works just like a facade handing over requests to a pool of
threads managed by the node class?
If it is the case then I need to have one singleton instance of "node"
and just obtain one "Client" from it, and if the load picks up it
means that I have to add a new box, right?
To give you a better picture, I am creating a high level API on top of
ES so I do not want to introduce any performance bottlenecks, and this
is why I try to understand how node-client bundle work.
Thanks,
Ali
I don't have ES experience in production, but I made some debugging
during
development.
As far as I understood each client requires initializing local node,
registering node in cluster, build pool of connection threads (to speak
with
other nodes). If it's node with data - initialize shards, rebalance
cluster.
So a best option would be to have single node per JVM. If you see that
you
lack of connections, increase connection pool of client.
I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time less
than 100ms, large ec2 instance) requests can a singleton Client object
handle?
2- How heavy Client initialization is (based on answer to the question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
Hi Lukas,
You actually demystified it for me, ... One Node and one Client per
JVM, that is what I gotta do, and I have to create my DAO layer on top
of it ... I guess I got to look into Compass source code too.
Thanks,
Ali
If you are talking about Node client then I think each Node maintains
singleton instance of client, so each call to .client() method on Node
returns you the same instance of client. (see NodeClientModule codehttps://github.com/elasticsearch/elasticsearch/blob/master/modules/el...
).
Hi Alex,
Thanks for your quick response, I agree with you that it is better to
have one "node" per JVM. However, I was wondering if I could obtain
several "Client" objects from one "node" in case I needed to (I am not
sure if it is even needed, please read on)! Correct me if I am wrong,
so client works just like a facade handing over requests to a pool of
threads managed by the node class?
If it is the case then I need to have one singleton instance of "node"
and just obtain one "Client" from it, and if the load picks up it
means that I have to add a new box, right?
To give you a better picture, I am creating a high level API on top of
ES so I do not want to introduce any performance bottlenecks, and this
is why I try to understand how node-client bundle work.
Thanks,
Ali
I don't have ES experience in production, but I made some debugging
during
development.
As far as I understood each client requires initializing local node,
registering node in cluster, build pool of connection threads (to speak
with
other nodes). If it's node with data - initialize shards, rebalance
cluster.
So a best option would be to have single node per JVM. If you see that
you
lack of connections, increase connection pool of client.
I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time less
than 100ms, large ec2 instance) requests can a singleton Client object
handle?
2- How heavy Client initialization is (based on answer to the question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
Hi Lukas,
You actually demystified it for me, ... One Node and one Client per
JVM, that is what I gotta do, and I have to create my DAO layer on top
of it ... I guess I got to look into Compass source code too.
Thanks,
Ali
If you are talking about Node client then I think each Node maintains
singleton instance of client, so each call to .client() method on Node
returns you the same instance of client. (see NodeClientModule
codehttps://github.com/elasticsearch/elasticsearch/blob/master/modules/el.
..
).
Hi Alex,
Thanks for your quick response, I agree with you that it is better to
have one "node" per JVM. However, I was wondering if I could obtain
several "Client" objects from one "node" in case I needed to (I am not
sure if it is even needed, please read on)! Correct me if I am wrong,
so client works just like a facade handing over requests to a pool of
threads managed by the node class?
If it is the case then I need to have one singleton instance of "node"
and just obtain one "Client" from it, and if the load picks up it
means that I have to add a new box, right?
To give you a better picture, I am creating a high level API on top of
ES so I do not want to introduce any performance bottlenecks, and this
is why I try to understand how node-client bundle work.
Thanks,
Ali
I don't have ES experience in production, but I made some debugging
during
development.
As far as I understood each client requires initializing local node,
registering node in cluster, build pool of connection threads (to
speak
with
other nodes). If it's node with data - initialize shards, rebalance
cluster.
So a best option would be to have single node per JVM. If you see
that
you
lack of connections, increase connection pool of client.
I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time
less
than 100ms, large ec2 instance) requests can a singleton Client
object
handle?
2- How heavy Client initialization is (based on answer to the
question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
Hi Lukas,
You actually demystified it for me, ... One Node and one Client per
JVM, that is what I gotta do, and I have to create my DAO layer on top
of it ... I guess I got to look into Compass source code too.
Thanks,
Ali
If you are talking about Node client then I think each Node maintains
singleton instance of client, so each call to .client() method on Node
returns you the same instance of client. (see NodeClientModule
codehttps://github.com/elasticsearch/elasticsearch/blob/master/modules/el.
..
).
Hi Alex,
Thanks for your quick response, I agree with you that it is better to
have one "node" per JVM. However, I was wondering if I could obtain
several "Client" objects from one "node" in case I needed to (I am not
sure if it is even needed, please read on)! Correct me if I am wrong,
so client works just like a facade handing over requests to a pool of
threads managed by the node class?
If it is the case then I need to have one singleton instance of "node"
and just obtain one "Client" from it, and if the load picks up it
means that I have to add a new box, right?
To give you a better picture, I am creating a high level API on top of
ES so I do not want to introduce any performance bottlenecks, and this
is why I try to understand how node-client bundle work.
Thanks,
Ali
I don't have ES experience in production, but I made some debugging
during
development.
As far as I understood each client requires initializing local node,
registering node in cluster, build pool of connection threads (to
speak
with
other nodes). If it's node with data - initialize shards, rebalance
cluster.
So a best option would be to have single node per JVM. If you see
that
you
lack of connections, increase connection pool of client.
I am designing a new service layer using ES Java API. I have got
couple of questions (I have not looked into ES source code yet)?
Based on your experience:
1- How many concurrent (per second, 2 million docs, response time
less
than 100ms, large ec2 instance) requests can a singleton Client
object
handle?
2- How heavy Client initialization is (based on answer to the
question
above)?
a- Client initialization is heavy, then I will need to create a
pool of the clients.
b- Client initialization is super fast, then I will just create
them as I the need comes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.