I'm investigating some performance issues with our application, and one of
the hot spots seems to be the transport of data from Elasticsearch into the
application. We've been using the Java Transport Client for this so far.
Since our data will easily fit a single machine and are rarely updated,
we're effectively running a "replication" architecture: each machine holds
one application instance and one instance of ES; the application will only
talk to the ES node on the same machine. Our application fetches quite a
few results from ES on each call, so there's a lot of data being
transferred within the same machine.
My optimisation idea now was to embed ES into the application to
short-circuit the TCP transport to the application, basically doing this:
Node node = nodeBuilder().node(); // settings loaded from
elasticsearch.yml
client = node.client();
While this functionally works as expected (that is, exactly the same as
before with each ES running in a separate VM), I'm still seeing a lot of
Netty activity in the profiler - which goes against my original assumption
that this client would talk to the node via local "transport" (i.e. passing
in-memory references), not TCP.
Looking through the API and org.elasticsearch.transport.TransportModule, it
seems that you can either create a local node and access that via local
transport, or create a non-local node and use TCP transport. Is there any
way to get a local transport client for a "regular" (but embedded) node?
You may want to use Node node = nodeBuilder().local(true).node() in your
app, this disables external communication via a network interface.
Jörg
Am 09.04.13 16:13, schrieb Klaus Brunner:
I'm investigating some performance issues with our application, and
one of the hot spots seems to be the transport of data from
Elasticsearch into the application. We've been using the Java
Transport Client for this so far.
Since our data will easily fit a single machine and are rarely
updated, we're effectively running a "replication" architecture: each
machine holds one application instance and one instance of ES; the
application will only talk to the ES node on the same machine. Our
application fetches quite a few results from ES on each call, so
there's a lot of data being transferred within the same machine.
My optimisation idea now was to embed ES into the application to
short-circuit the TCP transport to the application, basically doing this:
Node node = nodeBuilder().node(); // settings loaded from
elasticsearch.yml
client = node.client();
While this functionally works as expected (that is, exactly the same
as before with each ES running in a separate VM), I'm still seeing a
lot of Netty activity in the profiler - which goes against my original
assumption that this client would talk to the node via local
"transport" (i.e. passing in-memory references), not TCP.
Looking through the API and
org.elasticsearch.transport.TransportModule, it seems that you can
either create a local node and access that via local transport, or
create a non-local node and use TCP transport. Is there any way to get
a local transport client for a "regular" (but embedded) node?
On Tuesday, 9 April 2013 16:52:32 UTC+2, Jörg Prante wrote:
You may want to use Node node = nodeBuilder().local(true).node() in your
app, this disables external communication via a network interface.
But that would disable cluster communication entirely, wouldn't it? Our ES
instances are still supposed to be clustered across several machines, so
that data updates are replicated.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.