Dedicated bulk-loading socket interface?

Elastic Stack Elasticsearch
1

I'm evaluating ES for use in a very high volume messaging system processing
hundreds of millions of messages per day. And they want search on top of
that, of course :slight_smile:

I see ES has a bulk-loading interface, however you basically are batching
requests and still running over HTTP, etc. Does ES have a way to send
these same commands into a dedicated socket or anything similar? Basically
I'd have some number of sockets open to ES and feed it data I want to index
as I'm processing message streams. If all else fails I'll probably write
my own adapter, it doesn't seem like it would be terribly complicated to
do, I just wanted to see if anyone else had had this issue before.

thanks,

-nate

"The sky calls to us. If we do not destroy ourselves, we will one day
venture to the stars." -- Carl Sagan

2

Hi Nate,

You have two other options for sending data to ES, using the Java API
and Thrift. The Java API/client does not communicate via REST/HTTP,
but via a binary protocol. I am not aware of any other client beside
the Java client that uses the binary protocol.

There is also an additional module that exposes the REST interface in Thrift:
http://www.elasticsearch.org/guide/reference/modules/thrift.html

I have never used it, so I cannot comment on its usage.

--
Ivan

On Tue, Feb 21, 2012 at 10:41 AM, Nate Sammons nate@protomatter.com wrote:

I'm evaluating ES for use in a very high volume messaging system processing
hundreds of millions of messages per day. And they want search on top of
that, of course :slight_smile:

I see ES has a bulk-loading interface, however you basically are batching
requests and still running over HTTP, etc. Does ES have a way to send
these same commands into a dedicated socket or anything similar? Basically
I'd have some number of sockets open to ES and feed it data I want to index
as I'm processing message streams. If all else fails I'll probably write my
own adapter, it doesn't seem like it would be terribly complicated to do, I
just wanted to see if anyone else had had this issue before.

thanks,

-nate

"The sky calls to us. If we do not destroy ourselves, we will one day
venture to the stars." -- Carl Sagan

3

That's awesome, actually. I didn't even see the Java API, that's exactly
what I need (all systems are java here anyway, so its not an issue).

thanks!

-nate

"The sky calls to us. If we do not destroy ourselves, we will one day
venture to the stars." -- Carl Sagan

On Tue, Feb 21, 2012 at 12:02 PM, Ivan Brusic ivan@brusic.com wrote:

Hi Nate,

You have two other options for sending data to ES, using the Java API
and Thrift. The Java API/client does not communicate via REST/HTTP,
but via a binary protocol. I am not aware of any other client beside
the Java client that uses the binary protocol.

There is also an additional module that exposes the REST interface in
Thrift:
http://www.elasticsearch.org/guide/reference/modules/thrift.html

I have never used it, so I cannot comment on its usage.

--
Ivan

On Tue, Feb 21, 2012 at 10:41 AM, Nate Sammons nate@protomatter.com
wrote:

I'm evaluating ES for use in a very high volume messaging system
processing
hundreds of millions of messages per day. And they want search on top of
that, of course :slight_smile:

I see ES has a bulk-loading interface, however you basically are batching
requests and still running over HTTP, etc. Does ES have a way to send
these same commands into a dedicated socket or anything similar?
Basically
I'd have some number of sockets open to ES and feed it data I want to
index
as I'm processing message streams. If all else fails I'll probably
write my
own adapter, it doesn't seem like it would be terribly complicated to
do, I
just wanted to see if anyone else had had this issue before.

thanks,

-nate

"The sky calls to us. If we do not destroy ourselves, we will one day
venture to the stars." -- Carl Sagan

4