ZeroMQ transport layer plugin

Elastic Stack Elasticsearch
1

Hi all,

I don't know if you are interested in this, but I wrote a transport module
that exposes the REST interface of ES over ZeroMQ. This way, it is possible
to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:

CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version":1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:

<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The SimpleClient
class in test package shows how to create a simple ØMQ client and send
messages.

The implementation uses a router-dealer pattern (
http://www.zeromq.org/sandbox:dealer), where multiple XREP sockets are
listening to incoming messages (each in a dedicated thread).

The source code is located at
https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/zeromq ,
I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming messages on
many ES nodes) but your feedback will be greatly appreciated.

Thanks,

-- Tanguy

2

Hiya Tanguy

I don't know if you are interested in this, but I wrote a transport
module that exposes the REST interface of ES over ZeroMQ. This way, it
is possible to send REST-like messages with ØMQ clients and get the
replies back.

I'd be very interested in seeing this work.

I tried to build ES from your branch with gradlew, but it complained of
missing the jzmq dependency. Perhaps there is some missing config I
need to get the build working correctly?

Also, I've never actually used zeromq, just read the website a bit.

Could you provide some client code demonstrating how it should be used?

Your config specifies port 9700 and port 9800 - which one should the
client talk to?

thanks

clint

For example, a ØMQ client can send the following message:
POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:
CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version":1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:
<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The
SimpleClient class in test package shows how to create a simple ØMQ
client and send messages.

The implementation uses a router-dealer pattern
(The ROUTER and DEALER - zeromq), where multiple XREP sockets
are listening to incoming messages (each in a dedicated thread).

The source code is located
at https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/zeromq , I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming
messages on many ES nodes) but your feedback will be greatly
appreciated.

Thanks,

-- Tanguy

3

Hi Clitnton,

to build this plugin, you need to:

  1. build ZeroMQ distribution v2.1
  2. build ZeroMQ Java binding library (jzmq)
  3. configure or add the library to your local gradle repository

For step 1) and 2), I encourage you to read the procedure on the
ZeroMQ web site, which is very well documented. From my experience, it
will be easier on a Linux box.

This thread from the ZeroMQ mailing list can also help you:
http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/013762.html

You can also build the jzmq library with Maven and install it in your
local repo with mvn install.

For step 3), I declared a maven local repository in the build.gradle
file. This allow you to use your local maven rep to retrieve the
jzmq-2.1.0-SNAPSHOT.jar file.

Then, you can test the plugin with the SimpleClient file provided.
Here is an example of the args you can provide to the SimpleClass:

tcp://localhost:9700 PUT /twitter/tweet/2 "{... json content...}"

External XREQ sockets must connect to plugin with the tcp://localhost:9700
address (can be configured in ES config file). The other address is
used internally by the plugin. I updated the config file on github to
avoid confusion.

See u,

-- Tanguy

On 29 oct, 15:32, Clinton Gormley cl...@traveljury.com wrote:

Hiya Tanguy

I don't know if you are interested in this, but I wrote a transport
module that exposes the REST interface of ES over ZeroMQ. This way, it
is possible to send REST-like messages with ØMQ clients and get the
replies back.

I'd be very interested in seeing this work.

I tried to build ES from your branch with gradlew, but it complained of
missing the jzmq dependency. Perhaps there is some missing config I
need to get the build working correctly?

Also, I've never actually used zeromq, just read the website a bit.

Could you provide some client code demonstrating how it should be used?

Your config specifies port 9700 and port 9800 - which one should the
client talk to?

thanks

clint

For example, a ØMQ client can send the following message:
POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:
CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version": 1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:
<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The
SimpleClient class in test package shows how to create a simple ØMQ
client and send messages.

The implementation uses a router-dealer pattern
(The ROUTER and DEALER - zeromq), where multiple XREP sockets
are listening to incoming messages (each in a dedicated thread).

The source code is located
athttps://github.com/tlrx/elasticsearch/tree/master/plugins/transport/z..., I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming
messages on many ES nodes) but your feedback will be greatly
appreciated.

Thanks,

-- Tanguy

4

Hey,

Thanks for this!. I was dabbling with zeromq a bit, but never had proper
time to actually use it. Is there a chance that you can move it into a
completely separate plugin, not a fork of elasticsearch?

On Fri, Oct 28, 2011 at 4:49 PM, tlrx tlrx.dev@gmail.com wrote:

Hi all,

I don't know if you are interested in this, but I wrote a transport module
that exposes the REST interface of ES over ZeroMQ. This way, it is possible
to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:

CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version":1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:

<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The SimpleClient
class in test package shows how to create a simple ØMQ client and send
messages.

The implementation uses a router-dealer pattern (
The ROUTER and DEALER - zeromq), where multiple XREP sockets are
listening to incoming messages (each in a dedicated thread).

The source code is located at
https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/zeromq ,
I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming messages
on many ES nodes) but your feedback will be greatly appreciated.

Thanks,

-- Tanguy

5

Hi Shay,

It was a pleasure to code it :slight_smile: I forked ES since it was easier to
developp and debug the code, but I can move it in a separate plugin.

Do you have an example on github?

I will do this next week, I think. And write some doc.

-- Tanguy

On 31 oct, 18:11, Shay Banon kim...@gmail.com wrote:

Hey,

Thanks for this!. I was dabbling with zeromq a bit, but never had proper
time to actually use it. Is there a chance that you can move it into a
completely separate plugin, not a fork of elasticsearch?

On Fri, Oct 28, 2011 at 4:49 PM, tlrx tlrx....@gmail.com wrote:

Hi all,

I don't know if you are interested in this, but I wrote a transport module
that exposes the REST interface of ES over ZeroMQ. This way, it is possible
to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:

CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version": 1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:

<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The SimpleClient
class in test package shows how to create a simple ØMQ client and send
messages.

The implementation uses a router-dealer pattern (
The ROUTER and DEALER - zeromq), where multiple XREP sockets are
listening to incoming messages (each in a dedicated thread).

The source code is located at
https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/z...,
I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming messages
on many ES nodes) but your feedback will be greatly appreciated.

Thanks,

-- Tanguy

6

The RSS river is one example of a standalone plugin:
GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED).

On Tue, Nov 1, 2011 at 5:53 PM, tlrx tanguy.leroux@gmail.com wrote:

Hi Shay,

It was a pleasure to code it :slight_smile: I forked ES since it was easier to
developp and debug the code, but I can move it in a separate plugin.

Do you have an example on github?

I will do this next week, I think. And write some doc.

-- Tanguy

On 31 oct, 18:11, Shay Banon kim...@gmail.com wrote:

Hey,

Thanks for this!. I was dabbling with zeromq a bit, but never had
proper
time to actually use it. Is there a chance that you can move it into a
completely separate plugin, not a fork of elasticsearch?

On Fri, Oct 28, 2011 at 4:49 PM, tlrx tlrx....@gmail.com wrote:

Hi all,

I don't know if you are interested in this, but I wrote a transport
module
that exposes the REST interface of ES over ZeroMQ. This way, it is
possible
to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:

CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version":
1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:

<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The
SimpleClient
class in test package shows how to create a simple ØMQ client and send
messages.

The implementation uses a router-dealer pattern (
The ROUTER and DEALER - zeromq), where multiple XREP sockets are
listening to incoming messages (each in a dedicated thread).

The source code is located at
https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/z..
.,
I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming
messages
on many ES nodes) but your feedback will be greatly appreciated.

Thanks,

-- Tanguy

7

Hi Shay,

I moved the code to a separate plugin: GitHub - tlrx/transport-zeromq: ØMQ transport layer plugin for Elasticsearch

Now it can be installed with Elasticsearch plugin script. I also write
some documentation, inspired by the RSS river.

I still working on this plugin, but feedback will be appreciated :o)

-- Tanguy

On 1 nov, 19:09, Shay Banon kim...@gmail.com wrote:

The RSS river is one example of a standalone plugin:GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED).

On Tue, Nov 1, 2011 at 5:53 PM, tlrx tanguy.ler...@gmail.com wrote:

Hi Shay,

It was a pleasure to code it :slight_smile: I forked ES since it was easier to
developp and debug the code, but I can move it in a separate plugin.

Do you have an example on github?

I will do this next week, I think. And write some doc.

-- Tanguy

On 31 oct, 18:11, Shay Banon kim...@gmail.com wrote:

Hey,

Thanks for this!. I was dabbling with zeromq a bit, but never had
proper
time to actually use it. Is there a chance that you can move it into a
completely separate plugin, not a fork of elasticsearch?

On Fri, Oct 28, 2011 at 4:49 PM, tlrx tlrx....@gmail.com wrote:

Hi all,

I don't know if you are interested in this, but I wrote a transport
module
that exposes the REST interface of ES over ZeroMQ. This way, it is
possible
to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:

CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version":
1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:

<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The
SimpleClient
class in test package shows how to create a simple ØMQ client and send
messages.

The implementation uses a router-dealer pattern (
The ROUTER and DEALER - zeromq), where multiple XREP sockets are
listening to incoming messages (each in a dedicated thread).

The source code is located at
https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/z..
.,
I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming
messages
on many ES nodes) but your feedback will be greatly appreciated.

Thanks,

-- Tanguy

8

I moved the code to a separate plugin: GitHub - tlrx/transport-zeromq: ØMQ transport layer plugin for Elasticsearch

++ for the improved docs :slight_smile:

clint

Now it can be installed with Elasticsearch plugin script. I also write
some documentation, inspired by the RSS river.

I still working on this plugin, but feedback will be appreciated :o)

-- Tanguy

On 1 nov, 19:09, Shay Banon kim...@gmail.com wrote:

The RSS river is one example of a standalone plugin:GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED).

On Tue, Nov 1, 2011 at 5:53 PM, tlrx tanguy.ler...@gmail.com wrote:

Hi Shay,

It was a pleasure to code it :slight_smile: I forked ES since it was easier to
developp and debug the code, but I can move it in a separate plugin.

Do you have an example on github?

I will do this next week, I think. And write some doc.

-- Tanguy

On 31 oct, 18:11, Shay Banon kim...@gmail.com wrote:

Hey,

Thanks for this!. I was dabbling with zeromq a bit, but never had
proper
time to actually use it. Is there a chance that you can move it into a
completely separate plugin, not a fork of elasticsearch?

On Fri, Oct 28, 2011 at 4:49 PM, tlrx tlrx....@gmail.com wrote:

Hi all,

I don't know if you are interested in this, but I wrote a transport
module
that exposes the REST interface of ES over ZeroMQ. This way, it is
possible
to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user": "kimchy","post_date":
"2009-11-15T14:12:12","message": "You know, for Search"}

It will receive the following response:

CREATED|{"ok":true,"_index":"twitter","_type":"tweet","_id":"2","_version":
1}

The transport layer converts ØMQ messages in a given format into REST
request objects that can be handled by ES.

The expected format for incoming messages is:

<Method PUT,DELETE, POST...>|<URI, including parameters>|

But any other message format can be easely implemented. The
SimpleClient
class in test package shows how to create a simple ØMQ client and send
messages.

The implementation uses a router-dealer pattern (
The ROUTER and DEALER - zeromq), where multiple XREP sockets are
listening to incoming messages (each in a dedicated thread).

The source code is located at
https://github.com/tlrx/elasticsearch/tree/master/plugins/transport/z..
.,
I used the Thrift module as a good example.

I still working on this (tests, benchmarks, distribute incoming
messages
on many ES nodes) but your feedback will be greatly appreciated.

Thanks,

-- Tanguy

9

Hi Tanguy

On Sun, Nov 6, 2011 at 2:58 PM, Clinton Gormley clint@traveljury.comwrote:

I moved the code to a separate plugin:
GitHub - tlrx/transport-zeromq: ØMQ transport layer plugin for Elasticsearch

OK - i managed to get a Perl script talking to Elasticsearch via ZeroMQ

I've played with it very briefly, and opened 2 issues for bugs:

Looking forward to seeing this plugin become solid!

clint

10

Thanks for your feedback, Clint. I'll work on it as soon as
possible :slight_smile:

On Nov 6, 6:40 pm, Clinton Gormley cl...@traveljury.com wrote:

Hi Tanguy

On Sun, Nov 6, 2011 at 2:58 PM, Clinton Gormley cl...@traveljury.comwrote:

I moved the code to a separate plugin:
GitHub - tlrx/transport-zeromq: ØMQ transport layer plugin for Elasticsearch

OK - i managed to get a Perl script talking to Elasticsearch via ZeroMQ

I've played with it very briefly, and opened 2 issues for bugs:Issues · tlrx/transport-zeromq · GitHub

Looking forward to seeing this plugin become solid!

clint