[ANN] ElasticSearch Mock Solr Plugin (use Solr tools/clients with ElasticSearch)


(Matt Weber) #1

I just released a plugin that Mocks the Solr interface in
ElasticSearch. With this you can use tools and clients that are meant
to talk to Solr with ElasticSearch. Some examples are Nutch, Apache
ManifoldCF, SolrJ apps, etc. Currently, indexing and deleting of
documents is supported 100% for XML (/update request handler) and
JavaBin (/update/javabin request handler). Basic support for the Solr
search handler (/select) is also included for the Solr q, start, rows,
and fl parameters. The q parameter supports 100% of the lucene query
syntax. Both XML and JavaBin response formats are supported.

To use the plugin:

  1. Install

$ES_HOME/bin/plugin install mattweber/elasticsearch-mocksolrplugin

  1. Update your client code to point at ElasticSearch and the /_solr
    REST endpoint.

http://localhost:9200/index/type/_solr

Specifying the index and type is optional and will default to "solr"
for index, and "docs" for type.

  1. Use your Solr client as normal.

I have tested the plugin with Nutch and various SolrJ test code.
Using Nutch with ElasticSearch is the reason I wrote this plugin.
Instead of extending Nutch to support ElasticSearch as an endpoint, I
figured it would be much better to support any tool trying to talk to
Solr. This plugin should greatly reduce the effort in testing and/or
replacing Solr with ElasticSearch. It also opens the doors for using
tools that were previously not available to ElasticSearch users.

Please let me know if you have any questions, comments, or suggestions.

Source available on GitHub:

Thanks,
Matt Weber


(Karussell) #2

Thanks for this Matt, I really like the approach of 'integrating solr'
which opens up a lot of opportunities!

Peter.


(David Pilato) #3

WTF is the first that came into my mouth !

Very excellent work !
I'm very curious to have feedback from SOLR users if it's this "install & play"
plugin helps to migrate from SOLR to ES.

I'lltakemy hat off to you, Matt!

Le 13 décembre 2011 à 00:00, Matt Weber matt@mattweber.org a écrit :

I just released a plugin that Mocks the Solr interface in
ElasticSearch. With this you can use tools and clients that are meant
to talk to Solr with ElasticSearch. Some examples are Nutch, Apache
ManifoldCF, SolrJ apps, etc. Currently, indexing and deleting of
documents is supported 100% for XML (/update request handler) and
JavaBin (/update/javabin request handler). Basic support for the Solr
search handler (/select) is also included for the Solr q, start, rows,
and fl parameters. The q parameter supports 100% of the lucene query
syntax. Both XML and JavaBin response formats are supported.

To use the plugin:

  1. Install

$ES_HOME/bin/plugin install mattweber/elasticsearch-mocksolrplugin

  1. Update your client code to point at ElasticSearch and the /_solr
    REST endpoint.

http://localhost:9200/index/type/_solr

Specifying the index and type is optional and will default to "solr"
for index, and "docs" for type.

  1. Use your Solr client as normal.

I have tested the plugin with Nutch and various SolrJ test code.
Using Nutch with ElasticSearch is the reason I wrote this plugin.
Instead of extending Nutch to support ElasticSearch as an endpoint, I
figured it would be much better to support any tool trying to talk to
Solr. This plugin should greatly reduce the effort in testing and/or
replacing Solr with ElasticSearch. It also opens the doors for using
tools that were previously not available to ElasticSearch users.

Please let me know if you have any questions, comments, or suggestions.

Source available on GitHub:
https://github.com/mattweber/elasticsearch-mocksolrplugin

Thanks,
Matt Weber
--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet


(Shay Banon) #4

Looks great! :). One thing that I would suggest regarding the plugin is not
to use the same versioning as elasticsearch itself (cause then you need to
release a version of the plugin on each release of elasticsearch). You can
have a look at the mavenized plugins that are now on their own projects in
the elasticsearch repo.

On Tue, Dec 13, 2011 at 2:15 PM, david@pilato.fr david@pilato.fr wrote:

**

WTF is the first that came into my mouth !

Very excellent work !

I'm very curious to have feedback from SOLR users if it's this "install &
play" plugin helps to migrate from SOLR to ES.

I'll take my hat off to you, Matt!

Le 13 décembre 2011 à 00:00, Matt Weber matt@mattweber.org a écrit :

I just released a plugin that Mocks the Solr interface in
ElasticSearch. With this you can use tools and clients that are meant
to talk to Solr with ElasticSearch. Some examples are Nutch, Apache
ManifoldCF, SolrJ apps, etc. Currently, indexing and deleting of
documents is supported 100% for XML (/update request handler) and
JavaBin (/update/javabin request handler). Basic support for the Solr
search handler (/select) is also included for the Solr q, start, rows,
and fl parameters. The q parameter supports 100% of the lucene query
syntax. Both XML and JavaBin response formats are supported.

To use the plugin:

  1. Install

$ES_HOME/bin/plugin install mattweber/elasticsearch-mocksolrplugin

  1. Update your client code to point at ElasticSearch and the /_solr
    REST endpoint.

http://localhost:9200/index/type/_solr

Specifying the index and type is optional and will default to "solr"
for index, and "docs" for type.

  1. Use your Solr client as normal.

I have tested the plugin with Nutch and various SolrJ test code.
Using Nutch with ElasticSearch is the reason I wrote this plugin.
Instead of extending Nutch to support ElasticSearch as an endpoint, I
figured it would be much better to support any tool trying to talk to
Solr. This plugin should greatly reduce the effort in testing and/or
replacing Solr with ElasticSearch. It also opens the doors for using
tools that were previously not available to ElasticSearch users.

Please let me know if you have any questions, comments, or suggestions.

Source available on GitHub:
https://github.com/mattweber/elasticsearch-mocksolrplugin

Thanks,
Matt Weber

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet


(Piergiorgio Lucidi) #5

Hi Matt,

it seems a great contribution to the project, I hope to spend some
time on it during this week.
I would like to test this plugin with Apache ManifoldCF to understand
if it could be useful to implement a new and specific ElasticSearch
Output Connector or not.

Thank you.
Piergiorgio

On Dec 13, 12:00 am, Matt Weber m...@mattweber.org wrote:

I just released a plugin that Mocks the Solr interface in
ElasticSearch. With this you can use tools and clients that are meant
to talk to Solr with ElasticSearch. Some examples are Nutch, Apache
ManifoldCF, SolrJ apps, etc. Currently, indexing and deleting of
documents is supported 100% for XML (/update request handler) and
JavaBin (/update/javabin request handler). Basic support for the Solr
search handler (/select) is also included for the Solr q, start, rows,
and fl parameters. The q parameter supports 100% of the lucene query
syntax. Both XML and JavaBin response formats are supported.

To use the plugin:

  1. Install

$ES_HOME/bin/plugin install mattweber/elasticsearch-mocksolrplugin

  1. Update your client code to point at ElasticSearch and the /_solr
    REST endpoint.

http://localhost:9200/index/type/_solr

Specifying the index and type is optional and will default to "solr"
for index, and "docs" for type.

  1. Use your Solr client as normal.

I have tested the plugin with Nutch and various SolrJ test code.
Using Nutch with ElasticSearch is the reason I wrote this plugin.
Instead of extending Nutch to support ElasticSearch as an endpoint, I
figured it would be much better to support any tool trying to talk to
Solr. This plugin should greatly reduce the effort in testing and/or
replacing Solr with ElasticSearch. It also opens the doors for using
tools that were previously not available to ElasticSearch users.

Please let me know if you have any questions, comments, or suggestions.

Source available on GitHub:https://github.com/mattweber/elasticsearch-mocksolrplugin

Thanks,
Matt Weber


(Lukáš Vlček) #6

Hi Piergiorgio,

I would be interested in hearing what you learn, do you think you could share with us if that combo worked fine for you?

--
Regards,
Lukas

On Tuesday, December 13, 2011 at 9:41 PM, Piergiorgio Lucidi wrote:

Hi Matt,

it seems a great contribution to the project, I hope to spend some
time on it during this week.
I would like to test this plugin with Apache ManifoldCF to understand
if it could be useful to implement a new and specific ElasticSearch
Output Connector or not.

Thank you.
Piergiorgio

On Dec 13, 12:00 am, Matt Weber <m...@mattweber.org (http://mattweber.org)> wrote:

I just released a plugin that Mocks the Solr interface in
ElasticSearch. With this you can use tools and clients that are meant
to talk to Solr with ElasticSearch. Some examples are Nutch, Apache
ManifoldCF, SolrJ apps, etc. Currently, indexing and deleting of
documents is supported 100% for XML (/update request handler) and
JavaBin (/update/javabin request handler). Basic support for the Solr
search handler (/select) is also included for the Solr q, start, rows,
and fl parameters. The q parameter supports 100% of the lucene query
syntax. Both XML and JavaBin response formats are supported.

To use the plugin:

  1. Install

$ES_HOME/bin/plugin install mattweber/elasticsearch-mocksolrplugin

  1. Update your client code to point at ElasticSearch and the /_solr
    REST endpoint.

http://localhost:9200/index/type/_solr

Specifying the index and type is optional and will default to "solr"
for index, and "docs" for type.

  1. Use your Solr client as normal.

I have tested the plugin with Nutch and various SolrJ test code.
Using Nutch with ElasticSearch is the reason I wrote this plugin.
Instead of extending Nutch to support ElasticSearch as an endpoint, I
figured it would be much better to support any tool trying to talk to
Solr. This plugin should greatly reduce the effort in testing and/or
replacing Solr with ElasticSearch. It also opens the doors for using
tools that were previously not available to ElasticSearch users.

Please let me know if you have any questions, comments, or suggestions.

Source available on GitHub:https://github.com/mattweber/elasticsearch-mocksolrplugin

Thanks,
Matt Weber


(Matt Weber) #7

Thanks for the feedback everyone!

@Peter
Per your comment about using this plugin with SolrMeter on twitter.
Looks like it is a go, replace queries for the solr "text" field with
the "_all" field and you should just work. I plan on more of the
search functionality soon.

@David
That is exactly what I want. All the Solr people talk about the ease
of use and tools available for Solr so people can get started quickly.
Not that ElasticSearch is hard to get started, but hopefully this
plugin eliminates most of that nonsense. In fact I took the example
docs that ship with Solr, modified post.sh to point at my
elasticsearch instance, and did a "./post.sh *.xml" and almost all the
docs were indexed in ES no problem. The only issue I ran into were
the docs that used Solr's "date math". Shay, have you thought about
adding similar functionality?

@Shay
Thanks for the tip, I have updated the pom file to use the same format
as the official plugins.

@Piergiorgio
I briefly glanced at Apache ManifoldCF and it looks like it uses the
Solr extracting request handler. Unfortunately this plugin does not
mock that yet, so I don't think it will work yet. If it turns out
that they use the standard xml or javabin update handlers, then it
should work without issue. Please let me know when you find out.

Thanks,
Matt Weber

On Tue, Dec 13, 2011 at 12:54 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi Piergiorgio,

I would be interested in hearing what you learn, do you think you could
share with us if that combo worked fine for you?

--
Regards,
Lukas

On Tuesday, December 13, 2011 at 9:41 PM, Piergiorgio Lucidi wrote:

Hi Matt,

it seems a great contribution to the project, I hope to spend some
time on it during this week.
I would like to test this plugin with Apache ManifoldCF to understand
if it could be useful to implement a new and specific ElasticSearch
Output Connector or not.

Thank you.
Piergiorgio

On Dec 13, 12:00 am, Matt Weber m...@mattweber.org wrote:

I just released a plugin that Mocks the Solr interface in
ElasticSearch. With this you can use tools and clients that are meant
to talk to Solr with ElasticSearch. Some examples are Nutch, Apache
ManifoldCF, SolrJ apps, etc. Currently, indexing and deleting of
documents is supported 100% for XML (/update request handler) and
JavaBin (/update/javabin request handler). Basic support for the Solr
search handler (/select) is also included for the Solr q, start, rows,
and fl parameters. The q parameter supports 100% of the lucene query
syntax. Both XML and JavaBin response formats are supported.

To use the plugin:

  1. Install

$ES_HOME/bin/plugin install mattweber/elasticsearch-mocksolrplugin

  1. Update your client code to point at ElasticSearch and the /_solr
    REST endpoint.

http://localhost:9200/index/type/_solr

Specifying the index and type is optional and will default to "solr"
for index, and "docs" for type.

  1. Use your Solr client as normal.

I have tested the plugin with Nutch and various SolrJ test code.
Using Nutch with ElasticSearch is the reason I wrote this plugin.
Instead of extending Nutch to support ElasticSearch as an endpoint, I
figured it would be much better to support any tool trying to talk to
Solr. This plugin should greatly reduce the effort in testing and/or
replacing Solr with ElasticSearch. It also opens the doors for using
tools that were previously not available to ElasticSearch users.

Please let me know if you have any questions, comments, or suggestions.

Source available on
GitHub:https://github.com/mattweber/elasticsearch-mocksolrplugin

Thanks,
Matt Weber


(system) #8