Building a plugin with Gradle and Elasticsearch 5.0

[Damien_Alexandre]

Hi there,
I'm trying to build a plugin for Elasticsearch, but I have to tell the documentation is very thin .
Basically all we have is an example plugin in the ES core, but that doesn't say how to build, how to start from scratch, what are the best practices...
For example @dadoonet started an awesome tutorial about creating an Ingest plugin, but it's about Maven. As far as I understand, Gradle is the new standard in ES, so I tried to stick with it.

As I do not know the modern Java ecosystem, I had a really hard time figuring out how to make everything works. You can see my code here: https://github.com/jolicode/emoji-search/tree/esplugin/esplugin (it's in a branch ),

Anyway, here are my questions:

What are the release guidelines?

I read somewhere that since 5.0, plugin authors must release a new version of their plugins every-time Elasticsearch bump a version, is that correct? Even if nothing changes in the plugin?

How to declare conflicts?

My plugin require the latest ICU (58.1) and I have no idea what will happens if an user try to enable both my plugin (ICU 58.1) and the official analysis-icu plugin (ICU 54.1). Maybe I can declare that my plugin should not be enabled alongside analysis-icu? Or will it works ok to have two modules loading their own versions of ICU?

And is it possible to just extend the core analysis-icu plugin? That way my plugin could just add new tokenizer on top of it?

How to test a new tokenizer?

My test class is pretty simple, at the beginning I was trying to setup the same tests than the icu_tokenizer, but I never managed to get createTestAnalysis method to works, as seen here: https://github.com/elastic/elasticsearch/blob/master/plugins/analysis-icu/src/test/java/org/elasticsearch/index/analysis/IcuTokenizerFactoryTests.java#L103
Is there a magic I didn't see to import this method in the tests?

Any idea to make this code less ugly?

If you check out the code you will see that I'm loading ICU RuleBasedBreakIterator, I get the default rules, and I edit them with a String.replace...
I looked at the code, but didn't find a better way to have the default rules + my edits in place

That's it,
thanks for your time! I will probably write a blog post about this whole process as it's a first for me and some steps are really not obvious
Cheers

[nik9000 (Nik Everett)]

@Damien_Alexandre

I read somewhere that since 5.0, plugin authors must release a new version of their plugins every-time Elasticsearch bump a version, is that correct? Even if nothing changes in the plugin?

We check that the version that of Elasticsearch that the plugin was declared to have been built against matches the version you are running it with exactly. This is because plugins get access to all the internal interfaces and classes which we change as needed.

@Damien_Alexandre

My plugin require the latest ICU (58.1) and I have no idea what will happens if an user try to enable both my plugin (ICU 58.1) and the official analysis-icu plugin (ICU 54.1). Maybe I can declare that my plugin should not be enabled alongside analysis-icu? Or will it works ok to have two modules loading their own versions of ICU?

It should work fine. All plugins are loaded in child classloaders.

@Damien_Alexandre

And is it possible to just extend the core analysis-icu plugin? That way my plugin could just add new tokenizer on top of it?

Plugins can't extend plugins at this point. They can't see plugins at all because of the child classloader. We needed this to make plugin security work. You can certainly fork the analysis-icu plugin as a starting point.

@Damien_Alexandre

Is there a magic I didn't see to import this method in the tests?

It is a static method on ESTestCase so it ought to work....

@Damien_Alexandre

I looked at the code, but didn't find a better way to have the default rules + my edits in place

I don't really know ICU so I'm not sure. One advantage that you have is that you tightly control all the versions so while upgrading ICU may break it, no one is going to sneak that in on you.

[dadoonet (David Pilato)]

@Damien_Alexandre

For example @dadoonet started an awesome tutorial about creating an Ingest plugin, but it's about Maven. As far as I understand, Gradle is the new standard in ES, so I tried to stick with it.

Thank you Damien!

@spinscale also wrote something great related to gradle: GitHub - spinscale/cookiecutter-elasticsearch-ingest-processor: A cookiecutter template for an elasticsearch ingest processor plugin

You can have a look and it will build a skeleton for your plugin using gradle.

[Damien_Alexandre]

@dadoonet

wrote something great related to gradle: GitHub - spinscale/cookiecutter-elasticsearch-ingest-processor: A cookiecutter template for an elasticsearch ingest processor plugin

Yes I did use this example too, very useful! That's something I wish one could find in the documentation
There is a lot of options for Gradle, like licenseHeaders.enabled = false, that I didn't find anywhere else.

@nik9000 Thanks for your advices and answers, that helps a lot to go forward!

[jprante (Jörg Prante)]

@Damien_Alexandre

Any idea to make this code less ugly?

The code is not ugly at all. It's not an ES 5 question but a Lucene related one. I use the same anonymous class mechanism at https://github.com/jprante/elasticsearch-plugin-bundle/blob/master/src/main/java/org/xbib/elasticsearch/index/analysis/icu/IcuTokenizerFactory.java

Maybe you don't want to depend on break iterator default rule settings. Then you should just set up your own rules, like I do with custom rule files and set it to ICUTokenizerConfig. Then, you don't need to worry about .replace() will work in future versions or not.