Plugin issue with SPI and classpath

Elastic Stack Elasticsearch
1

Hi,

we have developed a plugin (siren) for elasticsearch that adds a new posting format. The last version of the plugin was released for elasticsearch 1.3.x. Today we are trying to upgrade it for elasticsearch 1.7.x, and we encounter an issue related to the SPI class that is not found in the classpath.

After installing the plugin in elasticsearch, all the jars including the one containing the SPI class and the META-INF/services/org.apache.lucene.codecs.PostingsFormat file referencing our posting format class are located in the plugins/siren-plugin directory. The plugin is loaded properly by elasticsearch, but when we try to index documents that use this posting format, we got the following exception:

Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'Siren10AFor' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090, completion090, XBloomFilter, Lucene40, Lucene41, IDVersion]
at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109)

If we move the jar containing the spi class directly to the lib directory of elasticsearch, then things are working properly. It looks like that the META-INF/services/org.apache.lucene.codecs.PostingsFormat file is not properly loaded when located in the plugins directory. We are aware of some changes regarding the support of custom posting formats in elasticsearch since 1.3.x, and we were wondering if those changes are not related to this issue ?

2

Indeed we used to be pretty lenient about the postings formats that could be configured, but later moved back as we noticed that users would happily optimize their elasticsearch setup by using special codecs or postings formats without understanding consequences in terms of backward compatibility. In 2.0 plugins are even getting their own classloader so the classes that they load are not visible to elasticsearch-core. So unfortunately, I don't see how you could expose custom postings formats through plugins in the future.

3
4

For the record, the issue with 1.x codeline was that PluginService adds plugin jars to the main classloader with https://docs.oracle.com/javase/7/docs/api/java/net/URLClassLoader.html#addURL(java.net.URL) but never reloads the classloader so the registries pick up the new entries.

5

Hi Adrien,

yes I am aware of the changes in 2.0 regarding custom posting formats, that's something a bit problematic for us when 2.0 will be out. Not sure yet how we will be able to port our plugin to 2.0. But at the moment, we are just trying to keep the plugin compatible with 1.7.x and subsequent versions of elasticsearch.

6

Hi Robert,

Thanks for the quick patch regarding lucene spi support for plugins. One question, is this patch resolving the issue with 1.x codeline ? The patch seems to have been tagged for 2.x only. Or is there another issue wrt 1.x ?

7

1.x is long dead to me. 2.x is dead to me and its not even out yet. I work on the master branch.

8

Hi Robert,

not sure I understand, do you mean that the issue with 1.x regarding the classloader will not be fixed ?

9

Rob hi,
i kindly make notice this patch in 1.x is quite important for us too.
I understand you're not working on that any longer would you please be able to forward this request appropriately or advice us on who to talk to.

thanks again
Giovanni

10

hey Giovanni,

I know you won't be happy about this answer but for 1.x this was never a tested scenario and in turn not an officially supported way of extending elasticsearch. We considered this functionality to be very very expert and low level but at this point we are more confident in 2.0 since we have lots of other checks in-place to ensure this actually works as expected. There are also a lot of improvements in Lucene 5.x that allows us to be more open along those lines. For 1.x we won't release any features anymore which means you won't be able to plug this in the way you did before.

11

Hi Simon,

understood for 1.x, we will then concentrate our effort for 2.0. However, I have some uncertainties about the integration of our plugin with 2.0 and maybe someone in the elastic team could help us clarify them.
In #9741, you have removed the ability to have custom per-field posting formats. In #8746, bleskes mentioned different possibilities to add custom posting formats in 2.0, either by using our own custom codec (with index.codec) or using custom java code. Looking at the source code of 2.0.0-beta1, I found the following comment in PerFieldMappingPostingFormat:

This allows users to change the low level postings format for individual fields per index in real time via the mapping API. If no specific postings format is configured for a specific field the default postings format is used.

Is this still relevant/valid for 2.0 ? How could someone use the mapping API to define a posting format for an individual field ?

Overall, I am trying to understand what would be the recommended way in 2.0 for a plugin to add a custom per-field posting format. Any pointers would be highly appreciated.

Thanks

1 Like
12

sorry for the late reply

you have to implement your own codec and return per-field postings format by overriding PerFieldMappingPostingFormatCodec#getPostingsFormatForField(String). You can't set the type in the mapping anymore it's considered an expert feature though. You can then set your codec per index with index.codec. You also need to expose your codec via the SPI loader but that hasn't change since previous versions.

simon