Plugin doesn't work with embedded elasticsearch in spring boot app's uber JAR

Elastic Stack Elasticsearch
1

Hi,

I'm running an embedded elasticsearch with attachment plugin inside one big spring-boot application ( as a stand-alone JAR). I have defined a contents field with the type = attachment and everything works fine when I run the app using mvn spring-boot:run command. If I build the final JAR using mvn clean install (or mvn package) and run it using java -jar <my_final_spring_boot_jar> this field will not get indexed at all! I have extended the Node class as follows :

public class NodeWithPlugins extends Node {
    
    private Version version;
    private Collection<Class<? extends Plugin>> plugins;
    
    public NodeWithPlugins(Environment environment, Version version, Collection<Class<? extends Plugin>> classpathPlugins) {
        super(environment, version, classpathPlugins);
        this.version = version;
        this.plugins = classpathPlugins;
      }
    
     public Collection<Class<? extends Plugin>> getPlugins() {
        return plugins;
    }

    public Version getVersion() {
      return version;
    }
}

...and I'm creating the node inside the @PostConstruct annotated method like this :

Collection plugins = new ArrayList<>();
        Collections.<Class<? extends Plugin>>addAll(plugins, MapperAttachmentsPlugin.class);
        node = new NodeWithPlugins(environment, Version.CURRENT, plugins);
        node.start();

My pom.xml has the following :

<!-- Elasticsearch -->
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>2.3.3</version>
</dependency>
                
 <dependency>
    <groupId>org.elasticsearch.plugin</groupId>
    <artifactId>mapper-attachments</artifactId>
    <version>2.3.3</version>
 </dependency>

...and I can see that the mapper plugin gets embedded inside the final JAR (inside libs).

Am I missing something? Please advise.

2

Do you see errors somewhere when running it through java -jar? Does it even start?

3

Hi Luca,

Yes, ES starts and it's working fine...no errors, nothing. It simply won't create attachment field in the index, all other fields (non-attachment fields like string, int etc) are there and have their expected values (I'm indexing some PDFs).

The only thing that comes to my mind is to increase the log level for Tika and attachment packages to TRACE, haven't tried that. Any other ideas?

Thanks.

4

So running Elasitcsearch inside another application isn't supported. It might work OK for now but it isn't a thing we test. Plugins are the thing I expect to be most brittle so I'm not surprised they broke.

The trouble is that testing embedded would bring a huge amount of cases to test and running embedded has somewhat divergent requirement from a security standpoint.

Again, it might work for you and you might be able to get it working but it is very likely to break unexpectedly because it isn't a thing we think about.

5

Can you quote the log messages when you start the node? There must be a line that is printing the installed plugins.

Why don't you just use the simple add() ?

Collection<Class<? extends Plugin>> plugins = new ArrayList<>();
plugins.add(MapperAttachmentsPlugin.class);
node = new NodeWithPlugins(environment, Version.CURRENT, plugins);
node.start();

Also the NodeWithPluginsyou specified, you can not use settings. This is very bad for set up any node! Just set up a constructor that can initialize from settings, like this

    public NodeWithPlugins(Settings settings, Collection<Class<? extends Plugin>> classpathPlugins) {
        super(InternalSettingsPreparer.prepareEnvironment(settings, null), Version.CURRENT, classpathPlugins);
    }
6

OK, so I have increased the log level to DEBUG for all org.elasticsearch packages and this is what I got (output.log). I'm seeing one exception (mustache), but not sure if this is causing it.

Btw, forgot to mention, I'm setting the mappings programmatically like this :slight_smile:

PutMappingRequestBuilder putMappingRequestBuilder = new PutMappingRequestBuilder(client.admin().indices(), PutMappingAction.INSTANCE);
PutMappingResponse actionGet = putMappingRequestBuilder
.setIndices(indexName)
.setType("file")
.setSource(
jsonBuilder()
.startObject()
.field("file")
.startObject()
.field("properties")

                                    .startObject()

                                        .field("contents")
                                            .startObject()
                                                .field("type", "attachment")
                                                .field("fields")
                                                    .startObject()
                                                        .field("content")
                                                            .startObject()
                                                                .field("index", "analyzed")
                                                                //.field("store", "no")
                                                                .field("store", "yes")
                                                                .field("term_vector", "with_positions_offsets")
                                                            .endObject()
                                                        .field("title")
                                                            .startObject()
                                                                .field("store", "yes")
                                                            .endObject()
                                                        .field("date")
                                                            .startObject()
                                                                .field("store", "yes")
                                                            .endObject()
                                                        .field("author")
                                                            .startObject()
                                                                .field("store", "yes")
                                                            .endObject()
                                                        .field("keywords")
                                                            .startObject()
                                                                .field("store", "yes")
                                                            .endObject()
                                                        .field("content_type")
                                                            .startObject()
                                                                .field("store", "yes")
                                                            .endObject()
                                                      
                                                        .field("language")
                                                            .startObject()
                                                                .field("store", "yes")
                                                            .endObject()
                                                    .endObject() // end of fields
                                            .endObject() // end of contents
                                        
                                    .endObject() // end of properties
                            
                            .endObject()
                            .endObject()
                    ).execute().actionGet();

Tried Jorg's suggestions but it didn't help - now my index is completely empty?

7

Here's two screenshots depicting how index looks on the disc and what happens when I open dir 0 in Luke.

luke.jpg866×599 92.1 KB