Incompatible Lucene versions between neo4j and ElasticSearch

Elastic Stack Elasticsearch
1

Hi. We are facing issues while upgrading Neo4j from 3.1 to 3.3 due to incompatible Lucene versions used by Neo4j and ElasticSearch. We are using Rest based client provided by ES to query data.

Please find the maven dependencies used for neo4j and elastic search rest client

NEO4J:
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>3.3.5</version>
</dependency>

<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-ogm-embedded-driver</artifactId>
<version>3.1.0</version>
<scope>test</scope>
</dependency>

ELASTICSEARCH:
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>6.2.0</version>
</dependency>

The lucene version used by Neo4j is 5.5.0 but the one required by ElasticSearch is 6.6.1.

When 5.5.0 version of Lucene is used, we are facing the following issue at runtime:

Caused by: java.lang.NoSuchFieldError: LUCENE_5_5_2 at org.elasticsearch.Version.<clinit>(Version.java:75) at org.elasticsearch.common.logging.DeprecationLogger.<clinit>(DeprecationLogger.java:159) at org.elasticsearch.common.ParseField.<clinit>(ParseField.java:35) at org.elasticsearch.client.RestHighLevelClient.lambda$getDefaultNamedXContents$47(RestHighLevelClient.java:673) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1696) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

We have also tried using the version 6.6.1, in which case the neo4j cypher queries are failing to get executed.
We are using one of the latest stable releases of elastic search Rest client. We have also tried shading approach but to no avail. Please provide suggestions as to how to resolve this issue.

2

May be you can shade one of the 2 clients but might be impossible to shade Lucene (can't recall).

Read https://www.elastic.co/blog/to-shade-or-not-to-shade

HTH

3

Unfortunately, as you said, I was unable to shade Lucene. It was causing a lot of conflicting dependency issues.

Do we have a pure Java REST API for querying ElasticSearch without lucene dependencies?

1.Going through the ES documentation, I understood that Native Java Client is gonna be deprecated in ES 7.0.
2.The lucene dependencies brought in by ElasticSearch High Level Client(6.x) is incompatible with Neo4j.
3.And it seems JEST is not yet upgraded to support ES 6.x.

Is there any other pure Java REST API Library that can talk to ES without any lucene dependencies?

Thanks in advance.

4

You can use the low level client which comes with very few deps.
But there is no fancy API exposed there. Just basic HTTP calls.

That's the situation for now. Things are going to evolve and hopefully in the future it won't be an issue anymore but this work will take a lot of time/work IMO.

5

Thanks for your suggestion. Using Rest Low level client resolved the issue since it had no lucene dependencies.

Need one quick suggestion on how to use the Bulk API in low level REST client. Currently I am using the Bulk API to post data for a single type and index.So if I have 10 types for the same index, I need to invoke the bulk API 10 times.

Is it possible to post data for multiple types but the same index using a single invocation of the bulk API. Have gone through the following documentation for the bulk API:

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

But the example explains usage of the bulk API for a single index and type.

6

Is it possible to post data for multiple types but the same index using a single invocation of the bulk API.

  1. Yes:
POST _bulk
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "foo" : "bar" }
{ "index" : { "_index" : "test", "_type" : "type2" } }
{ "foo" : "bar" }
  1. Types are going to be removed in next versions. So instead do something like:
POST _bulk
{ "index" : { "_index" : "test_type1", "_type" : "_doc" } }
{ "foo" : "bar" }
{ "index" : { "_index" : "test_type2", "_type" : "_doc" } }
{ "foo" : "bar" }

My 2 cents

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.