Using TransportClient within ingest plugin

Hi,

I have two indices.
One index is populated with data.
And from other index, when data is injected, I am calling index data (where data is populated) using transport client (via an inject plugin).
When I try to use this processor (plugin) within pipeline and simulate it, elasticsearch is running into error and exiting.

The error is
[2017-03-13T20:50:21,673][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[dSqUz60][management][T#1]], exiting
java.lang.NoClassDefFoundError: org/apache/http/auth/Credentials
at org.elasticsearch.index.reindex.ReindexPlugin.getSettings(ReindexPlugin.java:61) ~[?:?]
at org.elasticsearch.plugins.PluginsService.lambda$getPluginSettings$0(PluginsService.java:83) ~[elasticsearch-5.2.2.jar:5.2.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_121]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_121]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_121]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_121]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_121]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_121]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_121]
at org.elasticsearch.plugins.PluginsService.getPluginSettings(PluginsService.java:83) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:133) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:258) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:125) ~[?:?]
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:111) ~[?:?]
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:101) ~[?:?]
at com.leoforce.mk.elasticsearch.ingest.LocationCleanser.getClient(LocationCleanser.java:54) ~[?:?]
at com.leoforce.mk.elasticsearch.ingest.ProfileCleanProcessor.execute(ProfileCleanProcessor.java:78) ~[?:?]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:100) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:58) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.action.ingest.SimulateExecutionService.executeDocument(SimulateExecutionService.java:56) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.action.ingest.SimulateExecutionService$1.doRun(SimulateExecutionService.java:70) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.2.jar:5.2.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.ClassNotFoundException: org.apache.http.auth.Credentials
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_121]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_121]
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) ~[?:1.8.0_121]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_121]
... 26 more

I'm using the client as:
return new PreBuiltTransportClient(Settings.EMPTY)
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300));

Please let me know if I'm missing anything.

Thanks,
Mouli

Adding

org.apache.httpcomponents httpclient 4.5.3

in plugins maven dependencies fixed this issue.

Not sure if this is the right solution.

Comments are appreciated.

Thanks

Hey,

can you add the exact java code here as well? I am not sure if this is a bug, as you should not need that dependency for the TransportClient, but I'd like to be sure.

Also, which Elasticsearch version is this?

--Alex

Hi Alex,

This in on both elasticsearch version 5.2.0 and 5.2.2
And the code is,
public TransportClient getClient() {
try {
return new PreBuiltTransportClient(Settings.EMPTY)
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300));
} catch (Exception e) {
e.printStackTrace();
}
return null;
}

// in another method
String locationStr = ingestDocument.getFieldValue(FIELD_LOCATION, String.class, false);
QueryBuilder builder = QueryBuilders.termQuery("name.keyword", locationStr);
final TransportClient client = getClient();
if (null != client) {
SearchHits hits = client.prepareSearch("locations").setTypes("location").setQuery(builder).get();
client.close();
if(hits.totalHits() > 0){
Location location = parse(hits);
ingestDocument.setFieldValue("parsed_location", location);
}
}

And when I debugged, TransportReindexAction.REMOTE_CLUSTER_WHITELIST is giving the aforementioned error.

Few other questions I have:

  1. How do I ignore a document from my ingest plugin during bulk api? Is there any other way apart from throwing exception?
  2. Is creating a client from ingest plugin has any side affects? Is there a way I can close the client when server shutsdown?

Thanks,
Mouli

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.