Is it possible to make a ingest-plugin which processor is able to make an HTTP request

Elastic Stack Elasticsearch
1

The goal here is being able to send the data of documents to a service my companie has and then index the response as an anotation. Im having trouble opening a connection from the processor.

2

I started a similar thing 2 years ago and stopped it as it was not a very good idea at the end of the day because this is dramatically slowing down the index operations.

Instead it's better to use Logstash to do external lookups like this and then send the final document to the indexation process which will be fast then and won't block any indexing threads...

Coming back to your question. I think you can.

I'm having trouble opening a connection from the processor.

May be it's because of the security manager. What kind of error do you have?

3

Hey, thanks for the quick reply.

The error is the following:

Task :forbiddenApisMain FAILED
Forbidden method invocation: java.net.URLConnection#getInputStream() [Don't open socket connections]
in org.elasticsearch.plugin.ingest.library.LibraryProcessor (LibraryProcessor.java:61)
Forbidden method invocation: java.io.InputStreamReader#(java.io.InputStream) [Uses default charset]
in org.elasticsearch.plugin.ingest.library.LibraryProcessor (LibraryProcessor.java:61)
Scanned 3 class file(s) for forbidden API invocations (in 0.36s), 2 error(s).

4

Not sure if this is the problem here but I'd explore https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugin-authors.html#plugin-authors-jsm

5

I've wrapped my code in a Lambda function inside like this:

SecurityManager sm = System.getSecurityManager();
        if (sm != null) {
            // unprivileged code such as scripts do not have SpecialPermission
            sm.checkPermission(new SpecialPermission());
        }
        AccessController.doPrivileged((PrivilegedAction<Void>)
                () -> {
                    try {
                        InputStream is = con.getInputStream();
                        BufferedReader in = new BufferedReader(new InputStreamReader(is,StandardCharsets.UTF_8));
                        StringBuffer response = new StringBuffer();

                        String inputLine;
                        while ((inputLine = in.readLine()) != null) {
                            response.append(inputLine);
                        }
                        in.close();

                        //Read JSON response and print
                        JSONObject myResponse = new JSONObject(response.toString());
                        ingestDocument.setFieldValue(targetField, myResponse.getString("occupation"));
                        return null;
                    }catch(IOException ex){
                        return null;
                    }
                }
        );

and also granted socket permissions in the plugin-security.policy:

grant {
  permission java.io.FilePermission "*", "read,write";
  permission java.net.SocketPermission "*", "connect,accept";
};

But the problem still remains. Output of the Gradle clean check:

Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
Gradle Version : 5.2.1
OS Info : Linux 4.15.0-47-generic (amd64)
JDK Version : 12 (Oracle Corporation 12 [Java HotSpot(TM) 64-Bit Server VM 12+33])
JAVA_HOME : /usr/lib/jvm/jdk-12
Random Testing Seed : C4F42EF220CB8540
=======================================

Task :forbiddenApisMain FAILED
Forbidden method invocation: java.net.URLConnection#getInputStream() [Don't open socket connections]
in org.elasticsearch.plugin.ingest.library.LibraryProcessor (LibraryProcessor.java:83)
Scanned 3 class file(s) for forbidden API invocations (in 0.04s), 1 error(s).

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':forbiddenApisMain'.
> de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API calls failed, see log.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
11 actionable tasks: 11 executed
6

I don't know. May be @rjernst knows?

7

Sorry for pining you here but I'm quite lost on how to do this @rjernst

8

You'll need to look at @SuppressForbidden. Additionally, some of those forbidden uses you should just correct, for example with not providing a charset.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.