Database access via plugins

[Ivan (Ivan Brusic)]

I am finally rebooting an old plugin that supports database access within
Elasticsearch to 5.x. Was able to quickly get around the new plugin
infrastructure, but I am failing to get past the new security features.

Using a JDBC4 driver, the DriverManager is not able to find a registered
driver
java.sql.SQLException: No suitable driver found for
jdbc:postgresql://localhost/test

This scenario is common if the app container does not support dynamically
loading drivers via the ServiceLoader. The plugin class loader must not
execute this step. No problem, I will just revert to the old
Class.forName() hack. That attempts results in another exception:
java.security.AccessControlException: access denied
("java.lang.RuntimePermission" "getClassLoader")

What can be done to load a database driver? I was going to cleanup some of
this code by using DataSources (via an apache commons wrapper), but I
suspect it will also get stuck by security features. This plugin is generic
and the actual database driver is lazy loaded.

Cheers,

Ivan

[nik9000 (Nik Everett)]

I'd go with old Class.forName way, yeah. We're careful to make sure we properly load Lucene's SPI extensions with plugins but it looks like those incantations don't cover the jdbc drivers.

Can you make a gist of the stack trace of the security exception? I wonder if the PostgreSQL driver is doing things that we don't normally give plugins permission to do. In that case you can work around it by granting your plugin permission to do the thing and wrapping whatever stack is doing the thing in something like this.

[Ivan (Ivan Brusic)]

Actually, my problem was in the permissions. Wanted to quickly reply, but
this mailing list does not send out an email of the initial post to the
author.

I assumed my security policies were correct, but they were misplaced and
not getting bundled properly. I simply assumed the ClassLoader was off
limits, even after explicitly granted permissions. Fixed the assembly, but
still not working, but I should be on the right path.

[nik9000 (Nik Everett)]

Good luck!

[Ivan (Ivan Brusic)]

Thanks! Turns out the system needed permissions in more than one place
beside establishing a connection.

Was able to get the code up and running with a couple of more tweaks. Since
every good developer writes test, I decided to use the embedded Derby
database for integration tests. What a nightmare in terms of security!
Here is a sample of the security permissions needed:
https://db.apache.org/derby/docs/10.12/security/rsecpolicysample.html
Turns out the list is not as big, but I have learned a lot about the Java
security model, which I have never needed to learn. I have been out of the
webapps world for a bit, concentrating on non-public APIs.

The only issue I am encountering is a NPE with the ESTestCase, but the
tests still complete:

Exception thrown by subscriber method
onHeartbeat(com.carrotsearch.ant.tasks.junit4.events.aggregated.HeartBeatEvent)
on subscriber com.carrotsearch.gradle.junit4.TestReportLogger@63de1613 when
dispatching event:
com.carrotsearch.ant.tasks.junit4.events.aggregated.HeartBeatEvent@10eeb8f4

java.lang.NullPointerException

    at

com.carrotsearch.ant.tasks.junit4.ForkedJvmInfo.getPidString(ForkedJvmInfo.java:85)

[rjernst (Ryan Ernst)]

The only issue I am encountering is a NPE with the ESTestCase

This issue was fixed upstream. Which version of ES are you developing against?

Since every good developer writes test, I decided to use the embedded Derby database for integration tests. What a nightmare in terms of security!

Instead of launching a test instance in memory (where you must then give your plugin all permissions needed for testing), we recommend creating a test fixture. See, for example, how the hdfs test fixture is used to test against hdfs.

[Ivan (Ivan Brusic)]

rjernst https://discuss.elastic.co/u/rjernst Ryan Ernst
https://discuss.elastic.co/u/rjernst Elastic Team Member
September 26

This issue was fixed upstream
https://github.com/randomizedtesting/randomizedtesting/issues/250.
Which version of ES are you developing against?

I am running against 5.4.1 for now since that is the version I am on. Will
test against latest once the feature set is complete.

Instead of launching a test instance in memory (where you must then give
your plugin all permissions needed for testing), we recommend creating a
test fixture. See, for example, how the hdfs test fixture
https://github.com/elastic/elasticsearch/tree/master/test/fixtures/hdfs-fixture
is used to test against hdfs
https://github.com/elastic/elasticsearch/blob/master/plugins/repository-hdfs/build.gradle#L69
.

Fixtures are an interesting way to solve the issue. I said I needed to
wrote unit tests, not particularly writing more detailed ones. The
elasticsearch examples are a good starting point, but they are all nested
deeply within the current overall build structure. For example: where is
the 'elasticsearch.build' plugin defined? Need to parse out hundreds of
lines of gradle build to figure out what is going on before I can tackle
any issues.

[rjernst (Ryan Ernst)]

The elasticsearch.build plugin is the base plugin in our build-tools official gradle plugin to develop Elasticsearch plugins. You can use it by adding the following to your build file:

buildscript {
  repositories {
    mavenCentral()
    jcenter()
  }
  dependencies {
    classpath 'org.elasticsearch.gradle:build-tools:5.4.1'
  }
}
apply plugin: 'elasticsearch.esplugin'

// NOTE: this should not be needed, but RestSpecHack needs to be fixed
apply plugin: 'idea'

esplugin {
  name 'myplugin'
  description 'it does something'
  classname 'Foo'
}

There may be some included tasks which you wish to disable, for example licenseHeaders. We still need actual docs on using the plugin, but at least this could get you started.