Hi,
Sorry for the length of this post and thanks in advance for any insight here. I am hitting my head against a wall at this point.
I've made copy of default cloud-aws plugin and modified it to use kms server side encryption. Locally it works as expected but when I deploy it on EC2 instance is is not.
So I am instrumented the code with log statements (and then throws runtime) snippets and that code is never executed either though the stack clearly shows my classes be executed.
Here's some more detail.
I am working off of the 2.4 branch.
All my code is under org.elasticsearch.cloud.aws.kms as I started with the code in org.elasticsearch.cloud.aw and just tweaked it.
I build my plugin into s3-with-kms-2.4.5.zip and I looked a the sources in elasticsearch/plugins/s3-with-kms/target/ and it checks out.
ES is deployed in AWS on Ubuntu in /usr/share/elasticsearch
I upload s3-with-kms-2.4.5.zip to the instance.
I install my plugin code like this:
sudo /usr/share/elasticsearch/bin/plugin remove s3-with-kms
sudo /usr/share/elasticsearch/bin/plugin install -b file:///home/sam/s3-with-kms-2.4.5.zip
I restart ES. It gives a green health check. But when I try something like:
curl -XPUT 'localhost:9200/_snapshot/s3_repository?verify=false&pretty' -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "smarks-test",
"region":"us-east-1",
"kms_master_key_id":"kms_master_key_id"
}
I get this stack trace in the log
RemoteTransportException[[elasticsearch24-uba-master-02.dev3][10.40.89.25:9300][cluster:admin/repository/put]]; nested: RepositoryException[[cl-dev3-es-elasticsearch24-uba-backup] failed to create repository]; nested: NotSerializableExceptionWrapper[creation_exception: Guice creation errors:
- Error injecting constructor, java.lang.IllegalArgumentException: The default customer master key id must be specified
at org.elasticsearch.cloud.aws.kms.s3.S3Repository.(Unknown Source)
while locating org.elasticsearch.cloud.aws.kms.s3.S3Repository
while locating org.elasticsearch.repositories.Repository
1 error]; nested: IllegalArgumentException[The default customer master key id must be specified];
Caused by: RepositoryException[[cl-dev3-es-elasticsearch24-uba-backup] failed to create repository]; nested: NotSerializableExceptionWrapper[creation_exception: Guice creation errors:
- Error injecting constructor, java.lang.IllegalArgumentException: The default customer master key id must be specified
at org.elasticsearch.cloud.aws.kms.s3.S3Repository.(Unknown Source)
while locating org.elasticsearch.cloud.aws.kms.s3.S3Repository
while locating org.elasticsearch.repositories.Repository
1 error]; nested: IllegalArgumentException[The default customer master key id must be specified];
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:480)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:784)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: NotSerializableExceptionWrapper[creation_exception: Guice creation errors:
- Error injecting constructor, java.lang.IllegalArgumentException: The default customer master key id must be specified
at org.elasticsearch.cloud.aws.kms.s3.S3Repository.(Unknown Source)
while locating org.elasticsearch.cloud.aws.kms.s3.S3Repository
while locating org.elasticsearch.repositories.Repository
1 error]; nested: IllegalArgumentException[The default customer master key id must be specified];
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
... 11 more
Caused by: java.lang.IllegalArgumentException: The default customer master key id must be specified
at com.amazonaws.services.s3.model.KMSEncryptionMaterials.(KMSEncryptionMaterials.java:42)
at com.amazonaws.services.s3.model.KMSEncryptionMaterialsProvider.(KMSEncryptionMaterialsProvider.java:20)
at org.elasticsearch.cloud.aws.kms.InternalAwsS3Service.getClient(InternalAwsS3Service.java:150)
at org.elasticsearch.cloud.aws.kms.InternalAwsS3Service.client(InternalAwsS3Service.java:72)
at org.elasticsearch.cloud.aws.kms.s3.S3Repository.(S3Repository.java:138)
As you can see
org.elasticsearch.cloud.aws.kms.s3.S3Repository.(Unknown Source)
is getting executed - that's my code.
however none of my log statements in org.elasticsearch.cloud.aws.kms.s3.S3Repository are displayed. (Also, why is source unknown)
Also, this code is getting executed:
at org.elasticsearch.cloud.aws.kms.InternalAwsS3Service.getClient(InternalAwsS3Service.java:150)
but at the top of the getClient() method I have throw new Runtime() which never gets thrown.
So my big question is where is the code that actually getting executed and why can't I replace it with new code?
I have tried deleting /usr/share/elasticsearch/plugins/s3-with-km before
sudo /usr/share/elasticsearch/bin/plugin install -b file:///home/spmarks/s3-with-kms-2.4.5.zip
I also note that:
sudo /usr/share/elasticsearch/bin/plugin remove s3-with-kms
also deletes /usr/share/elasticsearch/plugins/s3-with-km
Any help really appreciated. Thanks.