Helping debugging s3 pluging with 2.4 / intellij

samiam · April 25, 2017, 5:13pm

Hi,

I am successfully debugging elasticsearch 2.4 with Intellij, but I want to be able to step through the s3 snapshot plugin. I can't seem to install it and was looking for some help as to how to.

My ultimate goal is to add support for KMS encryption in 2.4 (which is the version we have production) but then port the patch to 5.x and submit it, but before I can actually write some code I'd like to be able to step though existing code.

I am stuck here:

curl -XPUT 'localhost:9200/_snapshot/s3_repository?verify=false&pretty' -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "my_s3_bucket",
    "region": "us-east1"
  }
}
'
{
  "error" : {
    "root_cause" : [ {
      "type" : "repository_exception",
      "reason" : "[s3_repository] failed to create repository"
    } ],
    "type" : "repository_exception",
    "reason" : "[s3_repository] failed to create repository",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Unknown [repository] type [s3]"
    }
  },
  "status" : 500
}

Note, when I get the 500, I can step through the code and see that plugin is not installed.

Also, this confirms my problem:

curl -i -H "Accept: application/json" "localhost:9200/_cat/plugins?v"
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Content-Length: 33

name component version type url

Any pointers as to how I can install the org.elasticsearch.repositories.s3.S3Repository so that I can step though would be greatly appreciated.

Thanks in advance.

jasontedor · April 25, 2017, 5:44pm

Here's my recommendation.

Build Elasticsearch and the plugin from source. Install the built plugin to the built version of Elasticsearch and start Elasticsearch with the following options:

ES_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

Attach a remote debugger in IntelliJ. Go to Run -> Edit Configurations -> Click + -> Click Remote -> set the address to localhost, the port to 5005, name it "Elasticsearch" -> Click OK. This gives a debug configuration that you can select from the upper-right-hand corner, then click the bug icon to attach to Elasticsearch. Now you can set breakpoints until your is content.

I hope that helps.

samiam · April 25, 2017, 5:51pm

Thanks I will give this a try.

When you say build from source, I assume you mean:

mvn -am -pl dev-tools,distribution/zip,plugins/ -DskipTests -Drun -Pdev for core.

and

mvn install -DskipTests

for

elasticsearch/plugins/cloud-aws

How do I make sure the plugin is registered and installed though?

I think I just attach to this running instance, I still won't be able to step into the plugin code.

Thanks again.

jasontedor · April 25, 2017, 6:28pm

No, I meant like this:

$ /opt/maven/latest/bin/mvn -am -pl distribution/tar package -DskipTests
$ /opt/maven/latest/bin/mvn -am -pl plugins/cloud-aws package -DskipTests
$ tar xf distribution/tar/target/releases/elasticsearch-2.4.5-SNAPSHOT.tar.gz -C ~/elasticsearch/
$ ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/bin/plugin install file:///home/jason/src/elastic/elasticsearch-2.x/plugins/cloud-aws/target/releases/cloud-aws-2.4.5-SNAPSHOT.zip 
$ ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/bin/elasticsearch

and now attach the debugger in IntelliJ as I described. You can definitely step through the plugin code like this.

samiam · April 25, 2017, 7:47pm

Thanks for those details. They make sense. I am almost there. However, when I try to attach Intellij I get Unable to open debugger port (localhost:5005) java.net.ConnectionException. "Connection Refused"

Also, I got a JVM warning about the plugin permissions. Here's what I see when I execute the code you provided.

Trying file:/Users/sam/dev/elasticsearch/plugins/cloud-aws/target/releases/cloud-aws-2.4.5-SNAPSHOT.zip ...
Downloading ................................................................DONE
Verifying file:/Users/sam/dev/elasticsearch/plugins/cloud-aws/target/releases/cloud-aws-2.4.5-SNAPSHOT.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed cloud-aws into /Users/sam/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/plugins/cloud-aws
sam@wintermute ~/dev/elasticsearch (FED-1221_make_S3_backup_Plugin_use_KMS) 
 > ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/bin/elasticsearch
Listening for transport dt_socket at address: 5005
[2017-04-25 15:39:15,702][INFO ][node                     ] [Onslaught] version[2.4.5-SNAPSHOT], pid[22814], build[4bda225/2017-04-25T19:30:44Z]
[2017-04-25 15:39:15,703][INFO ][node                     ] [Onslaught] initializing ...
[2017-04-25 15:39:16,387][INFO ][plugins                  ] [Onslaught] modules [reindex, lang-expression, lang-groovy], plugins [cloud-aws], sites []
[2017-04-25 15:39:16,452][INFO ][env                      ] [Onslaught] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [61.6gb], net total_space [930.5gb], spins? [unknown], types [hfs]
[2017-04-25 15:39:16,453][INFO ][env                      ] [Onslaught] heap size [989.8mb], compressed ordinary object pointers [true]
[2017-04-25 15:39:16,453][WARN ][env                      ] [Onslaught] max file descriptors [10240] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-04-25 15:39:18,279][INFO ][node                     ] [Onslaught] initialized
[2017-04-25 15:39:18,279][INFO ][node                     ] [Onslaught] starting ...

Thanks again. I really appreciate it.

samiam · April 25, 2017, 8:10pm

I

chmod -R +x ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/plugins/cloud-aws/

and restarted this server and was able to connect no problem this time.

However, when I run:

curl -XGET 'http://localhost:9200/_nodes/plugins?pretty'

I get:

{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "mSl_4aMfTqG7885qRLTnTA" : {
      "name" : "D'Spayre",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1",
      "version" : "2.4.5-SNAPSHOT",
      "build" : "4bda225",
      "http_address" : "127.0.0.1:9200",
      "plugins" : [ {
        "name" : "cloud-aws",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
        "jvm" : true,
        "classname" : "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
        "isolated" : true,
        "site" : false
      } ],
      "modules" : [ {
        "name" : "lang-expression",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "Lucene expressions integration for Elasticsearch",
        "jvm" : true,
        "classname" : "org.elasticsearch.script.expression.ExpressionPlugin",
        "isolated" : true,
        "site" : false
      }, {
        "name" : "lang-groovy",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "Groovy scripting integration for Elasticsearch",
        "jvm" : true,
        "classname" : "org.elasticsearch.script.groovy.GroovyPlugin",
        "isolated" : true,
        "site" : false
      }, {
        "name" : "reindex",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "_reindex and _update_by_query APIs",
        "jvm" : true,
        "classname" : "org.elasticsearch.index.reindex.ReindexPlugin",
        "isolated" : true,
        "site" : false
      } ]
    }
  }
}

the S3 plugin is not listed.

jasontedor · April 25, 2017, 8:14pm

In 2.x the plugin was called cloud-aws, that includes the S3 repository functionality (look at src/main/java/org/elasticsearch/repositories/s3/S3Repository) in the source tree for cloud-aws. Also, note the description in your API output there:

"description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories."

samiam · April 25, 2017, 8:21pm

Thanks, I am happily able to debug now!

samiam · April 25, 2017, 8:23pm

And, of course, I just completely missed the cloud-aws json.

Thanks again.

I might experiment with seeing I get non remote debugging working (just for easy of use) but this is great.

Thank you very much.

S.

jasontedor · April 25, 2017, 8:34pm

You're very welcome.

jasontedor · April 25, 2017, 8:35pm

It's possible to, but I've found this to be a vastly easier workflow (you can script all the steps prior to attaching). Your mileage may vary.

samiam · April 25, 2017, 8:40pm

Yeah, scripting sounds easy enough. Being able to click on stack traces is also nice though.

now I am getting

 "error" : {
    "root_cause" : [ {
      "type" : "repository_exception",
      "reason" : "[s3_repository] failed to create repository"
    } ],
    "type" : "repository_exception",
    "reason" : "[s3_repository] failed to create repository",
    "caused_by" : {
      "type" : "creation_exception",
      "reason" : "Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error",
      "caused_by" : {
        "type" : "amazon_client_exception",
        "reason" : "Unable to load AWS credentials from any provider in the chain"
      }
    }
  },

Now I need to figure out pass my AWS credentials. I have them in ~/.aws but it looks like that's not sufficient. (I can put them in json at least for now).

jasontedor · April 25, 2017, 9:00pm

I would suggest opening a new topic for that one if you don't make progress on where it's breaking down, I don't think it's related to the debugger being attached at all though.

samiam · May 5, 2017, 5:11pm

Hi,
So I have been pretty successful working the 2.4 branch using the commands you gave me. Thanks again for that. Now, I am thinking about porting my code to the current rev (5.x). I am new gradle and I have gotten stuck.
I can attach the debugger to a running ES, but I have tried for a while now but can't figure out how to install a plugin so I can step throught that. I don't seem to have a valid elasiticsearch-plugin. I was hoping you could provide me with 5.x versions of the commands that you gave me previously. (I want to be able to step through the respository-s3 code. Thanks in advance.

jasontedor · May 6, 2017, 2:05am

Sure:

$ /opt/gradle/latest/bin/gradle :distribution:tar:assemble :plugins:repository-s3:assemble
$ tar xf distribution/tar/build/distributions/elasticsearch-5.5.0-SNAPSHOT.tar.gz -C ~/elasticsearch/
$ ~/elasticsearch/elasticsearch-5.5.0-SNAPSHOT/bin/elasticsearch-plugin install file:///home/jason/src/elastic/elasticsearch-5.x/plugins/repository-s3/build/distributions/repository-s3-5.5.0-SNAPSHOT.zip 
$ ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ~/elasticsearch/elasticsearch-5.5.0-SNAPSHOT/bin/elasticsearch

jasontedor · May 6, 2017, 2:06am

Note that as the version in the 5.x branch changes, the version constant 5.5.0 above will have to change with that version number. The commands above work right now, they will not in the future when the version changes but will if you adapt the version accordingly.

samiam · May 6, 2017, 7:21pm

Thanks again. Could you point me to the guidelines for submitting patches for approval to the distrubution?

jasontedor · May 6, 2017, 7:23pm

You're welcome.

Take a look at the contributing guidelines; I can't promise they are up to date (we try, but you know how these things go). Also, when you submit a PR, there is a template, be sure to read it too.

samiam · May 8, 2017, 12:53pm

Thanks. Will read both.

Here's another question. Perhaps I should start a new thread with it though.
I am not seeing my access_key or secret_key getting loaded when I do this:

curl -XPUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -H 'Content-Type: application/json' -d'
{"type": "s3",
  "settings": {
    "bucket": "bucket-test",
    "region": "us-east-1",
    "access_key": "foo",
    "secret_key": "fumb",
  }
}'

The settings stuff is completely reworked from 2.4 where I was able do this:

    curl -XPUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -H 'Content-Type: application/json' -d'
    {"type": "s3",
      "settings": {
        "bucket": "bucket-test",
        "region": "us-east-1",
        "access_key": "foo",
        "secret_key": "fumb",
        "kms_master_key_id": "fie"
      }
    }'

and get access to to access_key , secret_key , ** kms_master_key_id** (so that I could easily create and AmazonS3EncryptionClient

I am working through the new settings code, but any pointers appreciated. Also, happy to create a new thread for this topic.

Thanks,

jasontedor · May 13, 2017, 9:26pm

Sure, I think that would be best.