# Helping debugging s3 pluging with 2.4 / intellij

**URL:** <https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602>\
**Category:** Elasticsearch\
**Created:** [April 25, 2017, 5:13pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602 "2017-04-25T17:13:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 5:13pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/1 "2017-04-25T17:13:26Z")

</div>

Hi,

I am successfully debugging elasticsearch 2.4 with Intellij, but I want to be able to step through the s3 snapshot plugin. I can't seem to install it and was looking for some help as to how to.

My ultimate goal is to add support for KMS encryption in 2.4 (which is the version we have production) but then port the patch to 5.x and submit it, but before I can actually write some code I'd like to be able to step though existing code.

I am stuck here:

```
curl -XPUT 'localhost:9200/_snapshot/s3_repository?verify=false&pretty' -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "my_s3_bucket",
    "region": "us-east1"
  }
}
'
{
  "error" : {
    "root_cause" : [ {
      "type" : "repository_exception",
      "reason" : "[s3_repository] failed to create repository"
    } ],
    "type" : "repository_exception",
    "reason" : "[s3_repository] failed to create repository",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Unknown [repository] type [s3]"
    }
  },
  "status" : 500
}

```

Note, when I get the 500, I can step through the code and see that plugin is not installed.

Also, this confirms my problem:

```
curl -i -H "Accept: application/json" "localhost:9200/_cat/plugins?v"
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Content-Length: 33

name component version type url

```

Any pointers as to how I can install the `org.elasticsearch.repositories.s3.S3Repository` so that I can step though would be greatly appreciated.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 25, 2017, 5:44pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/2 "2017-04-25T17:44:56Z")

</div>

Here's my recommendation.

Build Elasticsearch and the plugin from source. Install the built plugin to the built version of Elasticsearch and start Elasticsearch with the following options:

```auto
ES_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

```

Attach a remote debugger in IntelliJ. Go to Run -\> Edit Configurations -\> Click + -\> Click Remote -\> set the address to localhost, the port to 5005, name it "Elasticsearch" -\> Click OK. This gives a debug configuration that you can select from the upper-right-hand corner, then click the bug icon to attach to Elasticsearch. Now you can set breakpoints until your ❤ is content.

I hope that helps.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 5:51pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/3 "2017-04-25T17:51:50Z")

</div>

Thanks I will give this a try.

When you say build from source, I assume you mean:

`mvn -am -pl dev-tools,distribution/zip,plugins/ -DskipTests -Drun -Pdev for core.`

and

`mvn install -DskipTests`

for

elasticsearch/plugins/cloud-aws

How do I make sure the plugin is registered and installed though?

I think I just attach to this running instance, I still won't be able to step into the plugin code.

Thanks again.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 25, 2017, 6:28pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/4 "2017-04-25T18:28:50Z")

</div>

No, I meant like this:

```auto
$ /opt/maven/latest/bin/mvn -am -pl distribution/tar package -DskipTests
$ /opt/maven/latest/bin/mvn -am -pl plugins/cloud-aws package -DskipTests
$ tar xf distribution/tar/target/releases/elasticsearch-2.4.5-SNAPSHOT.tar.gz -C ~/elasticsearch/
$ ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/bin/plugin install file:///home/jason/src/elastic/elasticsearch-2.x/plugins/cloud-aws/target/releases/cloud-aws-2.4.5-SNAPSHOT.zip 
$ ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/bin/elasticsearch

```

and now attach the debugger in IntelliJ as I described. You can definitely step through the plugin code like this.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 7:47pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/5 "2017-04-25T19:47:25Z")

</div>

Thanks for those details. They make sense. I am almost there. However, when I try to attach Intellij I get Unable to open debugger port (localhost:5005) java.net.ConnectionException. "Connection Refused"

 ![](https://us1.discourse-cdn.com/elastic/original/3X/e/f/ef739f6ade6dcc6a459ca14649f666eaa41c3856.png)

Also, I got a JVM warning about the plugin permissions. Here's what I see when I execute the code you provided.

```
Trying file:/Users/sam/dev/elasticsearch/plugins/cloud-aws/target/releases/cloud-aws-2.4.5-SNAPSHOT.zip ...
Downloading ................................................................DONE
Verifying file:/Users/sam/dev/elasticsearch/plugins/cloud-aws/target/releases/cloud-aws-2.4.5-SNAPSHOT.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed cloud-aws into /Users/sam/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/plugins/cloud-aws
sam@wintermute ~/dev/elasticsearch (FED-1221_make_S3_backup_Plugin_use_KMS) 
 > ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/bin/elasticsearch
Listening for transport dt_socket at address: 5005
[2017-04-25 15:39:15,702][INFO][node] [Onslaught] version[2.4.5-SNAPSHOT], pid[22814], build[4bda225/2017-04-25T19:30:44Z]
[2017-04-25 15:39:15,703][INFO][node] [Onslaught] initializing ...
[2017-04-25 15:39:16,387][INFO][plugins] [Onslaught] modules [reindex, lang-expression, lang-groovy], plugins [cloud-aws], sites []
[2017-04-25 15:39:16,452][INFO][env] [Onslaught] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [61.6gb], net total_space [930.5gb], spins? [unknown], types [hfs]
[2017-04-25 15:39:16,453][INFO][env] [Onslaught] heap size [989.8mb], compressed ordinary object pointers [true]
[2017-04-25 15:39:16,453][WARN][env] [Onslaught] max file descriptors [10240] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-04-25 15:39:18,279][INFO][node] [Onslaught] initialized
[2017-04-25 15:39:18,279][INFO][node] [Onslaught] starting ...

```

Thanks again. I really appreciate it.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 8:10pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/6 "2017-04-25T20:10:07Z")

</div>

I

`chmod -R +x ~/elasticsearch/elasticsearch-2.4.5-SNAPSHOT/plugins/cloud-aws/`

and restarted this server and was able to connect no problem this time.

However, when I run:

`curl -XGET 'http://localhost:9200/_nodes/plugins?pretty'`

I get:

```
{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "mSl_4aMfTqG7885qRLTnTA" : {
      "name" : "D'Spayre",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1",
      "version" : "2.4.5-SNAPSHOT",
      "build" : "4bda225",
      "http_address" : "127.0.0.1:9200",
      "plugins" : [ {
        "name" : "cloud-aws",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
        "jvm" : true,
        "classname" : "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
        "isolated" : true,
        "site" : false
      } ],
      "modules" : [ {
        "name" : "lang-expression",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "Lucene expressions integration for Elasticsearch",
        "jvm" : true,
        "classname" : "org.elasticsearch.script.expression.ExpressionPlugin",
        "isolated" : true,
        "site" : false
      }, {
        "name" : "lang-groovy",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "Groovy scripting integration for Elasticsearch",
        "jvm" : true,
        "classname" : "org.elasticsearch.script.groovy.GroovyPlugin",
        "isolated" : true,
        "site" : false
      }, {
        "name" : "reindex",
        "version" : "2.4.5-SNAPSHOT",
        "description" : "_reindex and _update_by_query APIs",
        "jvm" : true,
        "classname" : "org.elasticsearch.index.reindex.ReindexPlugin",
        "isolated" : true,
        "site" : false
      } ]
    }
  }
}

```

the S3 plugin is not listed.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 25, 2017, 8:14pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/7 "2017-04-25T20:14:13Z")

</div>

In 2.x the plugin was called `cloud-aws`, that includes the S3 repository functionality (look at `src/main/java/org/elasticsearch/repositories/s3/S3Repository`) in the source tree for `cloud-aws`. Also, note the description in your API output there:

```auto
"description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories."

```

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 8:21pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/8 "2017-04-25T20:21:24Z")

</div>

Thanks, I am happily able to debug now!

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 8:23pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/9 "2017-04-25T20:23:40Z")

</div>

> [@jasontedor](#):
>
> Amazon Web Service

And, of course, I just completely missed the cloud-aws json.

Thanks again.

I might experiment with seeing I get non remote debugging working (just for easy of use) but this is great.

Thank you very much.

S.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 25, 2017, 8:34pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/10 "2017-04-25T20:34:38Z")

</div>

You're very welcome.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 25, 2017, 8:35pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/11 "2017-04-25T20:35:13Z")

</div>

> [@samiam](#):
>
> I might experiment with seeing I get non remote debugging working (just for easy of use) but this is great.

It's possible to, but I've found this to be a vastly easier workflow (you can script all the steps prior to attaching). Your mileage may vary.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [April 25, 2017, 8:40pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/12 "2017-04-25T20:40:01Z")

</div>

Yeah, scripting sounds easy enough. Being able to click on stack traces is also nice though.

now I am getting

```
 "error" : {
    "root_cause" : [ {
      "type" : "repository_exception",
      "reason" : "[s3_repository] failed to create repository"
    } ],
    "type" : "repository_exception",
    "reason" : "[s3_repository] failed to create repository",
    "caused_by" : {
      "type" : "creation_exception",
      "reason" : "Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain\n at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n while locating org.elasticsearch.repositories.s3.S3Repository\n while locating org.elasticsearch.repositories.Repository\n\n1 error",
      "caused_by" : {
        "type" : "amazon_client_exception",
        "reason" : "Unable to load AWS credentials from any provider in the chain"
      }
    }
  },

```

Now I need to figure out pass my AWS credentials. I have them in ~/.aws but it looks like that's not sufficient. (I can put them in json at least for now).

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 25, 2017, 9:00pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/13 "2017-04-25T21:00:30Z")

</div>

I would suggest opening a new topic for that one if you don't make progress on where it's breaking down, I don't think it's related to the debugger being attached at all though.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [May 5, 2017, 5:11pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/14 "2017-05-05T17:11:44Z")

</div>

Hi,  
So I have been pretty successful working the 2.4 branch using the commands you gave me. Thanks again for that. Now, I am thinking about porting my code to the current rev (5.x). I am new gradle and I have gotten stuck.  
I can attach the debugger to a running ES, but I have tried for a while now but can't figure out how to install a plugin so I can step throught that. I don't seem to have a valid elasiticsearch-plugin. I was hoping you could provide me with 5.x versions of the commands that you gave me previously. (I want to be able to step through the respository-s3 code. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [May 6, 2017, 2:05am UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/15 "2017-05-06T02:05:36Z")

</div>

Sure:

```auto
$ /opt/gradle/latest/bin/gradle :distribution:tar:assemble :plugins:repository-s3:assemble
$ tar xf distribution/tar/build/distributions/elasticsearch-5.5.0-SNAPSHOT.tar.gz -C ~/elasticsearch/
$ ~/elasticsearch/elasticsearch-5.5.0-SNAPSHOT/bin/elasticsearch-plugin install file:///home/jason/src/elastic/elasticsearch-5.x/plugins/repository-s3/build/distributions/repository-s3-5.5.0-SNAPSHOT.zip 
$ ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ~/elasticsearch/elasticsearch-5.5.0-SNAPSHOT/bin/elasticsearch

```

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [May 6, 2017, 2:06am UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/16 "2017-05-06T02:06:22Z")

</div>

Note that as the version in the 5.x branch changes, the version constant 5.5.0 above will have to change with that version number. The commands above work right now, they will not in the future when the version changes but will if you adapt the version accordingly.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [May 6, 2017, 7:21pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/17 "2017-05-06T19:21:32Z")

</div>

Thanks again. Could you point me to the guidelines for submitting patches for approval to the distrubution?

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [May 6, 2017, 7:23pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/18 "2017-05-06T19:23:51Z")

</div>

You're welcome.

Take a look at the [contributing guidelines](https://github.com/elastic/elasticsearch/blob/master/CONTRIBUTING.md); I can't promise they are up to date (we try, but you know how these things go). Also, when you submit a PR, there is a template, be sure to read it too.

---

<div class="post-metadata">

**Author:** ![samiam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samiam/32/17671_2.png) [@samiam](https://discuss.elastic.co/u/samiam)\
**Post date:** [May 8, 2017, 12:53pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/19 "2017-05-08T12:53:10Z")

</div>

Thanks. Will read both.

Here's another question. Perhaps I should start a new thread with it though.  
I am not seeing my access\_key or secret\_key getting loaded when I do this:

```
curl -XPUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -H 'Content-Type: application/json' -d'
{"type": "s3",
  "settings": {
    "bucket": "bucket-test",
    "region": "us-east-1",
    "access_key": "foo",
    "secret_key": "fumb",
  }
}'

```

The settings stuff is completely reworked from 2.4 where I was able do this:

```
    curl -XPUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -H 'Content-Type: application/json' -d'
    {"type": "s3",
      "settings": {
        "bucket": "bucket-test",
        "region": "us-east-1",
        "access_key": "foo",
        "secret_key": "fumb",
        "kms_master_key_id": "fie"
      }
    }'

```

and get access to to **access\_key** , **secret\_key** , \*\* kms\_master\_key\_id\*\* (so that I could easily create and AmazonS3EncryptionClient

I am working through the new settings code, but any pointers appreciated. Also, happy to create a new thread for this topic.

Thanks,

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [May 13, 2017, 9:26pm UTC](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602/20 "2017-05-13T21:26:36Z")

</div>

> [@samiam](#):
>
> Also, happy to create a new thread for this topic.

Sure, I think that would be best.

[Next page](https://discuss.elastic.co/t/helping-debugging-s3-pluging-with-2-4-intellij/83602.md?page=2)