# Elasticsearch on ec2 tutorial - I can't get the S3 gateway to work

**URL:** <https://discuss.elastic.co/t/elasticsearch-on-ec2-tutorial-i-cant-get-the-s3-gateway-to-work/7044>\
**Category:** Elasticsearch\
**Created:** [March 18, 2012, 4:57pm UTC](https://discuss.elastic.co/t/elasticsearch-on-ec2-tutorial-i-cant-get-the-s3-gateway-to-work/7044 "2012-03-18T16:57:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jonathan\_Spooner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_spooner/32/2672_2.png) [@Jonathan\_Spooner](https://discuss.elastic.co/u/Jonathan_Spooner)\
**Post date:** [March 18, 2012, 4:57pm UTC](https://discuss.elastic.co/t/elasticsearch-on-ec2-tutorial-i-cant-get-the-s3-gateway-to-work/7044/1 "2012-03-18T16:57:39Z")

</div>

I'm following this tutorial[http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html](http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html) on  
setting up elasticsearch on ec2 and I can't get ES to use the S3 gateway.  
As the tutorial says I should see [gateway.s3] in the output but I don't.

The tutorial is a little outdated so kimchy told me to use  
elasticsearch-cloud-aws[https://github.com/elasticsearch/elasticsearch-cloud-aws](https://github.com/elasticsearch/elasticsearch-cloud-aws).  
So I installed cloud-aws withbin/plugin -install  
elasticsearch/elasticsearch-cloud-aws/1.4.0. After unsuccessfully hooking  
up to S3 I found this issue[https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8](https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8) that  
suggested installing like this bin/plugin -install  
elasticsearch/elasticsearch-cloud-aws

I've also played with cloud.aws.region but that didn't help.

So what should I try to get the S3 gateway working? Might this be an issue  
with the new 0.19.0 release?

Here is how I set up a micro instance with v0.19.0

wget [https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip](https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip)  
sudo unzip elasticsearch-0.19.0.zip -d /usr/local/elasticsearch  
cd /usr/local/elasticsearch/elasticsearch-0.19.0  
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.4.0  
sudo vim config/elasticsearch.yml  
sudo vim config/logging.yml  
sudo vim config/elasticsearch.yml  
ES\_MIN\_MEM=400mb  
ES\_MAX\_MEM=400mb  
echo $ES\_MAX\_MEM  
sudo bin/elasticsearch -f  
sudo vim config/elasticsearch.yml  
... snip  
sudo bin/plugin -install Aconex/elasticsearch-head  
sudo bin/elasticsearch -f  
sudo vim config/elasticsearch.yml  
... snip  
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws  
sudo bin/elasticsearch -f

elasticsearch.yml

cluster.name: elasticsearch-demo-js  
cloud:  
aws:  
access\_key: KEY  
secret\_key: SECRET\_KEY  
region: us-east  
discovery:  
type: ec2  
gateway:  
type: s3  
s3:  
bucket: es-demo-js  
gateway.recover\_after\_nodes: 1  
gateway.recover\_after\_time: 1m  
gateway.expected\_nodes: 2

logging.yml

rootLogger: INFO, console, file  
logger:

# log action execution errors for easier debugging

action: DEBUG

# reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

# gateway

gateway: DEBUG  
#index.gateway: DEBUG

# peer shard recovery

#indices.recovery: DEBUG

# discovery

discovery: TRACE

org.apache: WARN

index.search.slowlog: TRACE, index\_search\_slow\_log\_file

additivity:  
index.search.slowlog: false

appender:  
console:  
type: console  
layout:  
type: consolePattern  
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:  
type: dailyRollingFile  
file: ${path.logs}/${cluster.name}.log  
datePattern: "'.'yyyy-MM-dd"  
layout:  
type: pattern  
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index\_search\_slow\_log\_file:  
type: dailyRollingFile  
file: ${path.logs}/${cluster.name}\_index\_search\_slowlog.log  
datePattern: "'.'yyyy-MM-dd"  
layout:  
type: pattern  
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

OUTPUT

```
ec2-user@ip-xx elasticsearch-0.19.0]$ sudo bin/elasticsearch -f
[2012-03-18 16:36:10,786][WARN][bootstrap] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2012-03-18 16:36:10,791][INFO][node] [Roma] {0.19.0}[20285]: initializing ...
[2012-03-18 16:36:10,804][INFO][plugins] [Roma] loaded [cloud-aws], sites [head]
[2012-03-18 16:36:11,672][DEBUG][discovery.zen.ping.multicast] [Roma] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-03-18 16:36:11,675][DEBUG][discovery.zen.ping.unicast] [Roma] using initial hosts [], with concurrent_connects [10]
[2012-03-18 16:36:11,676][DEBUG][discovery.zen] [Roma] using ping.timeout [3s]
[2012-03-18 16:36:11,682][DEBUG][discovery.zen.elect] [Roma] using minimum_master_nodes [-1]
[2012-03-18 16:36:11,683][DEBUG][discovery.zen.fd] [Roma] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:11,685][DEBUG][discovery.zen.fd] [Roma] [node] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:12,400][DEBUG][gateway.local] [Roma] using initial_shards [quorum], list_timeout [30s]
[2012-03-18 16:36:12,589][DEBUG][gateway.local.state.shards] [Roma] took 51ms to load started shards state
[2012-03-18 16:36:12,639][DEBUG][gateway.local.state.meta] [Roma] took 49ms to load state
[2012-03-18 16:36:12,642][INFO][node] [Roma] {0.19.0}[20285]: initialized
[2012-03-18 16:36:12,642][INFO][node] [Roma] {0.19.0}[20285]: starting ...
[2012-03-18 16:36:12,703][INFO][transport] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.127.162.192:9300]}
[2012-03-18 16:36:12,719][TRACE][discovery] [Roma] waiting for 30s for the initial state to be set by the discovery
[2012-03-18 16:36:12,722][TRACE][discovery.zen.ping.multicast] [Roma] [4] sending ping request
[2012-03-18 16:36:14,224][TRACE][discovery.zen.ping.multicast] [Roma] [5] sending ping request
[2012-03-18 16:36:15,726][DEBUG][discovery.zen] [Roma] ping responses: {none}
[2012-03-18 16:36:15,729][INFO][cluster.service] [Roma] new_master [Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]], reason: zen-disco-join (elected_as_master)
[2012-03-18 16:36:15,762][TRACE][discovery] [Roma] initial state set from discovery
[2012-03-18 16:36:15,762][INFO][discovery] [Roma] elasticsearch-demo-js/KYVDhYLmSY-u8j4jVD7FaQ
[2012-03-18 16:36:15,763][DEBUG][gateway] [Roma] delaying initial state recovery for [1m]
[2012-03-18 16:36:15,766][INFO][http] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.127.162.192:9200]}
[2012-03-18 16:36:15,766][INFO][node] [Roma] {0.19.0}[20285]: started
[2012-03-18 16:37:15,778][DEBUG][gateway.local] [Roma] [twitter][0]: allocating [[twitter][0], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local] [Roma] [twitter][6]: allocating [[twitter][7], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local] [Roma] [twitter][8]: allocating [[twitter][9], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local] [Roma] [twitter][10]: allocating [[twitter][11], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local] [Roma] [twitter][12]: throttling allocation [[twitter][13], node[null], [P], s[UNASSIGNED]] to [[[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]]] on primary allocation
[2012-03-18 16:37:16,050][DEBUG][index.gateway] [Roma] [twitter][0] starting recovery from local ...
[2012-03-18 16:37:16,138][DEBUG][index.gateway] [Roma] [twitter][14] starting recovery from local ...
[2012-03-18 16:37:16,150][DEBUG][index.gateway] [Roma] [twitter][0] recovery completed from local, took [100ms]

```

Thanks for checking this out!

---

<div class="post-metadata">

**Author:** ![Jonathan\_Spooner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_spooner/32/2672_2.png) [@Jonathan\_Spooner](https://discuss.elastic.co/u/Jonathan_Spooner)\
**Post date:** [March 19, 2012, 1:02am UTC](https://discuss.elastic.co/t/elasticsearch-on-ec2-tutorial-i-cant-get-the-s3-gateway-to-work/7044/2 "2012-03-19T01:02:58Z")

</div>

Solved. Discovery and gateway were nested under cloud.aws.

Sorry

On Sunday, March 18, 2012 9:57:39 AM UTC-7, Jonathan Spooner wrote:

> I'm following this tutorial[http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html](http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html) on  
> setting up elasticsearch on ec2 and I can't get ES to use the S3 gateway.  
> As the tutorial says I should see [gateway.s3] in the output but I don't.
> 
> The tutorial is a little outdated so kimchy told me to use  
> elasticsearch-cloud-aws[https://github.com/elasticsearch/elasticsearch-cloud-aws](https://github.com/elasticsearch/elasticsearch-cloud-aws).  
> So I installed cloud-aws withbin/plugin -install  
> elasticsearch/elasticsearch-cloud-aws/1.4.0. After unsuccessfully hooking  
> up to S3 I found this issue[https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8](https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8) that  
> suggested installing like this bin/plugin -install  
> elasticsearch/elasticsearch-cloud-aws
> 
> I've also played with cloud.aws.region but that didn't help.
> 
> So what should I try to get the S3 gateway working? Might this be an issue  
> with the new 0.19.0 release?
> 
> Here is how I set up a micro instance with v0.19.0
> 
> wget [https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip](https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip)  
> sudo unzip elasticsearch-0.19.0.zip -d /usr/local/elasticsearch  
> cd /usr/local/elasticsearch/elasticsearch-0.19.0  
> sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.4.0  
> sudo vim config/elasticsearch.yml  
> sudo vim config/logging.yml  
> sudo vim config/elasticsearch.yml  
> ES\_MIN\_MEM=400mb  
> ES\_MAX\_MEM=400mb  
> echo $ES\_MAX\_MEM  
> sudo bin/elasticsearch -f  
> sudo vim config/elasticsearch.yml  
> ... snip  
> sudo bin/plugin -install Aconex/elasticsearch-head  
> sudo bin/elasticsearch -f  
> sudo vim config/elasticsearch.yml  
> ... snip  
> sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws  
> sudo bin/elasticsearch -f
> 
> elasticsearch.yml
> 
> cluster.name: elasticsearch-demo-js  
> cloud:  
> aws:  
> access\_key: KEY  
> secret\_key: SECRET\_KEY  
> region: us-east  
> discovery:  
> type: ec2  
> gateway:  
> type: s3  
> s3:  
> bucket: es-demo-js  
> gateway.recover\_after\_nodes: 1  
> gateway.recover\_after\_time: 1m  
> gateway.expected\_nodes: 2
> 
> logging.yml
> 
> rootLogger: INFO, console, file  
> logger:
> 
> # log action execution errors for easier debugging
> 
> action: DEBUG
> 
> # reduce the logging for aws, too much is logged under the default INFO
> 
> com.amazonaws: WARN
> 
> # gateway
> 
> gateway: DEBUG  
> #index.gateway: DEBUG
> 
> # peer shard recovery
> 
> #indices.recovery: DEBUG
> 
> # discovery
> 
> discovery: TRACE
> 
> org.apache: WARN
> 
> index.search.slowlog: TRACE, index\_search\_slow\_log\_file
> 
> additivity:  
> index.search.slowlog: false
> 
> appender:  
> console:  
> type: console  
> layout:  
> type: consolePattern  
> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
> 
> file:  
> type: dailyRollingFile  
> file: ${path.logs}/${cluster.name}.log  
> datePattern: "'.'yyyy-MM-dd"  
> layout:  
> type: pattern  
> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
> 
> index\_search\_slow\_log\_file:  
> type: dailyRollingFile  
> file: ${path.logs}/${cluster.name}\_index\_search\_slowlog.log  
> datePattern: "'.'yyyy-MM-dd"  
> layout:  
> type: pattern  
> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
> 
> OUTPUT
> 
> ```
> ec2-user@ip-xx elasticsearch-0.19.0]$ sudo bin/elasticsearch -f
> [2012-03-18 16:36:10,786][WARN][bootstrap] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
> [2012-03-18 16:36:10,791][INFO][node] [Roma] {0.19.0}[20285]: initializing ...
> [2012-03-18 16:36:10,804][INFO][plugins] [Roma] loaded [cloud-aws], sites [head]
> [2012-03-18 16:36:11,672][DEBUG][discovery.zen.ping.multicast] [Roma] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
> [2012-03-18 16:36:11,675][DEBUG][discovery.zen.ping.unicast] [Roma] using initial hosts [], with concurrent_connects [10]
> [2012-03-18 16:36:11,676][DEBUG][discovery.zen] [Roma] using ping.timeout [3s]
> [2012-03-18 16:36:11,682][DEBUG][discovery.zen.elect] [Roma] using minimum_master_nodes [-1]
> [2012-03-18 16:36:11,683][DEBUG][discovery.zen.fd] [Roma] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
> [2012-03-18 16:36:11,685][DEBUG][discovery.zen.fd] [Roma] [node] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
> [2012-03-18 16:36:12,400][DEBUG][gateway.local] [Roma] using initial_shards [quorum], list_timeout [30s]
> [2012-03-18 16:36:12,589][DEBUG][gateway.local.state.shards] [Roma] took 51ms to load started shards state
> [2012-03-18 16:36:12,639][DEBUG][gateway.local.state.meta] [Roma] took 49ms to load state
> [2012-03-18 16:36:12,642][INFO][node] [Roma] {0.19.0}[20285]: initialized
> [2012-03-18 16:36:12,642][INFO][node] [Roma] {0.19.0}[20285]: starting ...
> [2012-03-18 16:36:12,703][INFO][transport] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.127.162.192:9300]}
> [2012-03-18 16:36:12,719][TRACE][discovery] [Roma] waiting for 30s for the initial state to be set by the discovery
> [2012-03-18 16:36:12,722][TRACE][discovery.zen.ping.multicast] [Roma] [4] sending ping request
> [2012-03-18 16:36:14,224][TRACE][discovery.zen.ping.multicast] [Roma] [5] sending ping request
> [2012-03-18 16:36:15,726][DEBUG][discovery.zen] [Roma] ping responses: {none}
> [2012-03-18 16:36:15,729][INFO][cluster.service] [Roma] new_master [Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]], reason: zen-disco-join (elected_as_master)
> [2012-03-18 16:36:15,762][TRACE][discovery] [Roma] initial state set from discovery
> [2012-03-18 16:36:15,762][INFO][discovery] [Roma] elasticsearch-demo-js/KYVDhYLmSY-u8j4jVD7FaQ
> [2012-03-18 16:36:15,763][DEBUG][gateway] [Roma] delaying initial state recovery for [1m]
> [2012-03-18 16:36:15,766][INFO][http] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.127.162.192:9200]}
> [2012-03-18 16:36:15,766][INFO][node] [Roma] {0.19.0}[20285]: started
> [2012-03-18 16:37:15,778][DEBUG][gateway.local] [Roma] [twitter][0]: allocating [[twitter][0], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
> [2012-03-18 16:37:15,778][DEBUG][gateway.local] [Roma] [twitter][6]: allocating [[twitter][7], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
> [2012-03-18 16:37:15,778][DEBUG][gateway.local] [Roma] [twitter][8]: allocating [[twitter][9], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
> [2012-03-18 16:37:15,779][DEBUG][gateway.local] [Roma] [twitter][10]: allocating [[twitter][11], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
> [2012-03-18 16:37:15,779][DEBUG][gateway.local] [Roma] [twitter][12]: throttling allocation [[twitter][13], node[null], [P], s[UNASSIGNED]] to [[[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]]] on primary allocation
> [2012-03-18 16:37:16,050][DEBUG][index.gateway] [Roma] [twitter][0] starting recovery from local ...
> [2012-03-18 16:37:16,138][DEBUG][index.gateway] [Roma] [twitter][14] starting recovery from local ...
> [2012-03-18 16:37:16,150][DEBUG][index.gateway] [Roma] [twitter][0] recovery completed from local, took [100ms]
> 
> ```
> 
> Thanks for checking this out!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:35am UTC](https://discuss.elastic.co/t/elasticsearch-on-ec2-tutorial-i-cant-get-the-s3-gateway-to-work/7044/3 "2017-07-06T03:35:40Z")

</div>