I'm following this tutorialhttp://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html on
setting up elasticsearch on ec2 and I can't get ES to use the S3 gateway.
As the tutorial says I should see [gateway.s3] in the output but I don't.
The tutorial is a little outdated so kimchy told me to use
elasticsearch-cloud-awshttps://github.com/elasticsearch/elasticsearch-cloud-aws.
So I installed cloud-aws withbin/plugin -install
elasticsearch/elasticsearch-cloud-aws/1.4.0. After unsuccessfully hooking
up to S3 I found this issuehttps://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8 that
suggested installing like this bin/plugin -install
elasticsearch/elasticsearch-cloud-aws
I've also played with cloud.aws.region but that didn't help.
So what should I try to get the S3 gateway working? Might this be an issue
with the new 0.19.0 release?
Here is how I set up a micro instance with v0.19.0
wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip
sudo unzip elasticsearch-0.19.0.zip -d /usr/local/elasticsearch
cd /usr/local/elasticsearch/elasticsearch-0.19.0
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.4.0
sudo vim config/elasticsearch.yml
sudo vim config/logging.yml
sudo vim config/elasticsearch.yml
ES_MIN_MEM=400mb
ES_MAX_MEM=400mb
echo $ES_MAX_MEM
sudo bin/elasticsearch -f
sudo vim config/elasticsearch.yml
... snip
sudo bin/plugin -install Aconex/elasticsearch-head
sudo bin/elasticsearch -f
sudo vim config/elasticsearch.yml
... snip
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws
sudo bin/elasticsearch -f
elasticsearch.yml
cluster.name: elasticsearch-demo-js
cloud:
aws:
access_key: KEY
secret_key: SECRET_KEY
region: us-east
discovery:
type: ec2
gateway:
type: s3
s3:
bucket: es-demo-js
gateway.recover_after_nodes: 1
gateway.recover_after_time: 1m
gateway.expected_nodes: 2
logging.yml
rootLogger: INFO, console, file
logger:
log action execution errors for easier debugging
action: DEBUG
reduce the logging for aws, too much is logged under the default INFO
com.amazonaws: WARN
gateway
gateway: DEBUG
#index.gateway: DEBUG
peer shard recovery
#indices.recovery: DEBUG
discovery
discovery: TRACE
org.apache: WARN
index.search.slowlog: TRACE, index_search_slow_log_file
additivity:
index.search.slowlog: false
appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
OUTPUT
ec2-user@ip-xx elasticsearch-0.19.0]$ sudo bin/elasticsearch -f
[2012-03-18 16:36:10,786][WARN ][bootstrap ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2012-03-18 16:36:10,791][INFO ][node ] [Roma] {0.19.0}[20285]: initializing ...
[2012-03-18 16:36:10,804][INFO ][plugins ] [Roma] loaded [cloud-aws], sites [head]
[2012-03-18 16:36:11,672][DEBUG][discovery.zen.ping.multicast] [Roma] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-03-18 16:36:11,675][DEBUG][discovery.zen.ping.unicast] [Roma] using initial hosts [], with concurrent_connects [10]
[2012-03-18 16:36:11,676][DEBUG][discovery.zen ] [Roma] using ping.timeout [3s]
[2012-03-18 16:36:11,682][DEBUG][discovery.zen.elect ] [Roma] using minimum_master_nodes [-1]
[2012-03-18 16:36:11,683][DEBUG][discovery.zen.fd ] [Roma] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:11,685][DEBUG][discovery.zen.fd ] [Roma] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:12,400][DEBUG][gateway.local ] [Roma] using initial_shards [quorum], list_timeout [30s]
[2012-03-18 16:36:12,589][DEBUG][gateway.local.state.shards] [Roma] took 51ms to load started shards state
[2012-03-18 16:36:12,639][DEBUG][gateway.local.state.meta ] [Roma] took 49ms to load state
[2012-03-18 16:36:12,642][INFO ][node ] [Roma] {0.19.0}[20285]: initialized
[2012-03-18 16:36:12,642][INFO ][node ] [Roma] {0.19.0}[20285]: starting ...
[2012-03-18 16:36:12,703][INFO ][transport ] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.127.162.192:9300]}
[2012-03-18 16:36:12,719][TRACE][discovery ] [Roma] waiting for 30s for the initial state to be set by the discovery
[2012-03-18 16:36:12,722][TRACE][discovery.zen.ping.multicast] [Roma] [4] sending ping request
[2012-03-18 16:36:14,224][TRACE][discovery.zen.ping.multicast] [Roma] [5] sending ping request
[2012-03-18 16:36:15,726][DEBUG][discovery.zen ] [Roma] ping responses: {none}
[2012-03-18 16:36:15,729][INFO ][cluster.service ] [Roma] new_master [Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]], reason: zen-disco-join (elected_as_master)
[2012-03-18 16:36:15,762][TRACE][discovery ] [Roma] initial state set from discovery
[2012-03-18 16:36:15,762][INFO ][discovery ] [Roma] elasticsearch-demo-js/KYVDhYLmSY-u8j4jVD7FaQ
[2012-03-18 16:36:15,763][DEBUG][gateway ] [Roma] delaying initial state recovery for [1m]
[2012-03-18 16:36:15,766][INFO ][http ] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.127.162.192:9200]}
[2012-03-18 16:36:15,766][INFO ][node ] [Roma] {0.19.0}[20285]: started
[2012-03-18 16:37:15,778][DEBUG][gateway.local ] [Roma] [twitter][0]: allocating [[twitter][0], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local ] [Roma] [twitter][6]: allocating [[twitter][7], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local ] [Roma] [twitter][8]: allocating [[twitter][9], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local ] [Roma] [twitter][10]: allocating [[twitter][11], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local ] [Roma] [twitter][12]: throttling allocation [[twitter][13], node[null], [P], s[UNASSIGNED]] to [[[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]]] on primary allocation
[2012-03-18 16:37:16,050][DEBUG][index.gateway ] [Roma] [twitter][0] starting recovery from local ...
[2012-03-18 16:37:16,138][DEBUG][index.gateway ] [Roma] [twitter][14] starting recovery from local ...
[2012-03-18 16:37:16,150][DEBUG][index.gateway ] [Roma] [twitter][0] recovery completed from local, took [100ms]
Thanks for checking this out!