Elasticsearch on ec2 tutorial - I can't get the S3 gateway to work


(Jonathan Spooner) #1

I'm following this tutorialhttp://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html on
setting up elasticsearch on ec2 and I can't get ES to use the S3 gateway.
As the tutorial says I should see [gateway.s3] in the output but I don't.

The tutorial is a little outdated so kimchy told me to use
elasticsearch-cloud-awshttps://github.com/elasticsearch/elasticsearch-cloud-aws.
So I installed cloud-aws withbin/plugin -install
elasticsearch/elasticsearch-cloud-aws/1.4.0. After unsuccessfully hooking
up to S3 I found this issuehttps://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8 that
suggested installing like this bin/plugin -install
elasticsearch/elasticsearch-cloud-aws

I've also played with cloud.aws.region but that didn't help.

So what should I try to get the S3 gateway working? Might this be an issue
with the new 0.19.0 release?

Here is how I set up a micro instance with v0.19.0

wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip
sudo unzip elasticsearch-0.19.0.zip -d /usr/local/elasticsearch
cd /usr/local/elasticsearch/elasticsearch-0.19.0
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.4.0
sudo vim config/elasticsearch.yml
sudo vim config/logging.yml
sudo vim config/elasticsearch.yml
ES_MIN_MEM=400mb
ES_MAX_MEM=400mb
echo $ES_MAX_MEM
sudo bin/elasticsearch -f
sudo vim config/elasticsearch.yml
... snip
sudo bin/plugin -install Aconex/elasticsearch-head
sudo bin/elasticsearch -f
sudo vim config/elasticsearch.yml
... snip
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws
sudo bin/elasticsearch -f

elasticsearch.yml

cluster.name: elasticsearch-demo-js
cloud:
aws:
access_key: KEY
secret_key: SECRET_KEY
region: us-east
discovery:
type: ec2
gateway:
type: s3
s3:
bucket: es-demo-js
gateway.recover_after_nodes: 1
gateway.recover_after_time: 1m
gateway.expected_nodes: 2

logging.yml

rootLogger: INFO, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

gateway

gateway: DEBUG
#index.gateway: DEBUG

peer shard recovery

#indices.recovery: DEBUG

discovery

discovery: TRACE

org.apache: WARN

index.search.slowlog: TRACE, index_search_slow_log_file

additivity:
index.search.slowlog: false

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

OUTPUT

ec2-user@ip-xx elasticsearch-0.19.0]$ sudo bin/elasticsearch -f
[2012-03-18 16:36:10,786][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2012-03-18 16:36:10,791][INFO ][node                     ] [Roma] {0.19.0}[20285]: initializing ...
[2012-03-18 16:36:10,804][INFO ][plugins                  ] [Roma] loaded [cloud-aws], sites [head]
[2012-03-18 16:36:11,672][DEBUG][discovery.zen.ping.multicast] [Roma] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-03-18 16:36:11,675][DEBUG][discovery.zen.ping.unicast] [Roma] using initial hosts [], with concurrent_connects [10]
[2012-03-18 16:36:11,676][DEBUG][discovery.zen            ] [Roma] using ping.timeout [3s]
[2012-03-18 16:36:11,682][DEBUG][discovery.zen.elect      ] [Roma] using minimum_master_nodes [-1]
[2012-03-18 16:36:11,683][DEBUG][discovery.zen.fd         ] [Roma] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:11,685][DEBUG][discovery.zen.fd         ] [Roma] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:12,400][DEBUG][gateway.local            ] [Roma] using initial_shards [quorum], list_timeout [30s]
[2012-03-18 16:36:12,589][DEBUG][gateway.local.state.shards] [Roma] took 51ms to load started shards state
[2012-03-18 16:36:12,639][DEBUG][gateway.local.state.meta ] [Roma] took 49ms to load state
[2012-03-18 16:36:12,642][INFO ][node                     ] [Roma] {0.19.0}[20285]: initialized
[2012-03-18 16:36:12,642][INFO ][node                     ] [Roma] {0.19.0}[20285]: starting ...
[2012-03-18 16:36:12,703][INFO ][transport                ] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.127.162.192:9300]}
[2012-03-18 16:36:12,719][TRACE][discovery                ] [Roma] waiting for 30s for the initial state to be set by the discovery
[2012-03-18 16:36:12,722][TRACE][discovery.zen.ping.multicast] [Roma] [4] sending ping request
[2012-03-18 16:36:14,224][TRACE][discovery.zen.ping.multicast] [Roma] [5] sending ping request
[2012-03-18 16:36:15,726][DEBUG][discovery.zen            ] [Roma] ping responses: {none}
[2012-03-18 16:36:15,729][INFO ][cluster.service          ] [Roma] new_master [Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]], reason: zen-disco-join (elected_as_master)
[2012-03-18 16:36:15,762][TRACE][discovery                ] [Roma] initial state set from discovery
[2012-03-18 16:36:15,762][INFO ][discovery                ] [Roma] elasticsearch-demo-js/KYVDhYLmSY-u8j4jVD7FaQ
[2012-03-18 16:36:15,763][DEBUG][gateway                  ] [Roma] delaying initial state recovery for [1m]
[2012-03-18 16:36:15,766][INFO ][http                     ] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.127.162.192:9200]}
[2012-03-18 16:36:15,766][INFO ][node                     ] [Roma] {0.19.0}[20285]: started
[2012-03-18 16:37:15,778][DEBUG][gateway.local            ] [Roma] [twitter][0]: allocating [[twitter][0], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local            ] [Roma] [twitter][6]: allocating [[twitter][7], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local            ] [Roma] [twitter][8]: allocating [[twitter][9], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local            ] [Roma] [twitter][10]: allocating [[twitter][11], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local            ] [Roma] [twitter][12]: throttling allocation [[twitter][13], node[null], [P], s[UNASSIGNED]] to [[[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]]] on primary allocation
[2012-03-18 16:37:16,050][DEBUG][index.gateway            ] [Roma] [twitter][0] starting recovery from local ...
[2012-03-18 16:37:16,138][DEBUG][index.gateway            ] [Roma] [twitter][14] starting recovery from local ...
[2012-03-18 16:37:16,150][DEBUG][index.gateway            ] [Roma] [twitter][0] recovery completed from local, took [100ms]

Thanks for checking this out!


(Jonathan Spooner) #2

Solved. Discovery and gateway were nested under cloud.aws.

Sorry

On Sunday, March 18, 2012 9:57:39 AM UTC-7, Jonathan Spooner wrote:

I'm following this tutorialhttp://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html on
setting up elasticsearch on ec2 and I can't get ES to use the S3 gateway.
As the tutorial says I should see [gateway.s3] in the output but I don't.

The tutorial is a little outdated so kimchy told me to use
elasticsearch-cloud-awshttps://github.com/elasticsearch/elasticsearch-cloud-aws.
So I installed cloud-aws withbin/plugin -install
elasticsearch/elasticsearch-cloud-aws/1.4.0. After unsuccessfully hooking
up to S3 I found this issuehttps://github.com/elasticsearch/elasticsearch-cloud-aws/issues/8 that
suggested installing like this bin/plugin -install
elasticsearch/elasticsearch-cloud-aws

I've also played with cloud.aws.region but that didn't help.

So what should I try to get the S3 gateway working? Might this be an issue
with the new 0.19.0 release?

Here is how I set up a micro instance with v0.19.0

wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip
sudo unzip elasticsearch-0.19.0.zip -d /usr/local/elasticsearch
cd /usr/local/elasticsearch/elasticsearch-0.19.0
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.4.0
sudo vim config/elasticsearch.yml
sudo vim config/logging.yml
sudo vim config/elasticsearch.yml
ES_MIN_MEM=400mb
ES_MAX_MEM=400mb
echo $ES_MAX_MEM
sudo bin/elasticsearch -f
sudo vim config/elasticsearch.yml
... snip
sudo bin/plugin -install Aconex/elasticsearch-head
sudo bin/elasticsearch -f
sudo vim config/elasticsearch.yml
... snip
sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws
sudo bin/elasticsearch -f

elasticsearch.yml

cluster.name: elasticsearch-demo-js
cloud:
aws:
access_key: KEY
secret_key: SECRET_KEY
region: us-east
discovery:
type: ec2
gateway:
type: s3
s3:
bucket: es-demo-js
gateway.recover_after_nodes: 1
gateway.recover_after_time: 1m
gateway.expected_nodes: 2

logging.yml

rootLogger: INFO, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

gateway

gateway: DEBUG
#index.gateway: DEBUG

peer shard recovery

#indices.recovery: DEBUG

discovery

discovery: TRACE

org.apache: WARN

index.search.slowlog: TRACE, index_search_slow_log_file

additivity:
index.search.slowlog: false

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

OUTPUT

ec2-user@ip-xx elasticsearch-0.19.0]$ sudo bin/elasticsearch -f
[2012-03-18 16:36:10,786][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2012-03-18 16:36:10,791][INFO ][node                     ] [Roma] {0.19.0}[20285]: initializing ...
[2012-03-18 16:36:10,804][INFO ][plugins                  ] [Roma] loaded [cloud-aws], sites [head]
[2012-03-18 16:36:11,672][DEBUG][discovery.zen.ping.multicast] [Roma] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-03-18 16:36:11,675][DEBUG][discovery.zen.ping.unicast] [Roma] using initial hosts [], with concurrent_connects [10]
[2012-03-18 16:36:11,676][DEBUG][discovery.zen            ] [Roma] using ping.timeout [3s]
[2012-03-18 16:36:11,682][DEBUG][discovery.zen.elect      ] [Roma] using minimum_master_nodes [-1]
[2012-03-18 16:36:11,683][DEBUG][discovery.zen.fd         ] [Roma] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:11,685][DEBUG][discovery.zen.fd         ] [Roma] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-03-18 16:36:12,400][DEBUG][gateway.local            ] [Roma] using initial_shards [quorum], list_timeout [30s]
[2012-03-18 16:36:12,589][DEBUG][gateway.local.state.shards] [Roma] took 51ms to load started shards state
[2012-03-18 16:36:12,639][DEBUG][gateway.local.state.meta ] [Roma] took 49ms to load state
[2012-03-18 16:36:12,642][INFO ][node                     ] [Roma] {0.19.0}[20285]: initialized
[2012-03-18 16:36:12,642][INFO ][node                     ] [Roma] {0.19.0}[20285]: starting ...
[2012-03-18 16:36:12,703][INFO ][transport                ] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.127.162.192:9300]}
[2012-03-18 16:36:12,719][TRACE][discovery                ] [Roma] waiting for 30s for the initial state to be set by the discovery
[2012-03-18 16:36:12,722][TRACE][discovery.zen.ping.multicast] [Roma] [4] sending ping request
[2012-03-18 16:36:14,224][TRACE][discovery.zen.ping.multicast] [Roma] [5] sending ping request
[2012-03-18 16:36:15,726][DEBUG][discovery.zen            ] [Roma] ping responses: {none}
[2012-03-18 16:36:15,729][INFO ][cluster.service          ] [Roma] new_master [Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]], reason: zen-disco-join (elected_as_master)
[2012-03-18 16:36:15,762][TRACE][discovery                ] [Roma] initial state set from discovery
[2012-03-18 16:36:15,762][INFO ][discovery                ] [Roma] elasticsearch-demo-js/KYVDhYLmSY-u8j4jVD7FaQ
[2012-03-18 16:36:15,763][DEBUG][gateway                  ] [Roma] delaying initial state recovery for [1m]
[2012-03-18 16:36:15,766][INFO ][http                     ] [Roma] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.127.162.192:9200]}
[2012-03-18 16:36:15,766][INFO ][node                     ] [Roma] {0.19.0}[20285]: started
[2012-03-18 16:37:15,778][DEBUG][gateway.local            ] [Roma] [twitter][0]: allocating [[twitter][0], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local            ] [Roma] [twitter][6]: allocating [[twitter][7], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,778][DEBUG][gateway.local            ] [Roma] [twitter][8]: allocating [[twitter][9], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local            ] [Roma] [twitter][10]: allocating [[twitter][11], node[null], [P], s[UNASSIGNED]] to [[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]] on primary allocation
[2012-03-18 16:37:15,779][DEBUG][gateway.local            ] [Roma] [twitter][12]: throttling allocation [[twitter][13], node[null], [P], s[UNASSIGNED]] to [[[Roma][KYVDhYLmSY-u8j4jVD7FaQ][inet[/10.127.162.192:9300]]]] on primary allocation
[2012-03-18 16:37:16,050][DEBUG][index.gateway            ] [Roma] [twitter][0] starting recovery from local ...
[2012-03-18 16:37:16,138][DEBUG][index.gateway            ] [Roma] [twitter][14] starting recovery from local ...
[2012-03-18 16:37:16,150][DEBUG][index.gateway            ] [Roma] [twitter][0] recovery completed from local, took [100ms]

Thanks for checking this out!


(system) #3