Hi All,
I'm wondering if anyone can see what I may be doing wrong in my ES
configuration. A log and the resulting exception follow. Here is a
description of the relevant sections:
I am deploying to a single large instance on EC2.
// Section A - An ES data node is bootstrapped using Spring during the
initialization of a web app.
// [Grey King] starts successfully, elects itself as a master and the
cluster status is green.
// Please note that I am setting some sub-properties (i.e.
gateway.s3.bucket) even though I set
// the gateway.type to none. This makes it easier for me to switch between
dev, qa and
// production builds without maintaining completely separate configs for
each.
Loading properties file from class path resource
[elasticsearch-server.properties]
Starting the ElasticSearch server node:
[Sanders, James] {elasticsearch/0.13.1}[29609]: initializing ...
[Sanders, James] loaded [cloud-aws]
[Sanders, James] {elasticsearch/0.13.1}[29609]: initialized
Creating the ElasticSearch server node with these settings:
cloud.aws.access_key : AKIAJVHHVYQPNX5XMAGA
cloud.aws.secret_key :
cluster.name : roundtable-qa
discovery.type : ec2
gateway.s3.bucket : es-gateway-qa
gateway.type : none
http.enabled : false
http.port : 9311
index.analysis.analyzer.email_regex.lowercase : true
index.analysis.analyzer.email_regex.pattern : [@.]
index.analysis.analyzer.email_regex.stopwords : none
index.analysis.analyzer.email_regex.type : pattern
index.analysis.analyzer.email_split.filter.0 : lowercase
index.analysis.analyzer.email_split.tokenizer : letter
index.analysis.analyzer.email_split.type : custom
index.analysis.analyzer.lowercase_keyword.filter.0 : lowercase
index.analysis.analyzer.lowercase_keyword.tokenizer : keyword
index.analysis.analyzer.lowercase_keyword.type : custom
index.analysis.analyzer.porter_standard.char_filter.0 : html_strip
index.analysis.analyzer.porter_standard.filter.0 : standard
index.analysis.analyzer.porter_standard.filter.1 : lowercase
index.analysis.analyzer.porter_standard.filter.2 : stop
index.analysis.analyzer.porter_standard.filter.3 : porterStem
index.analysis.analyzer.porter_standard.tokenizer : standard
index.analysis.analyzer.porter_standard.type : custom
index.store.fs.memory.direct : true
index.store.memory.direct : true
index.store.memory.warm_cache : true
index.store.type : memory
name : Sanders, James
network.host : non_loopback
node.data : true
path.conf : /usr/local/tomcat/webapps/ROOT//WEB-INF/config
path.data :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/data
path.data_with_cluster :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/data/roundtable-qa
path.home :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target
path.logs :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/logs
path.work :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/work
path.work_with_cluster :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/work/roundtable-qa
transport.tcp.port : 9310
[Sanders, James] {elasticsearch/0.13.1}[29609]: starting ...
[Sanders, James] bound_address {inet[/10.192.10.240:9310]}, publish_address
{inet[/10.192.10.240:9310]}
Sending Request: POST http://ec2.amazonaws.com / Parameters: (Action:
DescribeInstances, SignatureMethod: HmacSHA256, AWSAccessKeyId:
AKIAJVHHVYQPNX5XMAGA, Version: 2010-08-31, SignatureVersion: 2, Timestamp:
2011-01-04T17:19:10.309Z, Signature:
8aWnQpe531bPMLVSg9/HRogzni9oQydEHZV/hPFU7kc=, )
Received successful response: 200, AWS Request ID:
04d4315a-9540-4ecc-b45f-1afbb415d7a1
Sending Request: POST http://ec2.amazonaws.com / Parameters: (Action:
DescribeInstances, SignatureMethod: HmacSHA256, AWSAccessKeyId:
AKIAJVHHVYQPNX5XMAGA, Version: 2010-08-31, SignatureVersion: 2, Timestamp:
2011-01-04T17:19:14.076Z, Signature:
mXcF8w6AabzmJVK3mqwW/ACyMi8+unozhXVQ8xOskAY=, )
Received successful response: 200, AWS Request ID:
9a4cda8e-dff2-4e15-b5d3-41deacbab06c
[Sanders, James] new_master [Sanders,
James][i4qlUhw0R1CfzyaKmDTEOQ][inet[/10.192.10.240:9310]], reason:
zen-disco-join (elected_as_master)
[Sanders, James] roundtable-qa/i4qlUhw0R1CfzyaKmDTEOQ
[Sanders, James] {elasticsearch/0.13.1}[29609]: started
ES cluster status is GREEN
// Section B - The Jetty web server is done initializing the servlet.
Root WebApplicationContext: initialization completed in 7941 ms
Jan 4, 2011 9:19:15 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jan 4, 2011 9:19:15 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jan 4, 2011 9:19:15 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/14 config=null
Jan 4, 2011 9:19:15 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 10661 ms
// Section C - This section of the log shows the creation of a non-data node
[Spyder], which
// I want to discover [Sanders, James] as master. Both of these nodes are in
the same VM,
// but only [Sanders, James] is a data node. Unfortunately, the nodes do not
discover each
// other.
[Spyder] {elasticsearch/0.13.1}[29609]: initializing ...
[Spyder] loaded [cloud-aws]
[Spyder] {elasticsearch/0.13.1}[29609]: initialized
[Spyder] {elasticsearch/0.13.1}[29609]: starting ...
[Spyder] bound_address {inet[/10.192.10.240:9312]}, publish_address {inet[/
10.192.10.240:9312]}
[Spyder] new_master
[Spyder][2lbzJ6yJQjGqcHQ3R5e7BQ][inet[/10.192.10.240:9312]]{data=false},
reason: zen-disco-join (elected_as_master)
[Spyder] roundtable-qa/2lbzJ6yJQjGqcHQ3R5e7BQ
[Spyder] {elasticsearch/0.13.1}[29609]: started
Creating the Hazelcast -> ElasticSearch client.
cloud.aws.access_key : AKIAJVHHVYQPNX5XMAGA
cloud.aws.secret_key :
cluster.name : roundtable-qa
http.enabled : false
index.analysis.analyzer.email_regex.lowercase : true
index.analysis.analyzer.email_regex.pattern : [@.]
index.analysis.analyzer.email_regex.stopwords : none
index.analysis.analyzer.email_regex.type : pattern
index.analysis.analyzer.email_split.filter.0 : lowercase
index.analysis.analyzer.email_split.tokenizer : letter
index.analysis.analyzer.email_split.type : custom
index.analysis.analyzer.lowercase_keyword.filter.0 : lowercase
index.analysis.analyzer.lowercase_keyword.tokenizer : keyword
index.analysis.analyzer.lowercase_keyword.type : custom
index.analysis.analyzer.porter_standard.char_filter.0 : html_strip
index.analysis.analyzer.porter_standard.filter.0 : standard
index.analysis.analyzer.porter_standard.filter.1 : lowercase
index.analysis.analyzer.porter_standard.filter.2 : stop
index.analysis.analyzer.porter_standard.filter.3 : porterStem
index.analysis.analyzer.porter_standard.tokenizer : standard
index.analysis.analyzer.porter_standard.type : custom
name : Spyder
network.host : non_loopback
node.data : false
path.data :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/data
path.data_with_cluster :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/data/roundtable-qa
path.home :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target
path.logs :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/logs
path.work :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/work
path.work_with_cluster :
/Users/jcook/Projects/Tracermedia/RoundTable/Development/roundtable/rest/target/work/roundtable-qa
transport.tcp.port : 9312-9349
[Spyder] [nep] creating index, cause [auto(index api)], shards [5]/[1],
mappings []
[Spyder] [nep] created and added to cluster_state
Failed to index [content], type [3c874dd6abdb414380f9f2f38fa84bfd], value:
{"key":"c202fffa-51ef-442b-8596-6af3a51f2f3b","ref":"7b6ec1fb-70a3-4f05-97de-b33335546a6f","locale":"ar","body":"
الخارجية
إستسلاماً مع
واشتدّت, ذلك تم بغزو إحتلال الحدود, ثم أما تصفح خصوصا
اليابانية. دنو فقامت بوزيرها للأراضي إذ, حرب صفحة والجنود ما, ان الى حقول
هجوم للصين. قد غينيا الشرقي العالم لكل, هو
Bahah","author":"James Cook","categories":[],"title":"Jabberwocky
fragment","reviewStatus":"Published","modified":"Jun 30, 2010 4:40:34
PM","created":"Jun 29, 2010 11:06:29 AM","description":"desc
updated","type":"roundTableContent","size":"131","mimetype":"text/html","_id":"3c874dd6abdb414380f9f2f38fa84bfd"}
org.elasticsearch.action.UnavailableShardsException: [nep][4] [2] shards,
[0] active : Timeout waiting for [1m], request: index
{[nep][content][3c874dd6abdb414380f9f2f38fa84bfd],
source[{"key":"c202fffa-51ef-442b-8596-6af3a51f2f3b","ref":"7b6ec1fb-70a3-4f05-97de-b33335546a6f","locale":"ar","body":"
الخارجية
إستسلاماً مع
واشتدّت, ذلك تم بغزو إحتلال الحدود, ثم أما تصفح خصوصا
اليابانية. دنو فقامت بوزيرها للأراضي إذ, حرب صفحة والجنود ما, ان الى حقول
هجوم للصين. قد غينيا الشرقي العالم لكل, هو
Bahah","author":"James Cook","categories":[],"title":"Jabberwocky
fragment","reviewStatus":"Published","modified":"Jun 30, 2010 4:40:34
PM","created":"Jun 29, 2010 11:06:29 AM","description":"desc
updated","type":"roundTableContent","size":"131","mimetype":"text/html","_id":"3c874dd6abdb414380f9f2f38fa84bfd"}]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$4.onTimeout(TransportShardReplicationOperationAction.java:401)
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:260)
at
org.elasticsearch.timer.TimerService$ThreadedTimerTask$1.run(TimerService.java:113)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
I assume the exception is triggered because Spyder is a non-data node which
did not discover Sanders, James which is the data node.
Is there anything wrong with my configuration, or perhaps my EC2
configuration has to open additional ports? I am assuming the ports will
have to be opened when I start to add additional instances, but this is a
single VM. I'm not sure what the potential reasons for a discovery failure
on EC2 might be.
Thanks,
jim