Retrying failed action with response code: 503

Elastic Stack Elasticsearch
1

I started both Elasticsearch(1.6.) and Logstash(1.5.1), but ES seems not working.
Here is the conf of LS:

input {
stdin { }
}
output {
elasticsearch
{ host => "zslesvm09:9300" } # it is the same if removing ":9300"
stdout { }
}

The output from LS:

INFO: [logstash-lozlnx23-53291-11306] started
Logstash startup completed
aaaaaaaaaaaa
retrying failed action with response code: 503 {:level=>:warn}
2015-06-30T15:16:12.805Z lozlnx23 aaaaaaaaaaaa
retrying failed action with response code: 503 {:level=>:warn}
retrying failed action with response code: 503 {:level=>:warn}
retrying failed action with response code: 503 {:level=>:warn}
too many attempts at sending event. dropping: 2015-06-30T15:16:12.805Z lozlnx23 aaaaaaaaaaaa {:level=>:error}

ES uses the default configuration and output:
....
[2015-06-30 11:16:16,219][INFO ][cluster.service ] [Living Eraser] added {[logstash-lozlnx23-53291-11306][10jEd5FUTm2B7X8a4x-L2w][lozlnx23][inet[/9.26.176.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[Ever][_qEXm0RvRKaOqkEEGcAuvQ][mako.torolab.ibm.com][inet[/9.26.54.179:9300]]])

....

Thanks,

2

What is the response if you curl zslesvm09:9200.

3

Sorry, the ES version is 1.5.2 (1.6.0 seems working),

curl zslesvm09:9200
{
"status" : 200,
"name" : "Citizen V",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.5.2",
"build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
"build_timestamp" : "2015-04-27T09:21:06Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}

4

A 503 from ES would usually indicate it's not available.

Next time you get that response I'd check _cat/health.

5

I found the problem, here is the solution:

  1. on ES config, need to define the cluster name (default is "elasticsearch" )
  2. on LS config file, add cluster definition as the above the example:
    output {
    elasticsearch
    { host => "zslesvm09:9300"
    cluster => <cluster_name>} # cluster_name defined @ ES
    stdout { }
    }

"cluster" is introduced by the latest LS 1.5.2.

Before we use the default definition that makes the LS cannot find the correct ES.

Thanks.

6

The cluster config option has been around for quite some time.