Logstash 6.0 to elasticsearch 5.6.4 get a 503

Elastic Stack Logstash
1

Hi,
I upgraded logstash to 6.0 to take advantage of multiple pipelines and logstash receives a 503 from the already working elasticsearch. below is my config and conf
logstash.yml

path.data: /var/lib/logstash
path.logs: /var/log/logstash
path.settings: /etc/logstash
config.reload.automatic: true

syslog_filebeat_ingest.conf

input {
  beats {
    port => 5044
  }
}
filter {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}
output {
  elasticsearch {
   hosts => ["tf-d-ubu-elk-e-00.edrcore.com:9200"]
    index => "system-beats-%{+YYYY.MM.dd}"
  }
}

Error:

[2017-11-22T15:54:21,327][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://redacted.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '503' contacting Elasticsearch at URL 'http://redacted.com:9200/'"}

pipelines.yml

- pipeline.id: elb
  path.config: "/etc/logstash/conf.d/elb_ingest.conf"
  pipeline.workers: 1
- pipeline.id: logs_beat
  path.config: "/etc/logstash/conf.d/syslog_filebeat_ingest.conf"
  pipeline.workers: 2

it's not a connectivity problem

from my logstash host

curl .redacted.com:9200
{
  "name" : "redacted.com",
  "cluster_name" : "redacted",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "5.6.4",
    "build_hash" : "8bbedf5",
    "build_date" : "2017-10-31T18:55:38.105Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

thanks for any help !
cheers,
rob

2

I am not sure I understand. You are showing a file named beats_input.conf, which does not appear to be part of your pipelines.yml file. What do the files you are specifying in pipelines.yml look like?

3

yeah that was just a typo, i did not supply the elb conf because it requires heavy redacting. if you need it I will supply it

4

What is the state of your Elasticsearch cluster? Is there anything in the Elasticsearch logs?

5

I can reach it no problem

ubuntu@xx:~$ telnet xx 9200
Trying xx.xx.xx.xx...
Connected to xx.com.
Escape character is '^]'.
^C^]
telnet> quit 
Connection closed.
ubuntu@xx:~$ curl xx:9200
{
  "name" : "xx.com",
  "cluster_name" : "xx",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "5.6.4",
    "build_hash" : "8bbedf5",
    "build_date" : "2017-10-31T18:55:38.105Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}
6

That is not the API I linked to.

7

@Christian_Dahlqvist thank you for your help i was under the assumption that if 9200 was open elastic is up. There was an error in my chef code that applied the prod config ( multi node) to my dev env that only has one node, so elastic was in fact not up.
thanks again
rob