Logstash 6.0 to elasticsearch 5.6.4 get a 503

Hi,
I upgraded logstash to 6.0 to take advantage of multiple pipelines and logstash receives a 503 from the already working elasticsearch. below is my config and conf
logstash.yml

path.data: /var/lib/logstash
path.logs: /var/log/logstash
path.settings: /etc/logstash
config.reload.automatic: true

syslog_filebeat_ingest.conf

input {
  beats {
    port => 5044
  }
}
filter {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}
output {
  elasticsearch {
   hosts => ["tf-d-ubu-elk-e-00.edrcore.com:9200"]
    index => "system-beats-%{+YYYY.MM.dd}"
  }
}

Error:

[2017-11-22T15:54:21,327][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://redacted.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '503' contacting Elasticsearch at URL 'http://redacted.com:9200/'"}

pipelines.yml

- pipeline.id: elb
  path.config: "/etc/logstash/conf.d/elb_ingest.conf"
  pipeline.workers: 1
- pipeline.id: logs_beat
  path.config: "/etc/logstash/conf.d/syslog_filebeat_ingest.conf"
  pipeline.workers: 2

it's not a connectivity problem

from my logstash host

curl .redacted.com:9200
{
  "name" : "redacted.com",
  "cluster_name" : "redacted",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "5.6.4",
    "build_hash" : "8bbedf5",
    "build_date" : "2017-10-31T18:55:38.105Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

thanks for any help !
cheers,
rob

I am not sure I understand. You are showing a file named beats_input.conf, which does not appear to be part of your pipelines.yml file. What do the files you are specifying in pipelines.yml look like?

yeah that was just a typo, i did not supply the elb conf because it requires heavy redacting. if you need it I will supply it

What is the state of your Elasticsearch cluster? Is there anything in the Elasticsearch logs?

I can reach it no problem

ubuntu@xx:~$ telnet xx 9200
Trying xx.xx.xx.xx...
Connected to xx.com.
Escape character is '^]'.
^C^]
telnet> quit 
Connection closed.
ubuntu@xx:~$ curl xx:9200
{
  "name" : "xx.com",
  "cluster_name" : "xx",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "5.6.4",
    "build_hash" : "8bbedf5",
    "build_date" : "2017-10-31T18:55:38.105Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

That is not the API I linked to.

@Christian_Dahlqvist thank you for your help i was under the assumption that if 9200 was open elastic is up. There was an error in my chef code that applied the prod config ( multi node) to my dev env that only has one node, so elastic was in fact not up.
thanks again
rob

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.