DNS lookup failure | Failed to connect to backoff(async(tcp | no such host

Elastic Stack Beats
, , ,
1

dear friends, please help: I have several servers with beats, kibana, logstash and elasticsearch (these are not docker(!)). trying to send logs from metricbeat to elasticsearch via logstash. here is my metricbeat.yml config with elasticsearch output:

output.elasticsearch:
hosts: ["https://kibana-linux.elk.com:9200","https://datahot01.elk.com:9200","https://datahot02.elk.com:9200", "https: //datahot03.elk.com:9200", "https://datawarm01.elk.com:9200", "https://datawarm02.elk.com:9200", "https://datawarm03.elk.com: 9200", "https://datawarm04.elk.com:9200", "https://datawarm05.elk.com:9200"]
protocol: "https"
username: "remote_monitoring_user"
password: "PASSWORD"
ssl.certificate_authorities: ["/etc/metricbeat/elasticsearch-ca.pem"]

with this output, the logs are transferred to elasticsearch without any problems. but now when I want to send logs to elasticsearch via logstash I use the following logstash.yml config:

node.name: node-logstash03
path.data: /opt/logstash/data
path.logs: /opt/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "PASSWORD"
xpack.monitoring.elasticsearch.hosts: ["https://kibana-linux.elk.com:9200","https://datahot01.elk.com:9200","https://datahot02.elk.com: 9200", "https://datahot03.elk.com:9200", "https://datawarm01.elk.com:9200", "https://datawarm02.elk.com:9200", "https:// datawarm03.elk.com:9200", "https://datawarm04.elk.com:9200", "https://datawarm05.elk.com:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/etc/logstash/elasticsearch-ca.pem"
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate

and my /etc/logstash/conf.d/beats_intra.conf

input {
  beat {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/logstash/ca.crt"
    ssl_key => "/etc/logstash/ca.pkcs8.key"
    ssl_verify_mode => "peer"
    ssl_certificate_authorities => "/etc/logstash/elasticsearch-ca.pem"
  }
}

output {
  elasticsearch {
    hosts => ["https://kibana-linux.elk.com:9200","https://datahot01.elk.com:9200","https://datahot02.elk.com:9200", "https ://datahot03.elk.com:9200", "https://datawarm01.elk.com:9200", "https://datawarm02.elk.com:9200", "https://datawarm03.elk.com :9200", "https://datawarm04.elk.com:9200", "https://datawarm05.elk.com:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    cacert => ["/etc/logstash/elasticsearch-ca.pem"]
    user => "elastic"
    password => "PASSWORD"
  }
}

metricbeat doesn't send data to logstash because it gets errors:

{"log.level":"error","@timestamp":"2022-08-28T22:02:56.020+0300","log.logger":"publisher_pipeline_output","log.origin":{"file. name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(async(tcp://[https://logstash03.elk.com:5044] :5044)): lookup https://logstash03.elk.com:5044: no such host","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2022-08-28T22:34:23.657+0300","log.logger":"transport","log.origin":{"file. name":"transport/tcp.go","file.line":52},"message":"DNS lookup failure "https://logstash03.elk.com:5044": lookup https://logstash03 .elk.com:5044: no such host","service.name":"metricbeat","ecs.version":"1.6.0"}

hostname logstash03.elk.com is in DNS on the managed server and I can telnet from the metricbeat server to logstash03.elk.com 5044

telnet logstash03.elk.com 5044
Trying 10.112.1.169...
Connected to logstash03.elk.com.
Escape character is '^]'.

there is a connection. I don't understand why there are errors...

my ELK version 8.3.3 (all components ELK Stack similar version 8.3.3)

2

Hi @kurdit

First, you did not show the output.logstash section in your metricbeats.yml So it's hard for us to help debug.

To me it looks like you put a

https://logstash03.elk.com:5044

In the output.logstash

When it should be

logstash03.elk.com:5044

Since that's not an HTTP connection is trying to DNS resolve that whole host with https:// as part of the host name when it should just be the host and the port. Give that a try. Let us know

3

hi! sorry i was wrong

here is my config

output.logstash:
  hosts: ["https://logstash03.elk.com:5044"]
  ssl.certificate_authorities: ["/etc/metricbeat/elasticsearch-ca.pem"]
  ssl.certificate: "/etc/metricbeat/certs/ca.crt"
  ssl.key: "/etc/metricbeat/certs/ca.key"

I will try your suggestion now

4

Right ... should be

hosts: ["logstash03.elk.com:5044"]

not https:// it is a tcp connection not https

5

I can't believe your recommendation helped! I spent many hours trying to solve this error. thank you for your help. now metricbit can't connect with logstash

{"log.level":"info","@timestamp":"2022-08-29T01:15:48.506+0300","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(async(tcp://logstash03.elk.com:5044)) with 6 reconnect attempt(s)","service.name":"metricbeat","ecs.version":"1.6.0"}

and

{"log.level":"error","@timestamp":"2022-08-29T01:18:30.498+0300","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(async(tcp://logstash03.elk.com:5044)): read tcp 10.112.1.166:20040->10.112.1.169:5044: read: connection reset by peer","service.name":"metricbeat","ecs.version":"1.6.0"}

I have to deal with this further (I think it's a certificate issue) thanks for the help!

6

Also your logstash "Passthrough" should look like this in case you use module(s) that has a pipeline.

Also since you are using 8.x (8.3.3) the stack uses data streams below config should work correct with datastreams tthe action => "create" is needed.

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" 
      pipeline => "%{[@metadata][pipeline]}" 
    }
  } else {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"
    }
  }
}
1 Like
7

You should only need the CA from the logstash server... not sure if you used the same CA as elasticsearch if so that should be correct otherwise it needs to be the CA you used to secure the logstash endpoint

1 Like
8

that is, should I make a separate CA on the server with logstash and issue a certificate for the client (metricbit)?

9

No a CA is a CA as long as you use the same. CA to generate the SSL logstash certificate It should be fine.

I Don't know how you created your logstash SSL certificate. Yes this is all cert stuff. It's always painful.

1 Like
10

Docs

Here and here

1 Like
11

thank u so mach :grinning:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.