# Filebeat unable to connect to ECE **URL:** \ **Category:** Elastic Cloud Enterprise (ECE)\ **Created:** [February 19, 2020, 10:59am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941 "2020-02-19T10:59:48Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 19, 2020, 10:59am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/1 "2020-02-19T10:59:48Z")
Hi all, I have installed filebeat to send logs to a production ECE deployment. Filebeat was installed successfully and I can see it is harvesting files, however, it is unable to ship them to ECE. These are the logs: > Feb 19 14:52:49 hrc-ece-centos74 filebeat[47017]: 2020-02-19T14:52:49.148+0400 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch([https://4f02406a0dashdiasd33a23725be1f.dns:9243](https://4f02406a0dashdiasd33a23725be1f.dns:9243))): Get [https://4f02406a0dashdiasd33a23725be1f.dns:9243](https://4f02406a0dashdiasd33a23725be1f.dns:9243): net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) > Feb 19 14:52:49 ece-centos74 filebeat[47017]: 2020-02-19T14:52:49.149+0400 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch([https://4f02406a0dashdiasd33a23725be1f.dns:9243](https://4f02406a0dashdiasd33a23725be1f.dns:9243))) with 75 reconnect attempt(s) > Feb 19 14:52:49 ece-centos74 filebeat[47017]: 2020-02-19T14:52:49.149+0400 INFO [publisher] pipeline/retry.go:196 retryer: send unwait-signal to consumer > Feb 19 14:52:49 ece-centos74 filebeat[47017]: 2020-02-19T14:52:49.149+0400 INFO [publisher] pipeline/retry.go:198 done > Feb 19 14:52:49 ece-centos74 filebeat[47017]: 2020-02-19T14:52:49.149+0400 INFO [publisher] pipeline/retry.go:173 retryer: send wait signal to consumer > Feb 19 14:52:49 ece-centos74 filebeat[47017]: 2020-02-19T14:52:49.149+0400 INFO [publisher] pipeline/retry.go:175 done This is my filebeat.yml file ``` ########Filebeat Configuration Example ############### #=========================== Filebeat inputs ============================= filebeat.inputs: - type: log enabled: true paths: - var/log/messages #==================== Elasticsearch template setting ========================== setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false output.elasticsearch.index: "test-%{[agent.version]}-%{+yyyy.MM.dd}" setup.template.name: "test-case" setup.template.pattern: "test-*" #================================ Outputs ===================================== #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["4f02406a0d9a3yfhdsufdsf23725be1f.dns:9243"] #index: "test-%{[agent.version]}-%{+yyyy.MM.dd}" # Protocol - either `http` (default) or `https`. protocol: "https" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "elastic" password: "changeme" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" ``` Any help is greatly appreciated. Thanks, Nhung ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [February 19, 2020, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/2 "2020-02-19T14:49:48Z")
Does `curl -k -u elastic:changeme 'https://$CLUSTER_ID.dns:9243'` from the same host work? If not what does it return (`-vvv`) My guess would be that there's a firewall or something in the way based on `Client.Timeout exceeded while awaiting headers` ... timeout would indicate that something is blocking the connection ---
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 20, 2020, 9:34am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/3 "2020-02-20T09:34:37Z")
Thanks for your help @Alex_Piggott 🙂 This is the response: ``` [root@ece-centos74 bin]# curl -k -u elastic:changeme 'https://$CLUSTER_ID.dns:9243' curl: (7) Failed connect to $CLUSTER_ID.dns:9243; Connection timed out [root@hrc-ece-centos74 bin]# curl -k -u elastic:changeme 'https://$CLUSTER_ID.dns:9243' -vvv * About to connect() to $CLUSTER_ID.dns:9243 (#0) * Trying $IP_ADDRESS... ``` I have resolved the issue now. The host was trying to connect to an IP it doesn't have access to. I've put in a manual entry to my `etc/hosts` file which routes the correct IP to my DNS entry. After doing so, ``` [root@hrc-ece-centos74 ~]# curl -k -u elastic:changeme 'https://$CLOUD_ID.dns:9243' -vvv * About to connect() to CLOUD_ID.dns port 9243 (#0) * Trying $IP_ADDRESS... * Connected to $CLOUD_ID.dns ($IP_ADDRESS) port 9243 (#0) ``` ---
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 23, 2020, 1:05pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/4 "2020-02-23T13:05:46Z")
Hi @Alex_Piggott, I am now getting these error messages: > Feb 23 16:55:42 ece-centos74 filebeat: 2020-02-23T16:55:42.524+0400#011ERROR#011pipeline/output.go:100#011Failed to connect to backoff(elasticsearch(https://$CLOUD\_ID.dns:9243)): Get https://$CLOUD\_ID.dns:9243: x509: certificate signed by unknown authority > > Feb 23 16:55:42 ece-centos74 filebeat: 2020-02-23T16:55:42.524+0400#011INFO#011pipeline/output.go:93#011Attempting to reconnect to backoff(elasticsearch(https://$CLOUD\_ID.dns:9243)) with 8 reconnect attempt(s) > > Feb 23 16:55:42 ece-centos74 filebeat: 2020-02-23T16:55:42.524+0400#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:733#011ES Ping(url=[https://4f02406a0d9a4181a44233a23725be1f.10.60.34.13:9243](https://4f02406a0d9a4181a44233a23725be1f.10.60.34.13:9243)) > > Feb 23 16:55:42 centos74 filebeat: 2020-02-23T16:55:42.524+0400#011INFO#011[publisher]#011pipeline/retry.go:196#011retryer: send unwait-signal to consumer > > Feb 23 16:55:42 hrc-ece-centos74 filebeat: 2020-02-23T16:55:42.524+0400#011INFO#011[publisher]#011pipeline/retry.go:198#011 done > > Feb 23 16:55:42 ece-centos74 filebeat: 2020-02-23T16:55:42.524+0400#011INFO#011[publisher]#011pipeline/retry.go:173#011retryer: send wait signal to consumer > > Feb 23 16:55:42 ece-centos74 filebeat: 2020-02-23T16:55:42.524+0400#011INFO#011[publisher]#011pipeline/retry.go:175#011 done > > Feb 23 16:55:42 ece-centos74 filebeat: 2020-02-23T16:55:42.537+0400#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:737#011Ping request failed with: Get https://$CLOUD\_ID.dns:9243: x509: certificate signed by unknown authority I am still able to curl successfully and in ECE, my certificates are fine. The certificates are single sign-on and I created them with elasticsearch-certutil on a different machine, NOT on one of my ECE machines. Thanks, Nhung ---
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 23, 2020, 2:31pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/5 "2020-02-23T14:31:33Z")
This is really interesting now The configuration below produces the above logs where I am getting a certificate error. ``` ###################### Filebeat Configuration Example ######################### #=========================== Filebeat inputs ============================= filebeat.inputs: - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\* - /var/log/elasticsearch/elasticsearch-*.json #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] index: "filebeat-test" setup.template: enabled: false # Protocol - either `http` (default) or `https`. protocol: "https" ssl.verification_mode: "none" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "elastic" password: "changeme" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["https://$CLOUD_ID.dns:9243"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat ``` This configuration below DOES NOT produce the above logs and I can see the index in elasticsearch: > ###################### Filebeat Configuration Example ######################### > #=========================== Filebeat inputs ============================= > filebeat.inputs: > - type: log > # Change to true to enable this input configuration. > enabled: true > # Paths that should be crawled and fetched. Glob based paths. > paths: > - /var/log/_.log > #- c:\programdata\elasticsearch\logs\* > - /var/log/elasticsearch/elasticsearch-_.json > #================================ Outputs ===================================== > # Configure what output to use when sending the data collected by the beat. > #-------------------------- Elasticsearch output ------------------------------ > output.elasticsearch: > # Array of hosts to connect to. > hosts: ["localhost:9200"] > #index: "filebeat-test" > #setup.template: > #enabled: false > # Protocol - either `http` (default) or `https`. > protocol: "https" > ssl.verification\_mode: "none" > # Authentication credentials - either API key or username/password. > #api\_key: "id:api\_key" > username: "elastic" > password: "changeme" > #----------------------------- Logstash output -------------------------------- > #output.logstash: > # The Logstash hosts > #hosts: ["https://$CLOUD\_ID.dns:9243"] > # Optional SSL. By default is off. > # List of root certificates for HTTPS server verifications > #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"] > # Certificate for SSL client authentication > #ssl.certificate: "/etc/pki/client/cert.pem" > # Client Certificate Key > #ssl.key: "/etc/pki/client/cert.key" > #================================ Logging ===================================== > # Sets log level. The default log level is info. > # Available log levels are: error, warning, info, debug > #logging.level: debug > # At debug level, you can selectively enable logging only for some components. > # To enable all selectors use ["_"]. Examples of other selectors are "beat", > # "publish", "service". > #logging.selectors: ["_"] > logging.level: info > logging.to\_files: true > logging.files: > path: /var/log/filebeat Have you seen this issue before? Thanks, Nhung ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [February 24, 2020, 2:12pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/6 "2020-02-24T14:12:47Z")
Are those 2 config files both definitely what you used? The top one appears to have no `output` configured at all, while the second has `https` but `hosts: ["localhost:9200"]` (which is the HTTP port) And in both files you have `output.logstash:` pointing to `hosts: ["https://$CLOUD_ID.dns:9243"]` but commented out? ---
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 25, 2020, 4:40am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/7 "2020-02-25T04:40:18Z")
Hi @Alex_Piggott, Sorry, that was my fault This filebeat.yml file (the only difference is that I am trying to change the index name) gives me the error messages above - `x509: certificate signed by unknown authority` ``` ###################### Filebeat Configuration Example ######################### #=========================== Filebeat inputs ============================= filebeat.inputs: - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\* - /var/log/elasticsearch/elasticsearch-*.json #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["https://$CLOUD_ID.dns:9243"] index: "filebeat-test" setup.template: name: "filebeat" pattern: "filebeat*" enabled: "false" # Protocol - either `http` (default) or `https`. protocol: "https" ssl.verification_mode: "none" ssl.enabled: "true" ssl.certificate_authorities: ["/usr/share/elasticsearch/ca/ca/ca.crt"] ssl.certificate: "/usr/share/elasticsearch/ca/instance/instance.crt" ssl.key: "/usr/share/elasticsearch/ca/instance/instance.key" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "elastic" password: "changeme" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat ``` This filebeat.yml file (the only difference is that I am using the default index name) gives me NO error messages and I can see the index in elasticsearch plus the logs ``` ###################### Filebeat Configuration Example ######################### #=========================== Filebeat inputs ============================= filebeat.inputs: - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\* - /var/log/elasticsearch/elasticsearch-*.json #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["https://$CLOUD_ID.dns:9243"] #index: "filebeat-test" #setup.template: #name: "filebeat" #pattern: "filebeat*" #enabled: "false" # Protocol - either `http` (default) or `https`. protocol: "https" ssl.verification_mode: "none" ssl.enabled: "true" ssl.certificate_authorities: ["/usr/share/elasticsearch/ca/ca/ca.crt"] ssl.certificate: "/usr/share/elasticsearch/ca/instance/instance.crt" ssl.key: "/usr/share/elasticsearch/ca/instance/instance.key" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "elastic" password: "changeme" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat ``` Thanks, Nhung ---
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 25, 2020, 4:43am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/8 "2020-02-25T04:43:56Z")
How I created my certificates: `/usr/share/elasticsearch/bin/elasticsearch- **cert** util cert --pem --dns '*.$ECE_IP_[1.ip.es.io](http://1.ip.es.io/)' --dns '*.$ECE_IP_[2.ip.es.io](http://2.ip.es.io/)' --dns '*.$ECE_IP_[3.ip.es.io](http://3.ip.es.io/)' --ip '$ECE_IP_1 --ip '$ECE_IP_2' --ip '$ECE_IP_3' --dns '*.$ECE_[LB.ip.es.io](http://lb.ip.es.io/)' -ip '$PUBLIC_IP_USED_TO_SEND_LOGS' --dns '*.$ECE_LB'` I get three outputs: - ca.crt - CA cert which is signed ( `All certificates that are generated by this command are signed by a CA` - [https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html)) - instance.key - private key - instance.crt - certificate I concatenated them and uploaded it successfully into the Cloud UI I then copied the certificates into this server I am trying to send logs from. The path to the certificates are referenced in the filebeat.yml ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [February 25, 2020, 2:56pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/9 "2020-02-25T14:56:44Z")
Oh I think I see ... all those `ssl.*` fields are under `setup.template` in the YAML, so don't get applied? When you comment out `setup.template` then they "switch" to being under `output.elasticsearch` instead and get applied?! Try moving `setup.template` to immediately above the "Logstash output" header line and comment things back in ---
**Author:** ![ntran](https://avatars.discourse-cdn.com/v4/letter/n/73ab20/32.png) [@ntran](https://discuss.elastic.co/u/ntran)\ **Post date:** [February 26, 2020, 4:20am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/10 "2020-02-26T04:20:04Z")
THANK YOU SO MUCH @Alex_Piggott!!! For those who had the same issue as me, this was my final filebeat.yml file ``` ###################### Filebeat Configuration Example ######################### #=========================== Filebeat inputs ============================= filebeat.inputs: - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\* - /var/log/elasticsearch/elasticsearch-*.json #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["https://$CLOUD_ID.dns:9243"] index: "filebeat-test" # Protocol - either `http` (default) or `https`. protocol: "https" ssl.verification_mode: "none" ssl.enabled: "true" ssl.certificate_authorities: ["/usr/share/elasticsearch/ca/ca/ca.crt"] ssl.certificate: "/usr/share/elasticsearch/ca/instance/instance.crt" ssl.key: "/usr/share/elasticsearch/ca/instance/instance.key" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "elastic" password: "changeme" setup.template: name: "filebeat" pattern: "filebeat*" enabled: "false" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2020, 4:20am UTC](https://discuss.elastic.co/t/filebeat-unable-to-connect-to-ece/219941/11 "2020-03-11T04:20:13Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.