ECE deployment cannot connect filebeats to kibana

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi,

I have deployment the on-premise ECE, following the guide. Just a simple POC, but it seems the the licence will expire before I can get some real data in the system.
Best to then stick with Splunk :slight_smile:

3 server
ece-3a-01.ece-elastic.xyz
ece-3b-01.ece-elastic.xyz
ece-3c-01.ece-elastic.xyz
I have a wildcard DNS and Wildcard SSL certificate *.ece-elastic.xyz

I have deploy a single deployment kibana, no customization.
I have not modified or changed the default kibana deployment as deploy from ECE.

I installed 2 ubuntu 16.04 beats servers and installed filebeat-6.4.2-amd64.deb

I modified the filebeat.yml, but the setting seem a bit of a mystery and not well documented for this configuration.

I was told not to use the cloud.id: or cloud.auth as this is for the cloud version

Here are my setting
I have tried various combinations, but does any know what is should be pointed to.
eg
Elasticsearch
https://261f41b5d7114d2fb96c403bed80c148.ece-elastic.xyz:9243/
kibana
https://2f2122e03fe0499db11412a9e0b69b4a.ece-elastic.xyz:9243/

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]

hosts: ["ece-3a-01.ece-elastic.xyz:9343"]

protocol: "https"
username: "elastic"
password: "3VC933BDccAl8Q90iZo0yJg9"

2

this was with port :9343

ubuntu@filebeats-2:/etc/filebeat$ sudo filebeat setup --e
2018-10-16T01:13:50.719Z INFO instance/beat.go:544 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-10-16T01:13:50.720Z INFO instance/beat.go:551 Beat UUID: c3de4395-5173-4333-8a8a-b13e2ff374c9
2018-10-16T01:13:50.720Z INFO [beat] instance/beat.go:768 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "c3de4395-5173-4333-8a8a-b13e2ff374c9"}}}
2018-10-16T01:13:50.720Z INFO [beat] instance/beat.go:777 Build info {"system_info": {"build": {"commit": "e193f6d68b25b7ddbe3a3ed8d60bc07fea1ef800", "libbeat": "6.4.2", "time": "2018-09-26T12:42:46.000Z", "version": "6.4.2"}}}
2018-10-16T01:13:50.720Z INFO [beat] instance/beat.go:780 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.10.3"}}}
2018-10-16T01:13:50.721Z INFO [beat] instance/beat.go:784 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-16T00:11:53Z","containerized":false,"hostname":"filebeats-2","ips":["127.0.0.1/8","::1/128","192.168.201.19/24","fe80::f816:3eff:fe76:28e5/64"],"kernel_version":"4.4.0-135-generic","mac_addresses":["fa:16:3e:76:28:e5"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.5 LTS (Xenial Xerus)","major":16,"minor":4,"patch":5,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0,"id":"e947bef37f61461a800be9372cd93f9e"}}}
2018-10-16T01:13:50.721Z INFO [beat] instance/beat.go:813 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 26533, "ppid": 26532, "seccomp": {"mode":"disabled"}, "start_time": "2018-10-16T01:13:50.060Z"}}}
2018-10-16T01:13:50.722Z INFO instance/beat.go:273 Setup Beat: filebeat; Version: 6.4.2
2018-10-16T01:13:50.722Z INFO elasticsearch/client.go:163 Elasticsearch url: https://ece-3a-01.ece-elastic.xyz:9343
2018-10-16T01:13:50.723Z INFO pipeline/module.go:98 Beat name: filebeats-2
2018-10-16T01:13:50.723Z INFO elasticsearch/client.go:163 Elasticsearch url: https://ece-3a-01.ece-elastic.xyz:9343
2018-10-16T01:13:51.008Z ERROR elasticsearch/elasticsearch.go:214 Error connecting to Elasticsearch at https://ece-3a-01.ece-elastic.xyz:9343: Get https://ece-3a-01.ece-elastic.xyz:9343: net/http: HTTP/1.x transport connection broken: malformed HTTP status code "acceptable"
2018-10-16T01:13:51.008Z ERROR instance/beat.go:743 Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://ece-3a-01.ece-elastic.xyz:9343: Get https://ece-3a-01.ece-elastic.xyz:9343: net/http: HTTP/1.x transport connection broken: malformed HTTP status code "acceptable"]
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://ece-3a-01.ece-elastic.xyz:9343: Get https://ece-3a-01.ece-elastic.xyz:9343: net/http: HTTP/1.x transport connection broken: malformed HTTP status code "acceptable"]

3

this was with port :9243

ubuntu@filebeats-2:/etc/filebeat$ sudo filebeat setup --e
2018-10-16T01:32:48.571Z INFO instance/beat.go:544 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-10-16T01:32:48.571Z INFO instance/beat.go:551 Beat UUID: c3de4395-5173-4333-8a8a-b13e2ff374c9
2018-10-16T01:32:48.571Z INFO [beat] instance/beat.go:768 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "c3de4395-5173-4333-8a8a-b13e2ff374c9"}}}
2018-10-16T01:32:48.572Z INFO [beat] instance/beat.go:777 Build info {"system_info": {"build": {"commit": "e193f6d68b25b7ddbe3a3ed8d60bc07fea1ef800", "libbeat": "6.4.2", "time": "2018-09-26T12:42:46.000Z", "version": "6.4.2"}}}
2018-10-16T01:32:48.572Z INFO [beat] instance/beat.go:780 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.10.3"}}}
2018-10-16T01:32:48.573Z INFO [beat] instance/beat.go:784 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-16T00:11:53Z","containerized":false,"hostname":"filebeats-2","ips":["127.0.0.1/8","::1/128","192.168.201.19/24","fe80::f816:3eff:fe76:28e5/64"],"kernel_version":"4.4.0-135-generic","mac_addresses":["fa:16:3e:76:28:e5"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.5 LTS (Xenial Xerus)","major":16,"minor":4,"patch":5,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0,"id":"e947bef37f61461a800be9372cd93f9e"}}}
2018-10-16T01:32:48.573Z INFO [beat] instance/beat.go:813 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 26581, "ppid": 26580, "seccomp": {"mode":"disabled"}, "start_time": "2018-10-16T01:32:47.910Z"}}}
2018-10-16T01:32:48.574Z INFO instance/beat.go:273 Setup Beat: filebeat; Version: 6.4.2
2018-10-16T01:32:48.574Z INFO elasticsearch/client.go:163 Elasticsearch url: https://ece-3a-01.ece-elastic.xyz:9243
2018-10-16T01:32:48.575Z INFO pipeline/module.go:98 Beat name: filebeats-2
2018-10-16T01:32:48.576Z INFO elasticsearch/client.go:163 Elasticsearch url: https://ece-3a-01.ece-elastic.xyz:9243
2018-10-16T01:32:49.131Z ERROR elasticsearch/elasticsearch.go:214 Error connecting to Elasticsearch at https://ece-3a-01.ece-elastic.xyz:9243: 404 Not Found: {"ok":false,"message":"Unknown cluster."}
2018-10-16T01:32:49.132Z ERROR instance/beat.go:743 Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://ece-3a-01.ece-elastic.xyz:9243: 404 Not Found: {"ok":false,"message":"Unknown cluster."}]
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://ece-3a-01.ece-elastic.xyz:9243: 404 Not Found: {"ok":false,"message":"Unknown cluster."}]

4

ubuntu@filebeats-2:/etc/filebeat$ curl -sv '*' https://ece-3a-01.ece-elastic.xyz:9343

  • Rebuilt URL to: */
  • Trying ::1...
  • connect to ::1 port 80 failed: Connection refused
  • Trying 127.0.0.1...
  • connect to 127.0.0.1 port 80 failed: Connection refused
  • Failed to connect to * port 80: Connection refused
  • Closing connection 0
  • Rebuilt URL to: https://ece-3a-01.ece-elastic.xyz:9343/
  • Trying 10.243.196.19...
  • Connected to ece-3a-01.ece-elastic.xyz (10.243.196.19) port 9343 (#1)
  • found 148 certificates in /etc/ssl/certs/ca-certificates.crt
  • found 592 certificates in /etc/ssl/certs
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  •    server certificate verification OK

  •    server certificate status verification SKIPPED

  •    common name: *.ece-elastic.xyz (matched)

  •    server certificate expiration date OK

  •    server certificate activation date OK

  •    certificate public key: RSA

  •    certificate version: #3

  •    subject: CN=*.ece-elastic.xyz

  •    start date: Sat, 13 Oct 2018 06:54:11 GMT

  •    expire date: Fri, 11 Jan 2019 06:54:11 GMT

  •    issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3

  •    compression: NULL
  • ALPN, server did not agree to a protocol

GET / HTTP/1.1
Host: ece-3a-01.ece-elastic.xyz:9343
User-Agent: curl/7.47.0
Accept: /

  • Connection #1 to host ece-3a-01.ece-elastic.xyz left intact
    *â–’â–’â–’â–’â–’No acceptable header received.
5

9343 is the transport protocol port, can you try 9243 which is the HTTPS port? I believe beats uses that (and curl definitely does)

Sorry I only read the last message, reading the rest now

Alex

6

OK I see the problem:

(Note first that 9243 is the correct port as described above)

Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://ece-3a-01.ece-elastic.xyz:9243: 404 Not Found: {"ok":false,"message":"Unknown cluster."}]

The formats of the cluster URLs are of the form <clusterid>.<domain> eg if your cluster Id (listed in the overview page on the UI / returned from the create API call) is 261f41b5d7114d2fb96c403bed80c148 and as you said your domain prefix is *.ece-elastic.xyz then your cluster URL would look like https://261f41b5d7114d2fb96c403bed80c148.ece-elastic.xyz:9243

So the error you see above is expected since ece-3a-01 is not a cluster ID

Now ... currently your 3 hosts are eg ece-3*-01.ece-elastic.xyz so you need to make your wildcard DNS point *.ece-elastic.xyz to a round-robin of the 3 ECE hosts (assuming you have a proxy role on each of them)

Once you have done that then you can check the DNS setup works by doing eg host ANYTHING.ece-elastic.xyz (or dig) and it should return one of the 3 IPs of ece-3a-01, ece-3b-01 and ece-3c-01

(At this point you can also set the cname in the settings page of the UI to ece-elastic.xyz and it will auto-generate the correct URLs on the cluster overview pages)

Once you have confirmed that then you would just set eg:

hosts: ["https://261f41b5d7114d2fb96c403bed80c148.ece-elastic.xyz:9243"]

protocol: "https"
username: "elastic"
password: "3VC933BDccAl8Q90iZo0yJg9"

(note: I'm assuming your setup is internal - otherwise you should change all the passwords etc :slight_smile: )

and it should connect (note I used the cluster ID of elasticsearch not Kibana)

Does that make sense?

Sorry that all the network set-up is a pain (for a quick out-of-the box experience, we provide the ip.es.io service which just maps <IP>.ip.es.io to IP <IP> so you can route easily to a single node of an ECE install via <CLUSTERID>.<IP>.ip.es.io without having to mess about with DNS

Alex

7

Good news

From the ECE the deployment:
elasticsearch
https://261f41b5d7114d2fb96c403bed80c148.ece-elastic.xyz:9243
kibana
https://2f2122e03fe0499db11412a9e0b69b4a.ece-elastic.xyz:9243

With the above added to the filebeats.yml
sudo vi /etc/filebeat/filebeat.yml

setup.kibana:
#host: "localhost:5601"
host: "https://2f2122e03fe0499db11412a9e0b69b4a.ece-elastic.xyz:9243"

output.elasticsearch:
#Array of hosts to connect to.
#hosts: ["localhost:9200"]
hosts: ["261f41b5d7114d2fb96c403bed80c148.ece-elastic.xyz:9243"]
#Optional protocol and basic auth credentials.
protocol: "https"
username: "elastic"
password: "3VC933BDccAl8Q90iZo0yJg9"

sudo filebeat modules enable system
sudo filebeat setup --e
sudo service filebeat start

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.