Metricbeat Error while enrolling: fail to execute the HTTP POST request: Post https://kibana

[Ryan_Downey]

Elasticsearch 6.5.1
XPack enabled, TLS/SSL encryption enabled
Setup: Metricbeat-->Logstash-->Elasticsearch<--Kibana

I've been trying to enroll a metricbeat on a Windows Server 2016 box through the Central Managaement feature and I've had no luck getting it to register. Here are the troubleshooting steps that I've gone through so far.

1. Followed the thread here: https://github.com/elastic/beats/issues/9385
   -Deleted data/json
   .-Established a new token after deletion of data/json, beat type: metricbeat platform: Windows
   -Made sure I prefixed the cmd with -e -v -d "*"

PS C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64> .\metricbeat.exe -e -v -d "*" enroll https://abc.gov 4dcf5218c47740af8467729eaa93e25c
2019-01-04T09:49:54.576-0500    INFO    instance/beat.go:592    Home path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64] Config path: [C:\Program Files\Metricbeat\m
etricbeat-6.5.3-windows-x86_64] Data path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data] Logs path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_6
4\logs]
2019-01-04T09:49:54.578-0500    DEBUG   [beat]  instance/beat.go:629    Beat metadata path: C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data\meta.json
2019-01-04T09:49:54.597-0500    INFO    instance/beat.go:599    Beat UUID: 60ce4282-8666-480c-b088-c2208799c0ef
2019-01-04T09:49:54.597-0500    INFO    kibana/client.go:118    Kibana url: https://abc.gov:5601
Error while enrolling: fail to execute the HTTP POST request: Post https://abc.gov:5601/api/beats/agent/60ce4282-8666-480c-b088-c2208799c0ef: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)**

2. Also added .management-beats to action.auto_create_index in our ES yaml to which I can see a management-beats index when I click on Include System Indicies

3. Tried to use some of the other options from here: https://www.elastic.co/guide/en/beats/metricbeat/current/enroll-beats.html

PS C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64> .\metricbeat.exe -e -v -d "*" enroll https://abc.gov --username elastic --password stdin
2019-01-04T09:56:42.515-0500    INFO    instance/beat.go:592    Home path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64] Config path: [C:\Program Files\Metricbeat\m
etricbeat-6.5.3-windows-x86_64] Data path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data] Logs path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_6
4\logs]
2019-01-04T09:56:42.517-0500    DEBUG   [beat]  instance/beat.go:629    Beat metadata path: C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data\meta.json
2019-01-04T09:56:42.523-0500    INFO    instance/beat.go:599    Beat UUID: 60ce4282-8666-480c-b088-c2208799c0ef
Enter password:
2019-01-04T09:56:50.500-0500    INFO    kibana/client.go:118    Kibana url: https://abc.gov:5601
Error creating a new enrollment token: fail to execute the HTTP POST request: Post https://abc.gov:5601/api/beats/enrollment_tokens: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

4. Tried a different version with no luck:

PS C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64> .\metricbeat.exe -e -v -d "*" enroll https://abcgov --username elastic --password password [--force]
2019-01-04T09:59:25.329-0500    INFO    instance/beat.go:592  Home path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64] Config path: [C:\Program Files\Metricbeat\m
etricbeat-6.5.3-windows-x86_64] Data path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data] Logs path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_6
4\logs]
2019-01-04T09:59:25.330-0500    DEBUG   [beat]  instance/beat.go:629    Beat metadata path: C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data\meta.json
2019-01-04T09:59:25.336-0500    INFO    instance/beat.go:599    Beat UUID: 60ce4282-8666-480c-b088-c2208799c0ef
2019-01-04T09:59:25.337-0500    INFO    kibana/client.go:118    Kibana url: https://abc.gov:5601
Error while enrolling: fail to execute the HTTP POST request: Post https://abc.gov:5601/api/beats/agent/60ce4282-8666-480c-b088-c2208799c0ef: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
PS C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64>

4. Tried username and password verification

PS C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64> .\metricbeat.exe -e -v -d "*" enroll https://abc.gov --username kibana --password stdin
2019-01-04T10:13:14.221-0500    INFO    instance/beat.go:592    Home path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64] Config path: [C:\Program Files\Metricbeat\m
etricbeat-6.5.3-windows-x86_64] Data path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data] Logs path: [C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_6
4\logs]
2019-01-04T10:13:14.223-0500    DEBUG   [beat]  instance/beat.go:629    Beat metadata path: C:\Program Files\Metricbeat\metricbeat-6.5.3-windows-x86_64\data\meta.json
2019-01-04T10:13:14.228-0500    INFO    instance/beat.go:599    Beat UUID: 60ce4282-8666-480c-b088-c2208799c0ef
Enter password:
2019-01-04T10:13:20.911-0500    INFO    kibana/client.go:118    Kibana url: https://abc.gov:5601
Error creating a new enrollment token: fail to execute the HTTP POST request: Post https://kibana-qa.svc.ny.gov:5601/api/beats/enrollment_tokens: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

5. I checked this thread and had no luck: Central Management filebeat enrollment fail
   ***Update: I went through the steps here and it still didn't seem to work: http://www.sqlservermart.com/HowTo/Windows_Import_Certificate.aspx
   I'm running into the same issue. By "add the CA to the OS trust store" I assume that if I'm working on a Windows Server 2016 that it needs to be added somewhere in the MS truststore. Any info on how to do that? I've been reading documentation online but its not getting me anywhere. Any help would be appreciated.

Any help would be appreciated. For our environment using Central Management would be really useful so we'd like to get it working.

[pierhugues (Pier-Hugues Pellerin)]

@Ryan_Downey

net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The above error look more like a firewall issue, can you reach kibana using curl from that machine?

[Ryan_Downey]

Pier,

Sorry for my delay but I appreciate your response awhile back. I just got back around to troubleshooting this as some other projects took precedence over this one for the time being. I was able to work around this issue by pointing the autogenerated token link to a specific server within our Kibana cluster and remvoing the kiban.gov part. Our setup like many has a VIP that directs data to one of the three Kibana servers in the pool. So we have kibana-test.gov with abc.gov, def.gov and ghi.gov servers and once I took out the kibana-test.gov from the link and put in abc.gov with the username and password it worked. For this example, metricbeat on Windows Server 2016 pointing to RPM Linux Kibana, the below command worked.

 .\metricbeat.exe enroll -username elastic --password stdin --force http://abc.gov 14axxxxxxxxxxxxxxxxxxxxxxxxx

Saving a copy of current settings to metricbeat.yml.bak
Enrolled and ready to retrieve settings from Kibana

One of the other issues that popped up was that due to the fact that our Kibana servers sit behind the VIP I had to remove the https part and change it to http. The error message that I got with https was:

Error while enrolling: fail to execute the HTTP POST request: Post https://abc.gov:5601/api/beats/agent
/60ce4282-8666-480c-b088-c2208799c0ef: http: server gave HTTP response to HTTPS client

Overall removal of the link that Kibana generated and inputting a specific server within our DNS pool worked. Thanks for the help and hopefully this helps some others out as well. Enjoy the rest of your day.