I ran in to a few issues using the Beats Central Management feature so I figured I'd put my troubleshooting steps out there as it seems like some others might be experiencing the same issues. We have 6.5.4 ES/Kibana/Logstash nodes with a Metricbeat (MB) reporting from one of our other servers to Logstash and then to ES. Note that our MB was placed on Windows Server 2016 so some of the following commands may be Powershell specific.
So to make your troubleshooting a little easier its preferable that you start with a known working configuration. For us we had an unencrypted MB pointing to Logstash and then encrypted TLS communication from Logstash to ES. Data was populating with date math for our MB so we had a known working config. The issue we had was that once I tried to enroll our beat from abc.gov server no data would reach our ES cluster.
Before you do any of this I'd recommend saving your working MB/FB yaml and logstash config files so that you can go back if you need to. Although for MB it saves the old yaml as metricbeat.yml.bak
First thing I found was that I needed to update our Logstash pipeline. I'm not sure if I missed something but we had a "working" pipeline at the time so I didnt think I need to update anything Logstash wise but I did. Maybe user error but I ended up going to Management-->Pipelines-->Create Pipeline
- Pipeline ID: main ///I used the same name as the already exiting pipeline and it worked
- Description: Testing Metricbeat
- Pipeline: !!!Note I copied and pasted the text of the logstash.conf file right into this portion!!!
- ***All other settings I left as default.***
Working logstash.conf file that I put in:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["https://abc.gov:9200",
"https://def.gov:9200","https://ghi.gov:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "logstash_internal"
password => "xxxxxxxxxxxxxxxxx"
ssl => true
#ssl_certificate_verification => false
cacert => "/etc/logstash/abc/ca/ca.crt"
}
}
I may have restarted logstash at this point but I dont remember. If data is still populating correctly then it should be good.
After the Logstash update the goal was to enroll our MB through Central Management (CM).
- STOP the beat that you want to setup with CM
- Remove the service if MB was setup to run as a service. I found this link helpful. We were working with MB on a Windows platform. Other installations will obviously be different and may need to be handeled differently. https://www.itprotoday.com/compute-engines/how-do-i-delete-service
- Close out regedit.
- Close out Services window if needed.
- In Kibana click on Management-->Central Management-->Enroll Beats.
- Beat Type: Metricbeat
- Platform: Windows
- On the host where your Metricbeat is installed run: Note: copy the link that populates here to use in Powershell
- Open up Powershell and run as admin.
- Navigate to the appropriate MB folder and run the link that you copied from Kibana but make sure you add .\ to the front of it in order for the command to run properly.
Hopefully this works! If not read the error logs or whatever prints out after you run the command to troubleshoot. Some troubleshooting links and directions can be found below. If you have a DNS name or IP address that goes to a pool of Kibana nodes you might need to check out #2.
-
If you have encryption enabled your probably going to have to pass a username and password through. In our case we used .\metricbeat.exe enroll http://xyz.gov:5601 70f4b584e8024b96b682c46125a8d81a --username elastic --password stdin --force which worked on our sandbox but didnt work in our larger QA environment due to the pool of DNS after our VIP.
-
The other issues we had in QA can be found and solved here:
Metricbeat Error while enrolling: fail to execute the HTTP POST request: Post https://kibana
Once the beat enrolled we clicked Next
- Create a Tag Name and color, for this beat we went with Windows-Sandbox and a Windows blue color.
- Then click Add Configuration Block:
a. Select Metricbeat Module for the Type
b. Enter a description if needed
c. Select the module you want to get metrics from in our case we used the basic System module.
d. Click Add Configuration Block again since we have Logstash running
e. Select Output
f. Enter a description if needed
g. Select the appropriate Output Type in this case its Logstash
h. Enter the DNS or IP of the Logstash host that you have, ie logstash-test.gov
i. Enter username and password if needed then click save.
If the Config Status in Kibana still doesn't change to OK check out the logs. I found in both cases of the beat enrollment process that the meta.json file, which in our Windows environment can be found in the ProgramData-->metribeat folder, had not been changed to the UUID of the enrolled beat. So the error messages were something like "error retriving new configurations, will use cached ones: Beat "1570704b-6f6d-4ada-801e-a16db6fc02c8" not found" which means the meta.json file is looking for an old beat UUID.
- To fix this go back to Kibana--> Management-->Central Management
a. Click on the Beat Name you need the UUID of
b. Replace the meta.json UUID with the correct one if they are different.
c. Save the new meta.json file.
d. Restart MB or FB
After this the beat enrolled, config status changed to OK and data populated correctly into ES. It was quite the process.
Hope this helps anyone running into issues with their Metricbeat or Filebeat--->Logstash---Elasticsearch issues.