Question about setting up beats on multiple hosts

Elastic Stack Beats
1

Hi,

We have a 4-machine ES and Kibana cluster and I'm now setting up various beats.

For starters, I set up metricbeat on host1 and enabled nginx metricbeat that grabs metrics from host1, host2, host3, host4.

This created a single host1 entry in Metrics, but I can then query nginx metrics data for all 4 hosts.

So to get the other 3 hosts to show up in Metrics, I presume I need to enable metricbeat on those hosts as well. I was going to copy the same configs, but then realized that then all 4 hosts will monitor all 4 nginx servers. Would this situation create 4x the data in nginx metricbeat or would it somehow intelligently dedupe it and figure out that multiple hosts are querying the same servers for nginx data?

If this approach is not good, do you recommend just having each host query its own nginx and have a one-to-one relationship rather than one-to-all?

Thank you.

2

I would recommend a 1-1 relationship — 1 Metricbeat per nginx host, querying the nginx data from localhost. This has the benefit of resiliency - if one host goes down, you still get data about other hosts. It also has the slight benefit of performance - Metricbeat is talking to nginx over the loopback interface.

In general Beats are considered "lightweight agents", intended to run along side the service they are monitoring.

Shaunak

3

Thanks, that's what I figured. So there would be a ton of duplicates otherwise, right?

I need to figure out how to elegantly query the local server without hardcoding the server hostname so that the metricbeat configs can be easily copied across servers without the fear of overwriting the per-host hostnames.

For nginx, localhost/nginx_status goes to the first hosted domain (say, www.oursite.com), which is behind SSL, and the SSL check fails, even if I pass in a header for "Host: www.oursite.com". For curl, I could pass -k, but I'm not sure how to disable strict SSL key verification for metricbeat.

For memcached and other services, they only listen on 192.168.* for security purposes, so they can't be accessed with "localhost" - only the IP or hostname.

I tried using {$host} in the YAML configs as part of the strings (per https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format-env-vars.html) (something like hosts: ['https://{$host}.oursite.com'], but none of it worked.

Any advice for using variables inside configs when they're part of the strings (I didn't find string concatenation features in YAML)? nginx, apache, and many other software packages have this figured out. Even Elasticsearch itself kind of makes it work, but Kibana and Beat seem to be lacking.

Thanks.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.