One cluster to monitor them all

ffknob · April 5, 2020, 10:07pm

Hi there

So I'm planning to deploy a single cluster to centralize logs and metrics from a group of different clients. The scope of this would be metrics from hosts and services (Metricbeat), log files (Filebeat), network (Packetbeat) and auditing info (Auditbeat and Winlogbeat). I'm also planning to monitor client's own Elasticsearch clusters.

I have a lot of questions on how to deploy this solution, most of them related to how to integrate components with different versions and also how to segment data from each client so one client can't see other client's data.

Monitoring Cluster: Elastic Cloud 7.6.2

Here are some of my doubts:

Should all my Beats' version be compatible with my cluster's? Say exaclty 7.6.2, or maybe I can install a future 7.6.3 release? What about 7.7.0 (remember my cluster is 7.6.2)?
Can a 7.6.2 Metricbeat get data from a 5.6 Elasticsearch cluster?
Are all Kibana Dashboards ready to handle missing Metricbeat data (suppose 5.6 doesn't have some attribute)?
I'm currently sending some tags and some root level custom fields (client.id, client.name, ...) from the Beats. I think I'll use the client.* to build some Dahsboards per client, maybe create some Kibana Spaces with a query for that field... Is that the way to go if I plan to give the client access to his data?

Thank you very much for all thoughts on this and stay safe!

warkolm · April 6, 2020, 6:43am

The safest route is to stay with the supported versioning - https://www.elastic.co/support/matrix#matrix_compatibility
Send it to? No, see above link
Depends what you mean by handle. It may show error/warnings if it can't find the data
That's a sane way if you have access to field level security

A single cluster may or may not be a good idea, but putting every single user into a single index is not. You should look to use ILM and then setup a structure like customername-metrics and customername-logs to keep the different data structures separate. This will also help with retention management, as logs are usually larger than metrics.

Using ILM with a size based roll over approach also means you don't end up oversharding, and customers can scale a lot easier without you having to worry about shards, and additionally, being able to provide them hot/warm/cold tiering and associated cost savings.

ffknob · April 6, 2020, 11:50am

At first I discarded changing index's name to avoid some complexity. But I see how that would benefit us. The only thing is, I'm not getting it correctly right now.

This is an example metricbeat.yml config:

name: "coyote.acme.co"
tags: ["ACME", "acme.co"]
fields_under_root: true
fields:
  client.id: 1
  client.name: "ACME"
  client.domain: "acme.co"

output.elasticsearch:
  hosts: ["https://...us-east-1.aws.found.io:..."]
  username: "monitor"
  password: "${MONITOR_PASSWORD}"
  compression_level: 3

setup.template:
  overwrite: true
  name: "metricbeat-acme"
  pattern: "metricbeat-acme-*"
  append_fields:
    - name: client.id
      type: short
    - name: client.name
      type: keyword
    - name: client.domain
      type: keyword

setup.ilm:
  overwrite: true
  policy_name: "metricbeat-acme"
  pattern: "{now/M{yyyy.MM}}-000001"

I then ran the following to set the index template:

./metricbeat setup --index-management -E 'output.elasticsearch.hosts: ["https://....us-east-1.aws.found.io:..."]' -E 'output.elasticsearch.username: "monitor"' -E 'output.elasticsearch.password: "${MONITOR_PASSWORD}"'

And then Metricbeat:

2020-04-06T08:28:10.996-0300	INFO	elasticsearch/client.go:757	Attempting to connect to Elasticsearch version 7.6.1
2020-04-06T08:28:11.201-0300	INFO	[license]	licenser/es_callback.go:50	Elasticsearch license: Platinum
2020-04-06T08:28:11.406-0300	INFO	[index-management]	idxmgmt/std.go:258	Auto ILM enable success.
2020-04-06T08:28:11.713-0300	INFO	[index-management]	idxmgmt/std.go:271	ILM policy successfully loaded.
2020-04-06T08:28:11.713-0300	INFO	[index-management]	idxmgmt/std.go:410	Set setup.template.name to '{metricbeat-7.6.2 {now/M{yyyy.MM}}-000001}' as ILM is enabled.
2020-04-06T08:28:11.713-0300	INFO	[index-management]	idxmgmt/std.go:415	Set setup.template.pattern to 'metricbeat-7.6.2-*' as ILM is enabled.
2020-04-06T08:28:11.713-0300	INFO	[index-management]	idxmgmt/std.go:449	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.2 {now/M{yyyy.MM}}-000001} as ILM is enabled.
2020-04-06T08:28:11.713-0300	INFO	[index-management]	idxmgmt/std.go:453	Set settings.index.lifecycle.name in template to {metricbeat-acme {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-04-06T08:28:11.917-0300	INFO	template/load.go:169	Existing template will be overwritten, as overwrite is enabled.
2020-04-06T08:28:12.301-0300	WARN	[cfgwarn]	template/template.go:148	EXPERIMENTAL: append_fields is used.
2020-04-06T08:28:12.305-0300	INFO	template/load.go:109	Try loading template metricbeat-7.6.2 to Elasticsearch
2020-04-06T08:28:12.942-0300	INFO	template/load.go:101	template with name 'metricbeat-7.6.2' loaded.
2020-04-06T08:28:12.942-0300	INFO	[index-management]	idxmgmt/std.go:295	Loaded index template.
2020-04-06T08:28:13.106-0300	INFO	[index-management]	idxmgmt/std.go:306	Write alias successfully generated.

Goals:

I want to avoid Dashboards explosion, so I don't wnat to set Kibana Dashboards for each client (changing setup.dashboards.index to metricbeat-ffknob and then running ./metricbeat setup --dashboards)
I want tobe able to see all clients' data in the same Dashboard
I want to be able to define different ILM policies for each client (that's why I'm willing to give this approach a try)

So, I think that what I want is:

ALIAS metricbeat-* -> READ metricbeat-acme-*, metricbeat-otherclient-*, ... / WRITE ???

Questions:

With ILM set, how to I make the Beat write to an specifc index (metricbeat-acme-*)? Here is what the log says:

Set setup.template.name to '{metricbeat-7.6.2 {now/M{yyyy.MM}}-000001}' as ILM is enabled.
Set setup.template.pattern to 'metricbeat-7.6.2-*' as ILM is enabled.

Thank you

ffknob · April 6, 2020, 1:03pm

I think I got it:

cloud: 
  id: "monitor:...=="
  auth: "monitor:${MONITOR_PASSWORD}"
 
output.elasticsearch:
  hosts: ["https://....us-east-1.aws.found.io:..."]
  username: "monitor"
  password: "${MONITORA_PASSWORD}"
  compression_level: 3
 
setup.template:
  overwrite: true
  name: "metricbeat"
  pattern: "metricbeat-*"
  append_fields:
    - name: client.id
      type: short
    - name: client.name
      type: keyword
    - name: client.domain
      type: keyword
 
setup.kibana:
  host: "....us-east-1.aws.found.io:..."
  protocol: "https"
 
setup.ilm:
  overwrite: true
  policy_name: "metricbeat-acme"
  pattern: "{now/M{yyyy.MM}}-000001"
  rollover_alias: "metricbeat-%{[agent.version]}-acme"

system · May 4, 2020, 3:03pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.