So I'm planning to deploy a single cluster to centralize logs and metrics from a group of different clients. The scope of this would be metrics from hosts and services (Metricbeat), log files (Filebeat), network (Packetbeat) and auditing info (Auditbeat and Winlogbeat). I'm also planning to monitor client's own Elasticsearch clusters.
I have a lot of questions on how to deploy this solution, most of them related to how to integrate components with different versions and also how to segment data from each client so one client can't see other client's data.
Monitoring Cluster: Elastic Cloud 7.6.2
Here are some of my doubts:
Should all my Beats' version be compatible with my cluster's? Say exaclty 7.6.2, or maybe I can install a future 7.6.3 release? What about 7.7.0 (remember my cluster is 7.6.2)?
Can a 7.6.2 Metricbeat get data from a 5.6 Elasticsearch cluster?
Are all Kibana Dashboards ready to handle missing Metricbeat data (suppose 5.6 doesn't have some attribute)?
I'm currently sending some tags and some root level custom fields (client.id, client.name, ...) from the Beats. I think I'll use the client.* to build some Dahsboards per client, maybe create some Kibana Spaces with a query for that field... Is that the way to go if I plan to give the client access to his data?
Thank you very much for all thoughts on this and stay safe!
Depends what you mean by handle. It may show error/warnings if it can't find the data
That's a sane way if you have access to field level security
A single cluster may or may not be a good idea, but putting every single user into a single index is not. You should look to use ILM and then setup a structure like customername-metrics and customername-logs to keep the different data structures separate. This will also help with retention management, as logs are usually larger than metrics.
Using ILM with a size based roll over approach also means you don't end up oversharding, and customers can scale a lot easier without you having to worry about shards, and additionally, being able to provide them hot/warm/cold tiering and associated cost savings.
At first I discarded changing index's name to avoid some complexity. But I see how that would benefit us. The only thing is, I'm not getting it correctly right now.
2020-04-06T08:28:10.996-0300 INFO elasticsearch/client.go:757 Attempting to connect to Elasticsearch version 7.6.1
2020-04-06T08:28:11.201-0300 INFO [license] licenser/es_callback.go:50 Elasticsearch license: Platinum
2020-04-06T08:28:11.406-0300 INFO [index-management] idxmgmt/std.go:258 Auto ILM enable success.
2020-04-06T08:28:11.713-0300 INFO [index-management] idxmgmt/std.go:271 ILM policy successfully loaded.
2020-04-06T08:28:11.713-0300 INFO [index-management] idxmgmt/std.go:410 Set setup.template.name to '{metricbeat-7.6.2 {now/M{yyyy.MM}}-000001}' as ILM is enabled.
2020-04-06T08:28:11.713-0300 INFO [index-management] idxmgmt/std.go:415 Set setup.template.pattern to 'metricbeat-7.6.2-*' as ILM is enabled.
2020-04-06T08:28:11.713-0300 INFO [index-management] idxmgmt/std.go:449 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.2 {now/M{yyyy.MM}}-000001} as ILM is enabled.
2020-04-06T08:28:11.713-0300 INFO [index-management] idxmgmt/std.go:453 Set settings.index.lifecycle.name in template to {metricbeat-acme {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-04-06T08:28:11.917-0300 INFO template/load.go:169 Existing template will be overwritten, as overwrite is enabled.
2020-04-06T08:28:12.301-0300 WARN [cfgwarn] template/template.go:148 EXPERIMENTAL: append_fields is used.
2020-04-06T08:28:12.305-0300 INFO template/load.go:109 Try loading template metricbeat-7.6.2 to Elasticsearch
2020-04-06T08:28:12.942-0300 INFO template/load.go:101 template with name 'metricbeat-7.6.2' loaded.
2020-04-06T08:28:12.942-0300 INFO [index-management] idxmgmt/std.go:295 Loaded index template.
2020-04-06T08:28:13.106-0300 INFO [index-management] idxmgmt/std.go:306 Write alias successfully generated.
Goals:
I want to avoid Dashboards explosion, so I don't wnat to set Kibana Dashboards for each client (changing setup.dashboards.index to metricbeat-ffknob and then running ./metricbeat setup --dashboards)
I want tobe able to see all clients' data in the same Dashboard
I want to be able to define different ILM policies for each client (that's why I'm willing to give this approach a try)
So, I think that what I want is:
ALIAS metricbeat-* -> READ metricbeat-acme-*, metricbeat-otherclient-*, ... / WRITE ???
Questions:
With ILM set, how to I make the Beat write to an specifc index (metricbeat-acme-*)? Here is what the log says:
Set setup.template.name to '{metricbeat-7.6.2 {now/M{yyyy.MM}}-000001}' as ILM is enabled.
Set setup.template.pattern to 'metricbeat-7.6.2-*' as ILM is enabled.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.