At our organization, we have a system that has north of 1000 microservices, deployed across multiple environments. These are all instrumented with telemetry, communicating to our Elastic Stack. This past year we've migrated to native Open Telemetry, but everything is visible for the organization within Kibana's APM app.
According to this, the system writes APM/Telemetry data according to:
metrics-apm.app.<service.name>-<namespace>
Since each of our services have a unique service.name, they all get stored in their own index. With replica shards, that means 1500 services = 3000 shards. This is on top of any system indices, plus the logging we also ingest to the same system. Our system is nearing 5000 shards, and experiencing performance issues.
We can of course consolidate our services to share the same service.name for related services, but operationally that makes life very difficult, if they're no longer individually visible within the APM app in Kibana.
Does anyone know of a way to have a system with many services, but few indices? I wish the pattern for storing data were something like metrics-apm.app-<namespace>, and we wouldn't be concerned with this.
Thank you.