After upgrading the stack to 8.10 from 8.9 the filtering by environment on the Metrics tab (in APM) stopped working.
Specifying an Environment (e.g. production) results in no data showing. However selecting "All" in the drop down and adding service.environment:"production" to the search bar works correctly. This only happens on the Metrics tab; the other ones work fine.
I think I found the direct cause, but not an explanation. On the Metrics tab if I click on the menu of one of the visualizations (e.g. System memory usage) and choose "Explore data in Discover", on the discover page two filters are added
service.name and service.environment; both have the SAME VALUE. service.environment is filtered on the service name instead of the value selected on the Environment drop-down on the APM page.
It is exactly as you wrote, after upgrading kibana to 8.10.4 and apm server to 8.10.4 the dropdown list stopped filtering the data correctly putting the same value of the service.name field into service.environment which cause to break all visualizations in APM > Services > service_name > Metrics
Worth to mention this was working in 7.x.x Kibana / APM versions.
Thanks. I have referenced this discussion in the Bug report. I would imagine that if this was a widespread problem there would have been a lot more bug reports. It must be a specific use-case that provokes this behavior .