Moving applications logs to Elastic Observability

There's currently no reasonable workaround for APM Server to split logs into multiple data streams. When you run APM Server with Elastic Agent and Fleet, APM Server is given only limited privileges and cannot write to arbitrary logs data streams. I wouldn't really recommend this, but you could run APM Server in "legacy", AKA standalone mode: https://www.elastic.co/guide/en/apm/guide/current/install-and-run.html. If you configure APM Server's Elasticsearch output with sufficient privileges (e.g. to write to any data stream), then you could modify the ingest pipelines to use a script ingest processor to modify the data stream by setting the _index metadata field: Script processor | Elasticsearch Guide [8.11] | Elastic.

We have been talking about this issue internally for a little while, but as yet we do not have a solution. I've just opened an issue to enable dynamic data stream routing, if you would like to subscribe: [Fleet] Dynamic data stream namespaces · Issue #134971 · elastic/kibana · GitHub.

Originally we intended to produce application-specific log data streams. For example, say you had logs for Kibana and Elasticsearch: we would index these into two data streams, like logs-apm.app.kibana-<namespace> and logs-apm.app.elasticsearch-<namespace>. This turned out to create too much load on Elasticsearch for users with hundreds or thousands of unique services, hence why we currently send everything to one data stream per type of data.