Ingest strategy

[mark8]

Hi there, as a (broader) follow up on this topic, I am creating this new one about ingest strategy, as I am utterly running out of ideas.

My use case has many client machines, each logging various metrics (pretty common I believe). I need:

1. easy enrollment for new client machines
2. secure ingest (not open to anyone knowing the IP/port)
3. metric-specific type mapping, to be able to use said metrics in Kibana dashboards

I have tried making use of the "custom TCP log"+"elastic agent" integration, which ticked boxes 1 and 2 I believe, but:

• everything gets ingested on a single data stream
• I have not been able to achieve dynamic type mapping depending on message content in any way (I have a special name field indicating the metric type in the payload). I tried using a custom ingest pipeline on the Custom TCP log policy, to rename the meta field _index to a metric specific one => failed due to lack of permissions for the fleet agent user (even when prepending the target index name with logs-).

My specific questions are:

1. Did I make a wrong choice straight from the start (mis)using the "Custom TCP log" policy?
2. Can someone please describe an ingest strategy that fits this use case?

Thanks in advance