Ingest strategy

Elastic Stack Elasticsearch
1

Hi there, as a (broader) follow up on this topic, I am creating this new one about ingest strategy, as I am utterly running out of ideas.

My use case has many client machines, each logging various metrics (pretty common I believe). I need:

  1. easy enrollment for new client machines
  2. secure ingest (not open to anyone knowing the IP/port)
  3. metric-specific type mapping, to be able to use said metrics in Kibana dashboards

I have tried making use of the "custom TCP log"+"elastic agent" integration, which ticked boxes 1 and 2 I believe, but:

  • everything gets ingested on a single data stream
  • I have not been able to achieve dynamic type mapping depending on message content in any way (I have a special name field indicating the metric type in the payload). I tried using a custom ingest pipeline on the Custom TCP log policy, to rename the meta field _index to a metric specific one => failed due to lack of permissions for the fleet agent user (even when prepending the target index name with logs-).

My specific questions are:

  1. Did I make a wrong choice straight from the start (mis)using the "Custom TCP log" policy?
  2. Can someone please describe an ingest strategy that fits this use case?

Thanks in advance

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.