Best practice for re-routed datastream and index template mappings

Elastic Stack Elasticsearch
1

Hi community,

I'm wondering what is the best practice when it comes to rerouted datastreams from log integrations using fleet. For example default datastream is logs-docker.container_logs-default and it's rerouted to logs-my_app.logs-default.

Ingest pipeline can use reroute-processor to send data to a different datastream. If the target is a custom datastream, for example not part of any integration that matches an explicit index template - it will match the generic logs managed template.

Is it best practice to create a index template named logs-my_app.logs-* to match this datastream and indices? And is it actually required in order for mappings to apply correctly?

Thanks.

2

Hi @dot-mike

Yes, if there is no match, the data stream will fall back and match logs-*; however, in IMHO I reccomended to create a specific template for your new stream matching logs-my_app.logs-*.

This is what I suggest to my prod/enterprise customers

Why? More Control and Isolation. AND if you want to add a matching ingest pipeline or matching mappings, or matching @custom template etc... it provides that opportunity.

I just clone an existing template like the logs-docker.container_logs and make the updates

logs-my_app.logs-default
Index pattern
logs-my_app.logs-default-*
Priority
200
Component templates
logs@mappings
logs@settings
logs-my_app.logs-default@package
logs@custom
logs-my_app.logs-default@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

Don't forget to set your pipeline in logs-my_app.logs-default@package


{
  "index": {
    "lifecycle": {
      "name": "logs"
    },
    "default_pipeline": "logs-my_app.logs-1.0.0",
    "mapping": {
      "total_fields": {
        "limit": "1000"
      }
    }
  }
}

Then you will need to create a stub pipeline etc...

All this should take about 10 mins

That is how I suggest to my customers.

1 Like
3

Many thanks! Exactly what I wanted to hear