Elastic agent can't reroute docs to a different index

[lizozom (Liza Katz)]

I have fleet agents with a custom logs.
I want to route some of the logs to a different index.
I created the index task-logs and in the integration, I defined an reroute ingest pipeline. However,the documents don't reach the index, and instead I get an error.

This is the error I see in the main logs-* index:

I suspect this might be related to the permissions error mentioned in the docs.

How do I (a) prove it and (b) fix it?

Thank you!

[lizozom (Liza Katz)]

I instead used the dataset option to receive the tasks.
It created an index (?) called logs-taks for me.
Is this the correct way to go?

[yago82]

Hi,

Yes, using the dataset option is a common way to route different types of logs to different indices in Elastic Stack.

Regards

[stephenb (Stephen Brown)]

Hi @lizozom

Did you try to simply filling in static fields for the namespace and dataset?

I don't think just logs-task is proper syntax I think you need all three parts

logs-tasks-generic

or something like that

[yago82]

HI,

from this article:

The Elastic data stream naming scheme is made for time series data and consists of splitting datasets into different data streams using the following naming convention.

• type: Generic type describing the data
• dataset: Describes the data ingested and its structure
• namespace: User-configurable arbitrary grouping

These three parts are combined by a “-” and result in data streams like logs-nginx.access-production. In all three parts, the “-” character is not allowed. This means all data streams are named in the following way:

{type}-{dataset}-{namespace}

For both dataset and namespace there is a default value, which is dataset=generic and namespace=default. In the case of Elastic Agent, if a user just starts to ingest a log file, the data ends up in logs-generic-default.**

**
An introduction to the Elastic data stream naming scheme | Elastic Blog

Regards