Create custom data streams per service

[Namita_Jaokar]

Hi,

Is there any way I can create custom data stream per service in with elastic agent.?

I came across few solutions in the discussions , But probably its not been supported in version 8.6.1 or 8.x above.

Below is the solution that was being tried out:

Storing APM data in custom/specific indices

If you'd like to split your APM indices by the service, you can use the advice from this topic. You can change the index name by setting either output.elasticsearch.index or output.elasticsearch.indices. These configuration variables take an index name "format string", which can reference fields the events. For your use-case, you can include %{[service.name]} in the format string to create an index per service. e.g. output.elasticsearch: hosts: ["http://localhost:9200"] indices: - inde…

But seems above solution is not working.

Can someone please advice on this. Is it possible to create data streams for respective services

I want to maintain the transaction traces separately for every elastic agent.

Best Regards,
Namita Jaokar

[axw (Andrew Wilkins)]

@Namita_Jaokar sorry for the delay in responding. You're right that this won't work in newer versions of APM Server, as we no longer support output.elasticsearch.index or output.elasticsearch.indices. These days, APM Server will always write data to specific data streams. If you wish to route data to more specific data streams, you can do this in Elasticsearch using an ingest pipeline. See [OpenTelemetry] data_stream.namepace and data_stream.datastream aren't being respected · Issue #10191 · elastic/apm-server · GitHub

[Namita_Jaokar]

Hi @axw ,

Thank you for your response. I did check some links related to ingest pipeline as its a new concept for me to begin with.

I went through below link to understand the ingest pipeline

Parse data using ingest pipelines | APM User Guide [8.6] | Elastic

Sending other APM data to alternate data streams, like traces (traces-apm.* ), logs (logs-apm.* ), and internal metrics (metrics-apm.internal* ) is not currently supported.

This was mentioned in the documentation of ingest pipeline. Does that mean we cannot push apm-traces to a another/custom datastream.

Also, I tried to create a pipeline using GUI under stack management section. For the same, We need to create a processor.
When I try to add processor , It gives me below options

With which of the above options do I need to ask elasticsearch to create a new data stream for each of the service.name that are running.

Please correct me if my understanding is wrong.

Thanks in Advance,
Namita Jaokar

[axw (Andrew Wilkins)]

@Namita_Jaokar

Sending other APM data to alternate data streams, like traces (traces-apm.* ), logs (logs-apm.* ), and internal metrics (metrics-apm.internal* ) is not currently supported.

What this means is you can't for example send traces to a metrics data stream, or vice versa. This section needs a bit of tidying up, so I'll see if we can clarify this statement when we do.

With which of the above options do I need to ask elasticsearch to create a new data stream for each of the service.name that are running.

Sorry, I missed earlier that you were using 8.6. You will need to use a more recent version of the stack (8.10.x) so you can use the "reroute" ingest processor.

[Namita_Jaokar]

Thanks @axw for your help.
Will check if its within the scope of my requirement to upgrade to 8.10 or above.
Just wanted to confirm one more thing, I should be able to implement ingest processor specifically of type "reroute" with open source or do I need to have enterprise level subscription for it.

Best Regards,
Namita Jaokar

[axw (Andrew Wilkins)]

You do not need a enterprise subscription for reroute -- it is part of the core free and open functionality.

[Namita_Jaokar]

Thanks for confirming @axw .

Best Regards,
Namita