APM data stream span move to new index

Elastic Observability APM
1

Hi,
I'm new to this community.
I have some experience with Elasticseach for over 1.5 years.

Currently, I'm having an issue with disk space and need to reduce how long we keep some data.
Currently, our system is running a fleet server with an APM agent on an edge machine.
Not all data in traces-apm-default data stream is used for same time period.
For example, API calls are stored for 30 days, but span is not needed for so long.

I found an option how to remove documents that have span data from general traces-apm.
But the issue is the script is not ingesting span doc to a new index.

In traces-apm@custom I added a condition to be triggered if doc has a processor.event => span
I created a new pipeline with a process script

[
  {
    "script": {
      "source": "if (ctx.span.action == 'query') 
              {    
               ctx._index = 'apm-span-dpd';  
              }",
      "if": "ctx.processor.event == 'span'"
      "description": " doc with processor.event == 'span /Index in apm-span-dpd"
    }
  }
]

When running the pipeline all works as expected except adding docs to a new index.
I created the index manually.
after that created ILM and index template.

I don't see any errors in logs.

Maybe there is another solution to remove span to new index from data stream

Thank you for your help.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.