# Adding traces-apm.rum@custom pipeline breaks RUM traces

**URL:** <https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201>\
**Category:** Elastic Observability\
**Created:** [February 9, 2023, 10:24pm UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201 "2023-02-09T22:24:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jrhut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jrhut/32/117079_2.png) [@jrhut](https://discuss.elastic.co/u/jrhut)\
**Post date:** [February 9, 2023, 10:24pm UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/1 "2023-02-09T22:24:44Z")

</div>

Hi there,

RUM traces work fine in the UI until I create a traces-apm.rum@custom pipeline with a pipeline processor. The presence of the pipeline processor breaks RUM transaction traces creating a perpetual loading animation in the UI under trace sample in Kibana.

After extensive testing I can resolve this by adding a arbitrary set processor after the pipeline processor and also a arbitrary set processor in the failure processors. I can't understand why this is necessary, the pipeline processor does not fail and the conditional on the pipeline processor is working correctly.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/0/f053d1b94e25c3417a477f06dbaea40f72c42db3.png)

If I remove the set processors, tracing breaks. The failure processor is not hit. Is this a bug or am I missing something?

Cheers,  
Jamie

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [February 10, 2023, 3:13am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/2 "2023-02-10T03:13:32Z")

</div>

@jrhut welcome to the forum!

> The presence of the pipeline processor breaks RUM transaction traces creating a perpetual loading animation in the UI under trace sample in Kibana.

This sounds strange, and not something I've observed before. Can you share your `traces-apm.rum-genericsearch` pipeline? Can you see an errors in the browser console when the UI goes into this perpetual loading state?

---

<div class="post-metadata">

**Author:** ![jrhut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jrhut/32/117079_2.png) [@jrhut](https://discuss.elastic.co/u/jrhut)\
**Post date:** [February 10, 2023, 4:44am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/3 "2023-02-10T04:44:36Z")

</div>

Hi @axw cheers!

I can for sure, here it is:

```auto
[
  {
    "urldecode": {
      "field": "url.original",
      "target_field": "search.query"
    }
  },
  {
    "gsub": {
      "field": "search.query",
      "pattern": ".*queryParamSearchFilters=",
      "replacement": ""
    }
  },
  {
    "gsub": {
      "field": "search.query",
      "pattern": "[\"\\(\\)\\{\\}\\[\\]]",
      "replacement": ""
    }
  },
  {
    "gsub": {
      "field": "search.query",
      "pattern": "(filters:)",
      "replacement": ""
    }
  },
  {
    "kv": {
      "field": "search.query",
      "field_split": ",",
      "value_split": ":",
      "include_keys": [
        "filterKey",
        "filterValue"
      ]
    }
  },
  {
    "rename": {
      "field": "filterKey",
      "target_field": "search.keys"
    }
  },
  {
    "rename": {
      "field": "filterValue",
      "target_field": "search.values"
    }
  },
  {
    "remove": {
      "field": "search.query"
    }
  }
]

```

I think after a little more testing the loading state occurred because there was simply no trace sample for the transaction. I can't reproduce the loading trace as I have some trace samples now but if you set the time span to for example 1 second with no transactions it produces the same effect. This screenshot shows the disparity between total transactions and trace samples with a sample rate of 1.

![transaction-ss](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a2e08df2040f121348ed30658aea4dc6d71d892c.png)

Also, just to confirm that the data was definitely going to the cluster I checked the browser post requests and all the trace data was there and the request successful.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [February 10, 2023, 8:04am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/4 "2023-02-10T08:04:34Z")

</div>

> The failure processor is not hit.

I guess you verified that by checking if any docs had `custom.pipes: failure`? It _sounds_ like the ingest pipeline is failing, since if there's no processors defined there then it'll just throw an exception.

If you remove the `set` processors, but configure `ignore_failure: true` on the `pipeline` processor, does that allow trace events to be ingested?

Are there any errors reported in the APM Server log?

> I think after a little more testing the loading state occurred because there was simply no trace sample for the transaction. I can't reproduce the loading trace as I have some trace samples now but if you set the time span to for example 1 second with no transactions it produces the same effect. This screenshot shows the disparity between total transactions and trace samples with a sample rate of 1.

The latency distribution is based off pre-aggregated metrics (calculated by APM Server), so if individual trace events are being dropped due to a failing ingest pipeline then that may explain the disparity.

---

<div class="post-metadata">

**Author:** ![jrhut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jrhut/32/117079_2.png) [@jrhut](https://discuss.elastic.co/u/jrhut)\
**Post date:** [February 12, 2023, 7:58pm UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/5 "2023-02-12T19:58:46Z")

</div>

Hi @axw

I just tested with `ignore_failure: true` and still the same result. This time actually I don't get the trace but something does appear in the trace sample. See:

 ![Screenshot 2023-02-13 at 08-56-22 _ - Transactions - KMR_Frontend - Services - APM - Observability - Elastic](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3bc175df54839268a3dbb4fab3d3c9b4b8acad87.png)

I also set a failure set processor on the genericsearch pipeline but that was never hit.

I'm not sure how to access the APM Server log, my deployment is on Elastic Cloud?

Cheers,  
Jamie

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [February 13, 2023, 12:41am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/6 "2023-02-13T00:41:11Z")

</div>

> [@jrhut](#):
>
> I'm not sure how to access the APM Server log, my deployment is on Elastic Cloud?

In the [https://cloud.elastic.co](https://cloud.elastic.co) console you can enable logging & metrics, either sending them to a dedicated deployment or to the same deployment (not recommended for production, but fine for this kind of debugging). See [How to set up monitoring | Elasticsearch Service Documentation | Elastic](https://www.elastic.co/guide/en/cloud/current/ec-monitoring-setup.html#ec_enable_logs_and_metrics)

Alternatively you can DM me the deployment ID -- it'll be in the URL in the Elastic Cloud console -- and I can take a look at the server logs.

---

<div class="post-metadata">

**Author:** ![jrhut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jrhut/32/117079_2.png) [@jrhut](https://discuss.elastic.co/u/jrhut)\
**Post date:** [February 13, 2023, 1:00am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/7 "2023-02-13T01:00:59Z")

</div>

I just sent you a DM with the deployment ID if that's ok.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [February 13, 2023, 1:47am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/8 "2023-02-13T01:47:47Z")

</div>

> [@jrhut](#):
>
> I just tested with `ignore_failure: true` and still the same result. This time actually I don't get the trace but something does appear in the trace sample. See:

What do you mean by "I don't get the trace"? If there's a trace sample, what's missing?

---

<div class="post-metadata">

**Author:** ![jrhut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jrhut/32/117079_2.png) [@jrhut](https://discuss.elastic.co/u/jrhut)\
**Post date:** [February 13, 2023, 2:21am UTC](https://discuss.elastic.co/t/adding-traces-apm-rum-custom-pipeline-breaks-rum-traces/325201/9 "2023-02-13T02:21:25Z")

</div>

This is what it looks like when working. The photo I posted earlier wasn't really a trace just a page load.

 ![full_trace](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2b793bcb472eb2c6ffef32537df0d15d01573da3.png)