# APM Sidekiq Maxium duration for trace?

**URL:** <https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092>\
**Category:** APM\
**Tags:** ruby\
**Created:** [October 10, 2019, 6:26pm UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092 "2019-10-10T18:26:38Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dnorth98](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnorth98/32/54823_2.png) [@dnorth98](https://discuss.elastic.co/u/dnorth98)\
**Post date:** [October 10, 2019, 6:26pm UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092/1 "2019-10-10T18:26:39Z")

</div>

**Kibana version** : 7.4

**Elasticsearch version** : 7.4

**APM Server version** : 7.4

**APM Agent language and version** : Latest v3, Ruby

We're using Ruby APM to trace some Sidekiq jobs and the APM agent picks these up nicely. However, we have a large number of jobs that run for over an hour (some even run for mutliple hours - don't ask!) and I'm seeing that those transactions doesn't appear to be captured. Looking at the bucketing of transactions, the longest one I see is 15 minutes. Yet looking in sidekiq there are hundreds of jobs that are running for \> 1 hour.

One thing I notice is that for this service I did have the `ELASTIC_APM_TRANSACTION_SAMPLE_RATE` set to 0.7 so I'm wondering if it just wasn't sampling them. I'd have expected to see at least a couple though.

Is there some maximum limit to a transaction trace in either time or size and can it be tweaked if so?

Cheers

Dave

---

<div class="post-metadata">

**Author:** ![mikker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mikker/32/22118_2.png) [@mikker](https://discuss.elastic.co/u/mikker)\
**Post date:** [October 11, 2019, 10:46am UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092/2 "2019-10-11T10:46:49Z")

</div>

Hi Dave!

There's no max length on transactions nor spans, but to be honest we haven't tested the agent with such long-running Sidekiq jobs.

Your theory around sampling could be the reason but I'm not sure. I'll investigate a bit and get back to you.

---

<div class="post-metadata">

**Author:** ![mikker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mikker/32/22118_2.png) [@mikker](https://discuss.elastic.co/u/mikker)\
**Post date:** [October 11, 2019, 12:03pm UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092/3 "2019-10-11T12:03:30Z")

</div>

It could be because Kibana finds it to have too low of an _impact_ versus other jobs. Do you have many kinds of jobs?

Try running this in the developer console and see if the total number of _transaction groups_ is bigger than 100:

```auto
GET apm-*-transaction*/_search
{
  "size": 0,
  "query": {
    "term": {
      "service.name": "YOUR_SERVICE"
    }
  },
  "aggs": {
    "transaction_groups": {
      "cardinality": {
        "field": "transaction.name"
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![dnorth98](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnorth98/32/54823_2.png) [@dnorth98](https://discuss.elastic.co/u/dnorth98)\
**Post date:** [October 11, 2019, 12:25pm UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092/4 "2019-10-11T12:25:06Z")

</div>

Hi Mikkel,  
Ran the query..it's only 9. I did some digging around after seeing this and it was weird. I found that changing the time period in the APM console to "24 hours" showed me the longer traces even though they were within the last 2 hours. When I had the time set to "2 hours", they didn't appear. Can't quite explain that.

I looked at the logs on the agent found I was getting `Queue is full (256 items), skipping…` so I tweaked the sample rate and added 2 threads the pool. Coming in this morning, I've got a LOT more of these long running jobs now (some ran for 3 hours) so it's good to know there's actually not a limit on the size of a transaction in terms of duration.

Of course, what I've just hit now is the 500 spans limit so I'm going to experiement with tweaking `ELASTIC_APM_TRANSACTION_MAX_SPANS` to find a compromise between memory and getting full traces. I suspect I won't find one since I know from looking at the spans we do get that there's a LOT of calls these jobs are making (it's a long story but these jobs backup data from e-commerce stores using their API and there are a lot of calls that need to be made depending on the amount of data to be backed up).

Loving how tweakable all the APM parameters are though - really feels I tune this to get a good compromise between "all the data" and "something that's useful"

---

<div class="post-metadata">

**Author:** ![mikker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mikker/32/22118_2.png) [@mikker](https://discuss.elastic.co/u/mikker)\
**Post date:** [October 14, 2019, 12:06pm UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092/5 "2019-10-14T12:06:00Z")

</div>

Thanks a bunch for the update, Dave! Let us know if you need any further assistance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2019, 8:06am UTC](https://discuss.elastic.co/t/apm-sidekiq-maxium-duration-for-trace/203092/6 "2019-11-04T08:06:05Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.