Python RQ

[Josef_Rousek]

Hi,
I'm having hard time setting up APM with Python RQ, which is using fork. I was following Python agent only recording some background tasks, which led me to creating custom rq.job. However with removal of MAX_QUEUE_SIZE I don't know how to convince agent to send the transaction synchronously in the same thread as the background job.

The only way I got it to send a transaction was to call client._transport._process_queue() but then it never gets out of the loop.

import elasticapm
from elasticapm.contrib.django.client import get_client
from rq.job import Job


class TracingJob(Job):
    def perform(self):
        client = get_client()
        client.begin_transaction("backgroundjob")
        context = {"args": self.args, "kwargs": self.kwargs}
        print(context)
        print(self.func_name)
        elasticapm.set_custom_context(context)
        try:
            super().perform()
            client.end_transaction(self.func_name, "SUCCESS")
        except:
            client.capture_exception()
            raise
        finally:
            client._transport._process_queue()

I'm using elastic apm python agent 5.1.2.

[basepi (Colton Myers)]

Great question! It looks like MAX_QUEUE_SIZE was replaced with API_REQUEST_SIZE. So, if you want to make the agent send after each transaction, you can set API_REQUEST_SIZE to "1b" (1 byte) and it should give you the same behavior as with the old MAX_QUEUE_SIZE=1.

[Josef_Rousek]

Hi Colton,

Thanks for the suggestion. However it didn't do the trick. I tried to debug where the things go wrong but the code is just advanced for me. I created minimal Django app with Elastic APM working on the view and my attempt to implement TracingJob.

stlk/elastic-rq-demo

Readme describes how to setup the app and how to replicate my issue. It would be great if someone with understanding of the queue and the transport layer could take a look at it.

[basepi (Colton Myers)]

Thanks for providing a way to reproduce this! I don't want to lose track of this so can you replicate the important bits of this thread in an issue (including your demo repo) so that we can track it? I definitely want to take a look when I find a moment.

(We'll also want to track down if there is a bug in that API_REQUEST_SIZE config or if we're just never getting there for some reason.)

[Josef_Rousek]

Sure. Here's the issue with rephrased problem https://github.com/elastic/apm-agent-python/issues/637

Thanks a lot!