APM python agent stoped showing boto3 spans

Ahmad_Al-Masry · November 20, 2018, 1:29pm

Hi;
We are using elastic cloud and we were using 6.4.3 and apm agent 3.0.3, then, the boto3 instrumentation went so well, but after we upgraded to 6.5.0 and apm agent 4.0.1 the distributed tracing went well but the boto3 instrumentation stopped working.

I do not know if they are related, but if we disabled the distributed tracing, will boto3 instrumentation return to work? If yes, how can we disable the distributed tracing.

Note: boto3 calls appear as GET with now argument of to 169.* ip.

Thank you for your cooperation.

beniwohli · November 20, 2018, 4:15pm

Hi @Ahmad_Al-Masry

Sorry about that! Distributed tracing definitely shouldn't break boto3 support. I'll investigate this and report back when I find the problem. To help me get started, can you tell me which version of boto3 and botocore you use?

Ahmad_Al-Masry · November 20, 2018, 4:40pm

Hi;

We use the following both 1.4.1 and 1.4.4, both were working fine before the update.

And we use the following configs:

ELASTIC_APM = {

'SERVICE_NAME': 'lpm-aggregator',

'ENVIRONMENT': 'prod',

'CAPTURE_BODY': 'error',

'DEBUG': False,

'INSTRUMENT': True,

}

INSTALLED_APPS = [

.

'elasticapm.contrib.django',

.

]

MIDDLEWARE_CLASSES = [

'elasticapm.contrib.django.middleware.TracingMiddleware',

.

]

These services are using uwsgi to run.

Thank you for your cooperation.

BR

Ahmad_Al-Masry · November 22, 2018, 8:41am

Hi;
Is there any misconfiguration from our side, or there is another issue?
BR

beniwohli · November 22, 2018, 9:00am

I'm still investigating the issue. I added a test to our test suite that tests the full HTTP flow (https://github.com/elastic/apm-agent-python/pull/341), and that seems to work, so there must be something else going on.

One thing that I would suggest is to upgrade boto3 and botocore to a more recent version if possible. Versions of boto3 before 1.8.0 vendored the requests module, which in turn vendored the urllib3 module, which makes things very complicated for us.

Ahmad_Al-Masry · November 22, 2018, 1:38pm

Hi;
We upgraded to 1.9.50 "Latest boto3" and the same problem.
I want to note that it was running perfectly when we used apm agent 3.0.3 and elastic 6.4.3.
Thank you for your help.

toughrogrammer · November 29, 2018, 4:29pm

I got same problem.
I'm using boto3 1.9.50, apm python agent 3.0.4, and apm server 6.5.1.

system · December 20, 2018, 12:30pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

beniwohli · January 2, 2019, 8:29am

Hey everybody

turns out this was a bug in the agent sorry for not recognizing that earlier. It should be fixed in both 3.0.5 and 4.0.3 of the agent.