APM adding query params causing 400 bad request error during ES query

[gram (Graham H)]

Elasticsearch version: 7.0.1

APM Agent language and version: Python 4.2.2

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I am seeing the follow error multiple times in my logs:

RequestError(400, 'illegal_argument_exception', 
 'request [/my_index/_search] contains unrecognized parameters: [__elastic_apm_api_method_name], [__elastic_apm_body_ref]')  

It appears that APM is adding query params to the ES search request. When I run something similar with Kibana I see the same error:
GET /my_index/search?elastic_apm

  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "request [/my_index/_search] contains unrecognized parameter: [elastic_apm]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "request [/my_index/_search] contains unrecognized parameter: [elastic_apm]"
  },
  "status": 400
}

Can I force APM to not do this? Or, force ES to not throw because of unused params? Thanks!

Steps to reproduce:

1. Configure APM Python Agent
2. Have endpoint with ES query on that service

• service is using gunicorn & gevent & Flask

3. Run many calls against that endpoint and some will throw the exception

Provide logs and/or server output (if relevant):

[beniwohli (Benjamin Wohlwend @ APM)]

Hi Graham

thanks for the heads up! This does indeed look like a bug.

There were a few changes in the elasticsearch client for 7.0 that affect the APM agent. It's possible that this is the cause of the bug. I recently merged a fix, but didn't get around to publish a release yet.

If you have a test/staging environment where you could safely test this, you could try and install the agent from master to see if it fixes the issue:

pip install "https://github.com/elastic/apm-agent-python/archive/master.zip#egg=elastic-apm==100"

If that is not an option, you can disable the elasticsearch client instrumentation by setting the two environment variables SKIP_INSTRUMENT_ELASTICSEARCH and SKIP_INSTRUMENT_ELASTICSEARCH_CONNECTION to any value for your gunicorn process.

Cheers
Beni

[gram (Graham H)]

Thanks! I'll disable it until I can test the update.