Incompatibility while using co.elastic.apm:apm-opentracing:1.6.1 and elastic-apm-agent-1.6.1

[Barry_Zhong]

Kibana version:
6.5.4
Elasticsearch version:
6.5.4
APM Server version:
6.5.4
APM Agent language and version:
Java elastic-apm-agent-1.6.1

Browser version:
Chrome Version 74.0.3729.169 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
download page

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?
to enable Reactive Spring (reactor) APM, I am using the following combination:

   <dependency>
      <groupId>io.opentracing.contrib</groupId>
      <artifactId>opentracing-spring-cloud-starter</artifactId>
      <version>0.2.6</version>
    </dependency>
    <dependency>
      <groupId>co.elastic.apm</groupId>
      <artifactId>apm-opentracing</artifactId>
      <version>1.6.1</version>
    </dependency>

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
I thought it would be an obvious reason, so forgive me to skip this part.

Errors in browser console (if relevant):
n/a

Provide logs and/or server output (if relevant):

gateway_1         | 2019-05-24 10:58:26.651 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {"accepted":14,"errors":[{"message":"Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/0] allOf failed\n    I[#/context/tags/http_status_code] S[#/allOf/0/properties/context/properties/tags/patternProperties/%5E%5B%5E.%2A%22%5D%2A$/type] expected string or null, but got number","document":"{\"span\":{\"name\":\"GET\",\"timestamp\":1558695496561368,\"id\":\"9728c7a0a10689a0\",\"trace_id\":\"449b74106bba1b596a9072da1da6d00d\",\"transaction_id\":\"cb71b6fbd0152a53\",\"parent_id\":\"cb71b6fbd0152a53\",\"duration\":37.993,\"stacktrace\":[{\"filename\":\"TracingClientResponseSubscriber.java\",\"function\":\"onComplete\",\"library_frame\":true,\"lineno\":105,\"abs_path\":\"io.opentracing.contrib.spring.web.client.TracingClientResponseSubscriber\"},

[felixbarny (Felix Barnsteiner)]

Hi and thanks for your question. In order to leverage number and boolean tags (which opentracing-spring-cloud-starter apparently makes use of), you have to upgrade to APM Server 6.7.

[Barry_Zhong]

is there a migration guide?

[felixbarny (Felix Barnsteiner)]

You can find the release notes here: https://www.elastic.co/guide/en/apm/server/6.7/release-notes.html

Since 6.7 is the same major version you are using, upgrading should be straightforward.

[Barry_Zhong]

It works, thanks @felixbarny.

[Barry_Zhong]

what's the plan for upgrading io.opentracing:opentracing-api:0.32.0?

Since the latest apm-opentracing is only compatible with the latest Spring Cloud Opentracing on 0.2.x series

[felixbarny (Felix Barnsteiner)]

We have an issue for that here: https://github.com/elastic/apm-agent-java/issues/634

But note that OpenTracing is discontinued and will live on as OpenTelemetry with a completely different api. I still think it makes sense that the Java agent will have support for the final version though. But there are a lot of backwards incompatible changes which makes supporting multiple versions really difficult. While I'd like to tackle this and find solutions right away, it's difficult to prioritize this over other features ATM.

[Barry_Zhong]

While using spring cloud gateway (reactive) and apm, we came across a weird situation as following:

any clue?

[felixbarny (Felix Barnsteiner)]

Both the Elastic APM Java agent and the reactor instrumentation propagate the context for ExecutorServices. Possibly there's something going wrong with both doing it.

Could you try again with the latest snapshot (1.6.2-SNAPSHOT) of the agent? If you get the same warnings, please enable trace logging (set log_level=trace) and execute the minimum amount of requests to reproduce the problem.

[Barry_Zhong]

before trying that solution, I came across following error:

2019-06-01 13:33:48.066 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1