APM Server version: 1.29.0
APM Agent language and version:
[INFO] +- co.elastic.apm:apm-agent-api:jar:1.29.0:compile
[INFO] +- co.elastic.apm:elastic-apm-agent:jar:1.29.0:compile
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Under scale run - API call is getting rejected once in while by webclient(one in 100 times)
netty is rejecting the call because - there is a prohibited character getting inserted as part of the traceparent. Also traceparent is getting modified.
This is not happening very often. When it happens functionality is broken as the service to service call fails.
a header value contains a prohibited character '': 00-4a4a577c4bb47fb90ade48ffd8ef9fa3cba3a3635c53c76d676d6d-01
a header value contains a prohibited character '': 00-44f1278d295ed390916bdfb091679-30e5a88ffe7277267b--0101
a header value contains a prohibited character '': 00-c5708869a028e0c8847e63a42f675dc3-0c8847e63a42f675dc3df18fab8-01bedf18fab8-01
That prohibited character is: \0 (0x0) NUL
java.lang.IllegalArgumentException: a header value contains a prohibited character '': 00-c5708869a028e0c8847e63a42f675dc3-0c8847e63a42f675dc3df18fab8-01bedf18fab8-01
at io.netty.handler.codec.http.DefaultHttpHeaders$HeaderValueConverterAndValidator.validateValueChar(DefaultHttpHeaders.java:481)
at io.netty.handler.codec.http.DefaultHttpHeaders$HeaderValueConverterAndValidator.convertObject(DefaultHttpHeaders.java:461)
at io.netty.handler.codec.http.DefaultHttpHeaders$HeaderValueConverterAndValidator.convertObject(DefaultHttpHeaders.java:452)
at io.netty.handler.codec.DefaultHeaders.setObject(DefaultHeaders.java:496)
at io.netty.handler.codec.http.DefaultHttpHeaders.set(DefaultHttpHeaders.java:189)
at org.springframework.http.client.reactive.ReactorClientHttpRequest.lambda$applyHeaders$3(ReactorClientHttpRequest.java:127)
at java.base/java.util.Map.forEach(Unknown Source)
at org.springframework.http.client.reactive.ReactorClientHttpRequest.applyHeaders(ReactorClientHttpRequest.java:127)
at org.springframework.http.client.reactive.AbstractClientHttpRequest.lambda$null$0(AbstractClientHttpRequest.java:142)
This started happening after upgrading to 1.29.0
Steps to reproduce:
- Make an api call(using netty webclient) from within the completable future
- Run it in a loop
Provide logs and/or server output (if relevant):
WebClient.Builder webClientBuilder = WebClient.builder()
.exchangeStrategies(ExchangeStrategies.builder()
.codecs(configurer -> configurer
.defaultCodecs()
.maxInMemorySize(BYTE_COUNT))
.build())
.clientConnector(connector)
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.apply(oauth2.oauth2Configuration())
.filter(logRequestFilter())
.filter(tracingHeaderFilter());
private static ExchangeFilterFunction tracingHeaderFilter() {
return (clientRequest, next) -> {
Span span = ElasticApm.currentSpan();
ClientRequest.Builder filtered = ClientRequest.from(clientRequest);
span.injectTraceHeaders((name, value) -> {
if (name != null && value != null) {
filtered.header(name, value);
}
}
);
return next.exchange(filtered.build());
};
}
final Map<String, String> copyOfContextMap = getCopyOfContextMap();
List<List<String>> batchOfIds = Lists.partition(ids, 5);
batchOfIds.forEach(currentBatchIds -> currentBatchIds
.stream()
.map(id -> CompletableFuture.runAsync(() -> {
try {
MDC.setContextMap(copyOfContextMap);
// API call to get entity by id
} catch (Exception e) {
// throw e
} finally {
MDC.clear();
}
})).collect(Collectors.toList()).forEach(CompletableFuture::join));