# APM server producing errors (mapper\_parsing\_exception) when traces are sent through jaeger-agent
**URL:** https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457
**Category:** APM
**Tags:** docker, server
**Created:** [September 13, 2020, 8:20pm UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457 "2020-09-13T20:20:41Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Amirtheahmed](https://discuss.elastic.co/u/Amirtheahmed)
#### Post date: [September 13, 2020, 8:20pm UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/1 "2020-09-13T20:20:41Z")
**Kibana version** :
7.9.1
**Elasticsearch version** :
7.9.1
**APM Server version** :
7.9.1
**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:
Official docker images
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:
I'm trying to use Elastic APM as backend for distibuted tracing. I am using jaeger-agent to collect traces from laravel app. My previous tracing setup with docker-compose was working flawlessly (jaeger agent + collector + jaeger-queryUi), Now i modified my setup to use APM as a backend by sending traces from my app -\> Jaeger-Agent -\> APM server via gRPC. I followed this jaeger integration guide [https://www.elastic.co/guide/en/apm/server/current/jaeger.html](https://www.elastic.co/guide/en/apm/server/current/jaeger.html) but when i start sending traces i'm geting these errors in APM logs:
```
2020-09-13T20:07:31.202Z WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x148af8e0, ext:63735624444, loc:(*time.Location)(nil)}, Meta:{"pipeline":"apm"}, Fields:{"agent":{"name":"Jaeger","version":"unknown"},"ecs":{"version":"1.5.0"},"labels":{"Lounge":"GatewayService-JaegerProvider","peer.ipv4":"172.21.0.3","peer.port":"80","sampler.param":"true","sampler.type":"const"},"observer":{"ephemeral_id":"98acea66-a148-497b-9eaa-c360900a367a","hostname":"ee30089a825b","id":"f9fd9f44-dbc7-4cdd-9403-54576dba007a","type":"apm-server","version":"7.9.1","version_major":7},"processor":{"event":"transaction","name":"transaction"},"service":{"language":{"name":"unknown"},"name":"Gateway-Service"},"timestamp":{"us":1600027644344652},"trace":{"id":"163470aa489b76d9"},"transaction":{"duration":{"us":5532199},"id":"163470aa489b76d9","name":"GET: /csrfd","result":"Success","sampled":true,"type":"fpm"},"url":{"full":"http:///csrfd","original":"/csrfd","path":"/csrfd","scheme":"http"}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"class_cast_exception","reason":"class org.elasticsearch.index.mapper.ScaledFloatFieldMapper cannot be cast to class org.elasticsearch.index.mapper.ObjectMapper (org.elasticsearch.index.mapper.ScaledFloatFieldMapper is in unnamed module of loader java.net.FactoryURLClassLoader @50a13c2f; org.elasticsearch.index.mapper.ObjectMapper is in unnamed module of loader 'app')"}}
```
When i switch to jaeger-collector instead of APM - it gets all the tracing correctly.
Here is my docker-compose.yml:
```auto
version: "3"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
networks:
- author-book-network
ports:
- "9200:9200"
- "9300:9300"
restart: on-failure
environment:
- cluster.name=jaeger-cluster
- discovery.type=single-node
- http.host=0.0.0.0
- transport.host=127.0.0.1
- ES_JAVA_OPTS=-Xms1000m -Xmx1000m
- xpack.security.enabled=false
volumes:
- esdata:/usr/share/elasticsearch/data
kibana:
image: docker.elastic.co/kibana/kibana:7.9.1
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- SERVER_NAME=kibana
#- ES.TAGS-AS-FIELDS.ALL=true
#- SERVER_HOST=0.0.0.0
depends_on:
- elasticsearch
networks:
- author-book-network
apm-server:
image: docker.elastic.co/apm/apm-server:7.9.1
depends_on:
- elasticsearch
- kibana
cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
cap_drop: ["ALL"]
ports:
- 8200:8200
networks:
- author-book-network
command: >
apm-server -e
-E apm-server.jaeger.grpc.enabled=true
-E apm-server.jaeger.grpc.host=":14250"
-E apm-server.rum.enabled=true
-E setup.kibana.host=kibana:5601
-E setup.template.settings.index.number_of_replicas=0
-E apm-server.kibana.enabled=true
-E apm-server.kibana.host=kibana:5601
-E output.elasticsearch.hosts=["elasticsearch:9200"]
-E ELASTIC_APM_LOG_FILE=stderr
-E ELASTIC_APM_LOG_FILE=debug
# healthcheck:
# interval: 10s
# retries: 12
# test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/
# jaeger-collector:
# image: jaegertracing/jaeger-collector
# ports:
# #- "14269:14269"
# - "14250:14250"
# - "14268:14268"
# - "14267:14267"
# - "9411:9411"
# networks:
# - author-book-network
# restart: on-failure
# environment:
# - SPAN_STORAGE_TYPE=elasticsearch
# command: [
# "--es.server-urls=http://elasticsearch:9200",
# "--es.num-shards=1",
# "--es.num-replicas=0",
# "--log-level=error"
# ]
# depends_on:
# - elasticsearch
jaeger-agent:
image: jaegertracing/jaeger-agent
hostname: jaeger-agent
#command: ["--reporter.grpc.host-port=jaeger-collector:14250"]
command: ["--reporter.grpc.host-port=apm-server:14250"]
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
- "5778:5778"
networks:
- author-book-network
restart: on-failure
environment:
- SPAN_STORAGE_TYPE=elasticsearch
depends_on:
#- jaeger-collector
- apm-server
jaeger-query:
image: jaegertracing/jaeger-query
environment:
- SPAN_STORAGE_TYPE=elasticsearch
- no_proxy=localhost
ports:
- "16686:16686"
- "16687:16687"
networks:
- author-book-network
restart: on-failure
command: [
"--es.server-urls=http://elasticsearch:9200",
"--span-storage.type=elasticsearch",
"--log-level=debug"
]
depends_on:
- jaeger-agent
volumes:
esdata:
driver: local
networks:
author-book-network:
external: true
```
**Steps to reproduce** :
1. Install Elasticsearch, Kibana and APM through docker-compose
2. Use Jaeger-Agent to send tracing to APM
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [September 14, 2020, 1:56am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/2 "2020-09-14T01:56:58Z")
Welcome to the forum @Amirtheahmed!
Do you have multiple services instrumented? That sort of error would occur when the instrumentation provides two different data types for the same attribute.
If this is the case, then I think the simplest short-term solution would be to split services into multiple indices, to avoid these kinds of mapping conflicts:
> [@Storing APM data in custom/specific indices](https://discuss.elastic.co/t/storing-apm-data-in-custom-specific-indices/246381):
>
> If you'd like to split your APM indices by the service, you can use the advice from [this topic](https://discuss.elastic.co/t/how-to-create-index-for-each-service/242821). You can change the index name by setting either [output.elasticsearch.index](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#index-option-es) or [output.elasticsearch.indices](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#indices-option-es). These configuration variables take an index name "format string", which can reference fields the events. For your use-case, you can include %{[service.name]} in the format string to create an index per service. e.g. output.elasticsearch: hosts: ["http://localhost:9200"] indices: - inde…
---
### Author:  [@Amirtheahmed](https://discuss.elastic.co/u/Amirtheahmed)
#### Post date: [September 14, 2020, 4:34am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/3 "2020-09-14T04:34:22Z")
Hi and thanks for your reply,
Yes i have multiple services instrumented and i will look into the solution you provided when dealing with multiple services. But the error log i shared is from single service with a single span trace (gateway-service) as i deliberately turned off all other services for debugging sake yet still `class org.elasticsearch.index.mapper.ScaledFloatFieldMapper cannot be cast to class org.elasticsearch.index.mapper.ObjectMapper` persists.
Here is the result trace i see when i use pure jaeger-agent -\> jaeger-collector -\> jaeger-ui (with elasticsearch as storage): Note! It is sending only single span
 
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [September 14, 2020, 6:23am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/4 "2020-09-14T06:23:22Z")
It looks like you found a bug, sorry to say. I've opened [https://github.com/elastic/apm-server/issues/4190](https://github.com/elastic/apm-server/issues/4190) to get this fixed. I made an educated guess that you're using [https://github.com/jukylin/jaeger-php;](https://github.com/jukylin/jaeger-php;) please correct me in the issue if I got that wrong.
---
### Author:  [@Amirtheahmed](https://discuss.elastic.co/u/Amirtheahmed)
#### Post date: [September 14, 2020, 8:30am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/5 "2020-09-14T08:30:42Z")
yes am using [https://github.com/jukylin/jaeger-php;](https://github.com/jukylin/jaeger-php;)
---
### Author:  [@Amirtheahmed](https://discuss.elastic.co/u/Amirtheahmed)
#### Post date: [September 14, 2020, 8:53am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/6 "2020-09-14T08:53:53Z")
Is it fixed now ?
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [September 14, 2020, 9:11am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/7 "2020-09-14T09:11:51Z")
I've merged a fix, which will go into the next release. There should be a 7.9.2 release in the next 1-2 weeks.
If you would like to test it sooner, it's possible but requires building apm-server yourself:
- install Make and Go 1.13.10
- clone [https://github.com/elastic/apm-server/tree/7.9](https://github.com/elastic/apm-server/tree/7.9)
- run `make release TYPES=docker`
- run `docker image load -i build/distributions/apm-server-7.9.2-linux-amd64.docker.tar.gz`
---
### Author:  [@Amirtheahmed](https://discuss.elastic.co/u/Amirtheahmed)
#### Post date: [September 14, 2020, 1:59pm UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/8 "2020-09-14T13:59:57Z")
Thank you very much, i built apm-server myself as you outlined and the errors has gone away.
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [September 15, 2020, 2:05am UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/9 "2020-09-15T02:05:20Z")
Glad to hear it! Thanks for testing and reporting back.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 5, 2020, 10:05pm UTC](https://discuss.elastic.co/t/apm-server-producing-errors-mapper-parsing-exception-when-traces-are-sent-through-jaeger-agent/248457/10 "2020-10-05T22:05:28Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.