APM server producing errors (mapper_parsing_exception) when traces are sent through jaeger-agent

[Amirtheahmed (Amir Ahmed)]

Kibana version:
7.9.1

Elasticsearch version:
7.9.1

APM Server version:
7.9.1

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Official docker images

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I'm trying to use Elastic APM as backend for distibuted tracing. I am using jaeger-agent to collect traces from laravel app. My previous tracing setup with docker-compose was working flawlessly (jaeger agent + collector + jaeger-queryUi), Now i modified my setup to use APM as a backend by sending traces from my app -> Jaeger-Agent -> APM server via gRPC. I followed this jaeger integration guide https://www.elastic.co/guide/en/apm/server/current/jaeger.html but when i start sending traces i'm geting these errors in APM logs:

2020-09-13T20:07:31.202Z WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x148af8e0, ext:63735624444, loc:(*time.Location)(nil)}, Meta:{"pipeline":"apm"}, Fields:{"agent":{"name":"Jaeger","version":"unknown"},"ecs":{"version":"1.5.0"},"labels":{"Lounge":"GatewayService-JaegerProvider","peer.ipv4":"172.21.0.3","peer.port":"80","sampler.param":"true","sampler.type":"const"},"observer":{"ephemeral_id":"98acea66-a148-497b-9eaa-c360900a367a","hostname":"ee30089a825b","id":"f9fd9f44-dbc7-4cdd-9403-54576dba007a","type":"apm-server","version":"7.9.1","version_major":7},"processor":{"event":"transaction","name":"transaction"},"service":{"language":{"name":"unknown"},"name":"Gateway-Service"},"timestamp":{"us":1600027644344652},"trace":{"id":"163470aa489b76d9"},"transaction":{"duration":{"us":5532199},"id":"163470aa489b76d9","name":"GET: /csrfd","result":"Success","sampled":true,"type":"fpm"},"url":{"full":"http:///csrfd","original":"/csrfd","path":"/csrfd","scheme":"http"}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"class_cast_exception","reason":"class org.elasticsearch.index.mapper.ScaledFloatFieldMapper cannot be cast to class org.elasticsearch.index.mapper.ObjectMapper (org.elasticsearch.index.mapper.ScaledFloatFieldMapper is in unnamed module of loader java.net.FactoryURLClassLoader @50a13c2f; org.elasticsearch.index.mapper.ObjectMapper is in unnamed module of loader 'app')"}}

When i switch to jaeger-collector instead of APM - it gets all the tracing correctly.

Here is my docker-compose.yml:

  version: "3"
    services:
      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
        networks:
          - author-book-network
        ports:
          - "9200:9200"
          - "9300:9300"
        restart: on-failure
        environment:
          - cluster.name=jaeger-cluster
          - discovery.type=single-node
          - http.host=0.0.0.0
          - transport.host=127.0.0.1
          - ES_JAVA_OPTS=-Xms1000m -Xmx1000m
          - xpack.security.enabled=false
        volumes:
          - esdata:/usr/share/elasticsearch/data

      kibana:
        image: docker.elastic.co/kibana/kibana:7.9.1
        ports:
          - "5601:5601"
        environment:
          - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
          - SERVER_NAME=kibana
          #- ES.TAGS-AS-FIELDS.ALL=true
          #- SERVER_HOST=0.0.0.0
        depends_on:
          - elasticsearch
        networks:
          - author-book-network

      apm-server:
        image: docker.elastic.co/apm/apm-server:7.9.1
        depends_on:
          - elasticsearch
          - kibana
        cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
        cap_drop: ["ALL"]
        ports:
        - 8200:8200
        networks:
        - author-book-network
        command: >
           apm-server -e
             -E apm-server.jaeger.grpc.enabled=true
             -E apm-server.jaeger.grpc.host=":14250"
             -E apm-server.rum.enabled=true
             -E setup.kibana.host=kibana:5601
             -E setup.template.settings.index.number_of_replicas=0
             -E apm-server.kibana.enabled=true
             -E apm-server.kibana.host=kibana:5601
             -E output.elasticsearch.hosts=["elasticsearch:9200"]
             -E ELASTIC_APM_LOG_FILE=stderr
             -E ELASTIC_APM_LOG_FILE=debug
        # healthcheck:
        #   interval: 10s
        #   retries: 12
        #   test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/

      # jaeger-collector:
      #   image: jaegertracing/jaeger-collector
      #   ports:
      #     #- "14269:14269"
      #     - "14250:14250"
      #     - "14268:14268"
      #     - "14267:14267"
      #     - "9411:9411"
      #   networks:
      #     - author-book-network
      #   restart: on-failure
      #   environment:
      #     - SPAN_STORAGE_TYPE=elasticsearch
      #   command: [
      #     "--es.server-urls=http://elasticsearch:9200",
      #     "--es.num-shards=1",
      #     "--es.num-replicas=0",
      #     "--log-level=error"
      #   ]
      #   depends_on:
      #     - elasticsearch

      jaeger-agent:
        image: jaegertracing/jaeger-agent
        hostname: jaeger-agent
        #command: ["--reporter.grpc.host-port=jaeger-collector:14250"]
        command: ["--reporter.grpc.host-port=apm-server:14250"]
        ports:
          - "5775:5775/udp"
          - "6831:6831/udp"
          - "6832:6832/udp"
          - "5778:5778"
        networks:
          - author-book-network
        restart: on-failure
        environment:
          - SPAN_STORAGE_TYPE=elasticsearch
        depends_on:
          #- jaeger-collector
          - apm-server

      jaeger-query:
        image: jaegertracing/jaeger-query
        environment:
          - SPAN_STORAGE_TYPE=elasticsearch
          - no_proxy=localhost
        ports:
          - "16686:16686"
          - "16687:16687"
        networks:
          - author-book-network
        restart: on-failure
        command: [
          "--es.server-urls=http://elasticsearch:9200",
          "--span-storage.type=elasticsearch",
          "--log-level=debug"
        ]
        depends_on:
          - jaeger-agent

    volumes:
      esdata:
        driver: local

    networks:
      author-book-network:
        external: true

Steps to reproduce:

1. Install Elasticsearch, Kibana and APM through docker-compose
2. Use Jaeger-Agent to send tracing to APM

[axw (Andrew Wilkins)]

Welcome to the forum @Amirtheahmed!

Do you have multiple services instrumented? That sort of error would occur when the instrumentation provides two different data types for the same attribute.

If this is the case, then I think the simplest short-term solution would be to split services into multiple indices, to avoid these kinds of mapping conflicts:

Storing APM data in custom/specific indices

If you'd like to split your APM indices by the service, you can use the advice from this topic. You can change the index name by setting either output.elasticsearch.index or output.elasticsearch.indices. These configuration variables take an index name "format string", which can reference fields the events. For your use-case, you can include %{[service.name]} in the format string to create an index per service. e.g. output.elasticsearch: hosts: ["http://localhost:9200"] indices: - inde…

[Amirtheahmed (Amir Ahmed)]

Hi and thanks for your reply,
Yes i have multiple services instrumented and i will look into the solution you provided when dealing with multiple services. But the error log i shared is from single service with a single span trace (gateway-service) as i deliberately turned off all other services for debugging sake yet still class org.elasticsearch.index.mapper.ScaledFloatFieldMapper cannot be cast to class org.elasticsearch.index.mapper.ObjectMapper persists.

Here is the result trace i see when i use pure jaeger-agent -> jaeger-collector -> jaeger-ui (with elasticsearch as storage): Note! It is sending only single span

[axw (Andrew Wilkins)]

It looks like you found a bug, sorry to say. I've opened https://github.com/elastic/apm-server/issues/4190 to get this fixed. I made an educated guess that you're using https://github.com/jukylin/jaeger-php; please correct me in the issue if I got that wrong.

[Amirtheahmed (Amir Ahmed)]

yes am using https://github.com/jukylin/jaeger-php;

[Amirtheahmed (Amir Ahmed)]

Is it fixed now ?

[axw (Andrew Wilkins)]

I've merged a fix, which will go into the next release. There should be a 7.9.2 release in the next 1-2 weeks.

If you would like to test it sooner, it's possible but requires building apm-server yourself:

• install Make and Go 1.13.10
• clone https://github.com/elastic/apm-server/tree/7.9
• run make release TYPES=docker
• run docker image load -i build/distributions/apm-server-7.9.2-linux-amd64.docker.tar.gz

[Amirtheahmed (Amir Ahmed)]

Thank you very much, i built apm-server myself as you outlined and the errors has gone away.

[axw (Andrew Wilkins)]

Glad to hear it! Thanks for testing and reporting back.