APM is working, but not User Experience

Hi,

I'm facing some issues in my local apm agent setup.

I'm using Vue as framework. It's not loading any data into user experience however I can see both services running on APM section. I'm not seeing any errors on the console when I refresh the user experience dashboads.

I have to say that when I use the elastic cloud I can see data loaded into user experience dashboards with this same configuration (only changing server_url on the init).

logs:

apm-server_1     | {"log.level":"info","@timestamp":"2021-08-19T13:52:47.382Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"not modified","service.name":"apm-server","event.dataset":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"elasticapm-java/1.25.0","source.address":"172.18.0.1","http.request.body.bytes":439,"http.request.id":"4e1d09df-b3fa-4f75-bd93-3043bc2fb5fe","event.duration":80055700,"http.response.status_code":304,"ecs.version":"1.6.0"}
apm-server_1     | {"log.level":"info","@timestamp":"2021-08-19T13:52:50.076Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","service.name":"apm-server","event.dataset":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.29.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"4f102d15-578b-4767-884b-e1fc5188fea4","event.duration":1438900,"http.response.status_code":200,"ecs.version":"1.6.0"}
apm-server_1     | {"log.level":"info","@timestamp":"2021-08-19T13:53:00.184Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","service.name":"apm-server","event.dataset":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.29.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"6eaf03ae-ad01-48d4-ba01-3e7429c75fff","event.duration":398500,"http.response.status_code":200,"ecs.version":"1.6.0"}
apm-server_1     | {"log.level":"info","@timestamp":"2021-08-19T13:53:10.296Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","service.name":"apm-server","event.dataset":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.29.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"9a4ef5cc-a144-4a7d-9acb-d3debfa0578d","event.duration":1951100,"http.response.status_code":200,"ecs.version":"1.6.0"}
elasticsearch_1  | {"@timestamp":"2021-08-19T13:53:11.843Z", "log.level":"DEPRECATION",  "data_stream.dataset":"deprecation.elasticsearch", "data_stream.namespace":"default", "data_stream.type":"logs", "ecs.version":"1.7", "elasticsearch.event.category":"api", "elasticsearch.http.request.x_opaque_id":"1209a9e7-5581-4853-b27a-d123cde73135", "event.code":"open_system_index_access", "message":"this request accesses system indices: [.async-search, .tasks], but in a future major version, direct access to system indices will be prevented by default" , "service.name":"ES_ECS","process.thread.name":"elasticsearch[333d54448ed8][transport_worker][T#4]","log.logger":"org.elasticsearch.deprecation.cluster.metadata.IndexNameExpressionResolver","event.dataset":"deprecation.elasticsearch","elasticsearch.cluster.uuid":"3U-q71EXQGOqAjHaISjEOg","elasticsearch.node.id":"Tp4nuHneToemix5f2zhetw","elasticsearch.node.name":"333d54448ed8","elasticsearch.cluster.name":"docker-cluster"}
apm-server_1     | {"log.level":"info","@timestamp":"2021-08-19T13:53:16.050Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","service.name":"apm-server","event.dataset":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-java/1.25.0","source.address":"172.18.0.1","http.request.id":"911c2538-1eeb-4b2d-a449-58389e64266a","event.duration":10050176300,"http.response.status_code":202,"ecs.version":"1.6.0"}
kibana_1         | {"type":"log","@timestamp":"2021-08-19T13:53:17+00:00","tags":["error","plugins","apm"],"pid":1196,"message":"Error: Not Found\n    at handler (/usr/share/kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration.js:299:27)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at /usr/share/kibana/x-pack/plugins/apm/server/routes/register_routes/index.js:148:13\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9) {\n  data: null,\n  isBoom: true,\n  isServer: false,\n  output: {\n    statusCode: 404,\n    payload: { statusCode: 404, error: 'Not Found', message: 'Not Found' },\n    headers: {}\n  }\n}"}
elasticsearch_1  | {"@timestamp":"2021-08-19T13:53:17.453Z", "log.level":"DEPRECATION",  "data_stream.dataset":"deprecation.elasticsearch", "data_stream.namespace":"default", "data_stream.type":"logs", "ecs.version":"1.7", "elasticsearch.event.category":"api", "elasticsearch.http.request.x_opaque_id":"714edbdf-0655-449d-94bc-ccdfcd8907f4", "event.code":"get_license_accept_enterprise", "message":"Including [accept_enterprise] in get license requests is deprecated. The parameter will be removed in the next major version" , "service.name":"ES_ECS","process.thread.name":"elasticsearch[333d54448ed8][transport_worker][T#3]","log.logger":"org.elasticsearch.deprecation.xpack.core.rest.action.RestXPackInfoAction","event.dataset":"deprecation.elasticsearch","elasticsearch.cluster.uuid":"3U-q71EXQGOqAjHaISjEOg","elasticsearch.node.id":"Tp4nuHneToemix5f2zhetw","elasticsearch.node.name":"333d54448ed8","elasticsearch.cluster.name":"docker-cluster"}

docker-compose.yml

version: '2.2'
services:
apm-server:
image: docker.elastic.co/apm/apm-server:7.14.0
depends_on:
elasticsearch:
condition: service_healthy
kibana:
condition: service_healthy
cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
cap_drop: ["ALL"]
ports:
- 8200:8200
networks:
- elastic
command: >
apm-server -e
-E apm-server.rum.enabled=true
-E setup.kibana.host=kibana:5601
-E setup.template.settings.index.number_of_replicas=0
-E apm-server.kibana.enabled=true
-E apm-server.kibana.host=kibana:5601
-E output.elasticsearch.hosts=["elasticsearch:9200"]
healthcheck:
interval: 10s
retries: 12
test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
environment:
- bootstrap.memory_lock=true
- cluster.name=docker-cluster
- cluster.routing.allocation.disk.threshold_enabled=false
- discovery.type=single-node
- ES_JAVA_OPTS=-XX:UseAVX=2 -Xms1g -Xmx1g
ulimits:
memlock:
hard: -1
soft: -1
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
healthcheck:
interval: 20s
retries: 10
test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"'

kibana:
image: docker.elastic.co/kibana/kibana:7.14.0
depends_on:
elasticsearch:
condition: service_healthy
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200
ELASTICSEARCH_HOSTS: http://elasticsearch:9200
ports:
- 5601:5601
networks:
- elastic
healthcheck:
interval: 10s
retries: 20
test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:5601/api/status

volumes:
esdata:
driver: local

networks:
elastic:
driver: bridge

Does anyone faced the same issue?

Hi @Pedro_Ferreira ,

Seems like the issue is due to missing hard navigation data. User Experience App currently only works with hard-navigations aka page-load transactions. From the data you posted, I can see only route-change events.

Are you using our Vue Integration module - Vue integration | APM Real User Monitoring JavaScript Agent Reference [5.x] | Elastic ?

I wrote some details about the difference between APM and RUM UI here - RUM agent for angular application not capturing page clicks events - #6 by vigneshshanmugam

Thanks
Vignesh

Hi Vignesh,

Yes, I'm using Vue integration module. Everything works fine on the cloud but not on my local.

package.json
"@elastic/apm-rum-vue": "^1.3.1"

Actually I'm not seeing the page-load as you are referring. An error seems to be thrown at the page-load event.

{
  "accepted": 15,
  "errors": [
    {
      "message": "decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.Sampled: ReadBool: expect t or f,",
      "document": "{\"transaction\":{\"id\":\"02283c6dcb304a39\",\"trace_id\":\"${transaction.traceId}\",\"name\":\"/login\",\"type\":\"page-load\",\"duration\":1003,\"context\":{\"page\":{\"referer\":\"\",\"url\":\"http://localhost:8081/#/login\"},\"response\":{\"transfer_size\":1333,\"encoded_body_size\":1033,\"decoded_body_size\":1033}},\"marks\":{\"agent\":{\"firstContentfulPaint\":856,\"timeToFirstByte\":3,\"domInteractive\":814,\"domComplete\":1002},\"navigationTiming\":{\"fetchStart\":0,\"domainLookupStart\":0,\"domainLookupEnd\":0,\"connectStart\":0,\"connectEnd\":0,\"requestStart\":2,\"responseStart\":3,\"responseEnd\":4,\"domLoading\":15,\"domInteractive\":814,\"domContentLoadedEventStart\":814,\"domContentLoadedEventEnd\":814,\"domComplete\":1002,\"loadEventStart\":1002,\"loadEventEnd\":1002}},\"span_count\":{\"started\":15},\"sampled\":\"${transaction.sampled}\",\"sample_rate\":1,\"experience\":{\"tbt\":649,\"cls\":0.000032947902881246176,\"longtask\":{\"count\":2,\"sum\":749,\"max\":687}}}}"
    }
  ]
}

Does this help to troubleshoot?

Thanks,
Pedro Ferreira

Any update?

Apologies for the delay.

Are you trying to modify the transaction.sampled flag before sending to APM server?

"sampled\":\"${transaction.sampled}\" 

This feels wrong, the server can only accept a boolean value for the sampled flag which is why you are seeing the decoding error. Could you please double check on your end.

Thanks,
Vignesh
Thanks,
Vignesh

Hi Vignesh,

I'm not changing anything...I'm just sending the code to APM server as is!

Thanks,
Pedro Ferreira