Apm has many null points in "Time spent by span type" dashboards

[peng_xing]

I found some error logs in apm, but can not understand the reasons，how should i solve it？

apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:39:00.753Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "03cb02a6-b7e8-4806-9114-f8051703d7ba", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.38.200", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:39:00.760Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "bd36683a-768b-41d2-9ba2-baf10b38e128", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.36.116", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:44:21.961Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "d6275e39-32af-4bea-ba7d-ca1162c3d6ee", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.37.24", "user-agent": "elasticapm-node/3.9.0 elastic-apm-http-client/9.4.2 node/10.16.0", "response_code": 500, "error": "context canceled"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:10:05.150Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "5c7d5027-1e75-4e20-aec5-76c6056da925", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.29.244", "user-agent": "elasticapm-node/3.8.0 elastic-apm-http-client/9.4.1 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}
apm-server-apm-server-685fcfc89d-jztrg apm-server 2021-03-26T03:52:16.986Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "40670ba4-7ee8-4483-89bb-8b1e4bad5d7c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.31.172", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:52:33.816Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "63a7570f-50b7-4441-a831-d6e7c2158471", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.37.33", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:52:48.056Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "1e590f35-278d-4c7a-a0cb-f53261e04ddc", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.27.66", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:52:57.782Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "51e3a146-2152-4616-914c-4ac4d10a2005", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.30.244", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "context canceled"}
apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T04:01:04.948Z	ERROR	[request]	middleware/log_middleware.go:95	internal error	{"request_id": "8b168fbe-3b3e-4492-8f61-21b048987c76", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.20.37.106", "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4", "response_code": 500, "error": "unexpected EOF"}

I can not find error log in elasticsearch.

apm server version: 7.9.3
nodejs agent: elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4

This is my apm server config:

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
spec:
  config:
    output.elasticsearch.bulk_max_size: 4096
    output.elasticsearch.worker: 4
    queue.mem.events: 16384
    setup.template.settings:
      index.number_of_replicas: 1
      index.number_of_shards: 3
  count: 3
  elasticsearchRef:
    name: es-otel
  http:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
    tls:
      certificate: {}
  kibanaRef:
    name: kibana-otel
  podTemplate:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - name: apm-server
        resources:
          limits:
            cpu: 1000m
            memory: 2000Mi
          requests:
            cpu: 600m
            memory: 2000Mi
  version: 7.9.3

[alanstorm]

Sorry to hear about the issue you're running into @peng_xing. I don't know exactly what's going wrong in your situation, but I can explain what the errors you're seeing mean.

apm-server-apm-server-685fcfc89d-rr8wn apm-server 2021-03-26T03:39:00.753Z ERROR [request] middleware/log_middleware.go:95 internal error

{"request_id": "03cb02a6-b7e8-4806-9114-f8051703d7ba", 
 "method": "POST", 
 "URL": "/intake/v2/events", 
 "content_length": -1, 
 "remote_address": "172.20.38.200", 
 "user-agent": "elasticapm-node/3.12.1 elastic-apm-http-client/9.6.0 node/14.15.4",
 "response_code": 500, 
 "error": "unexpected EOF"
}

These errors appear to be from when the agent sends its data to APM Server. The /intake/v2/events URL is the URL that agents send data to. So APM Server is responding with a response code of 500, with usually indicates an error on APM Server's side.

The two next steps I'd take if I was debugging this

1. Are there any errors in your APM Server logs?

2. Is this an application that receives constant traffic? (in other words, is it possible that these gaps are accurately representing the state of the application?)