Error in elastic search apm during performance testing

Elastic Observability APM
1

Hello i installed an APM in one of my machine and started a spring boot app with apm jar to send the metrics, i am receving the data from the app correctly but when we do performance testing on the app with 1000 requests per minute my apm server is throwing below ERROR in the log file and data is not getting ingested to elasticsearch. This is only happening while doin gperformance testing do i need to change any settings in my config to handle the heavy load.

ERROR [request] beater/handlers.go:384 error handling request {"request_id": "6cca8ddc-4d67-4b12-bdc9-6ff49ec3d263", "error": "while decoding: unexpected EOF"}

2

Hi and thanks for trying out Elastic APM!

Are you using the Java agent? The version 0.6.0 contains a bug which only occurs when the application is under some stress. I’ll release a bug fix ASAP. In the mean time, you can try a current snapshot: https://github.com/elastic/apm-agent-java/blob/master/README.md#snapshots

3

Yes i am using the java agent version 0.6.0

4

Hi Felix is there any expected date when this bug is going to be fixed in the new version

5

Did the snapshot work for you? I’ll release a bug fix this week.

6

This is a direct link to the snapshot: https://oss.sonatype.org/content/repositories/snapshots/co/elastic/apm/elastic-apm-agent/0.7.0-SNAPSHOT/elastic-apm-agent-0.7.0-20180806.093124-36.jar

7

I have released 0.6.1 with a fix for this

8

@felixbarny this fix solved the problem thanks

1 Like
9

Hi Felix,

I have the same problem, but in my case with normal usage, not under heavy load.

It occurs when the payload has more than 1 element,

OK
2018-08-27 12:47:40.587 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server http://sapelk00.bsh.corp.bshg.com:8200
2018-08-27 12:47:40.591 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 202

KO
2018-08-27 12:48:53.587 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 8 elements to APM server http://sapelk00.bsh.corp.bshg.com:8200
2018-08-27 12:48:53.590 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400
2018-08-27 12:48:53.591 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: unexpected EOF

I tried with versions
0.6.0
0.6.2
and the snapshot
0.7.0-20180827.093505-60

with no luck.

I'm using apm-server version 6.3.2

Log in server side
2018-08-27T12:27:26.179+0200 ERROR [request] beater/handlers.go:414 error handling request {"request_id": "34a774a7-b5f7-4b64-bbd6-4bd9b6336e28", "method": "POST", "URL": "/v1/transactions", "content_length": -1, "remote_address": "10.129.14.61", "user-agent": "apm-agent-java unknown", "response_code": 400, "error": "data decoding error: unexpected EOF"}

Any help would be appreciated! :slight_smile:

Thanks in advance,
Raul

10

Hi again,

Forget it. Value of jar to be loaded was not properly changed, and the same jar (0.6.0) was used in all cases.

It's working now with version 0.6.2.

BR,
Raul

1 Like
11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.