EOF while reading

Elastic Observability APM
1

Hello!

I have a problem with debugging apm-server. I take example request body from here.

curl --request POST http://*.*.*.*:*/intake/v2/events -H "Content-Type: application/x-ndjson" -d @payload.ndjson

I get response:

{
  "accepted": 0,
  "errors": [
    {
      "message": "EOF while reading metadata",
      "document": "..."
    }
}

In code I see that exception - link. Golang documentation says:

When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call.

APM-server is installed by the installing the .deb package, not by Fleet-server. Elasticsearch and APM-Server vesrions is 8.4.1.

What an error could caused by? What about this error actually? How can I decide it?

UPD: also I can't enable APM integration in the Kibana :c

image
image1044×267 22.9 KB

2

Hi @leschev ,

try using --data-binary instead of -d:
curl --request POST http://*.*.*.*:*/intake/v2/events -H "Content-Type: application/x-ndjson" --data-binary @payload.ndjson

3

It's done, thanks a lot!

What should I do with problem I updated in the topic? I'm talking about "No Apm Server detected".

4

Happy to hear we solved the first issue.

Have you installed the APM Integration? You can follow the guide for installing the APM Integration. This is required for setting up index templates, ingest pipelines etc and the APM Server won't start ingesting data to Elasticsearch until it can find these assets.

5

I tried to install APM Integration, but when I click on "Check APM Server status" I see this error:

image
image1058×281 23 KB

APM Server is installed as .deb package.

6

You described earlier that you managed to successfully send sample data to the APM Server. Could you verify if these data are actually indexed into Elasticsearch or if there is any error logged in the APM Server logs?

7

Oh, I deeped down to the problem and created and configured roles, users and index templates for apm-server:

  • role apm-server with all (for test) privilleges for metrics-apm*, logs-apm* and traces-apm* and with monitor, cluster:admin/xpack/monitoring/bulk cluster privilleges
  • user apm-server with role apm-server
  • index templates logs-apm, metrics-apm and traces-apm configured as data streams with patterns logs-apm*, metrics-apm* and traces-apm*

It is absolutely true that my test requests are processing cause sample data are indexing. I see indices in the UI and in responses using Search API.

After several apm-server restarts I get a few errors.

IDK why I see this error. I have index template with neccessary index pattern:

Nov 29 10:22:26 fleet-server apm-server[11648]: {"log.level":"error","@timestamp":"2022-11-29T10:22:26.852Z","log.logger":"beater","log.origin":{
"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for in
tegration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason
\":\"index template matching [logs-apm.error] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-
apm.error] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.n
ame":"apm-server","ecs.version":"1.6.0"}

And the second error:

Nov 29 10:22:26 fleet-server apm-server[11648]: {"log.level":"error","@timestamp":"2022-11-29T10:22:26.858Z","log.logger":"kibana","log.origin":{
"file.name":"kibana/connecting_client.go","file.line":79},"message":"failed to obtain connection to Kibana: fail to get the Kibana version: fail
to unmarshal the response from GET https://{KIBANA}:443/api/status. Response: <!DOCTYPE html> <html class=\"login-pf\">  <head>
     <meta charset=\"utf-8\">     <meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />     <meta name=\"robots\" content=\"n
oindex, nofollow\">     <link rel=\"icon\" type=\"image/x-icon\" href=\".... (truncated). Kibana status api returns: invalid character '<' lookin
g for beginning of value","service.name":"apm-server","ecs.version":"1.6.0"}

This happened after I configured apm-server.kibana section in apm-server.yml. I haven't this section before so it possible that it is cause with apm-server integration installing issue. But it is not work. I think cause is enabled SSO on Kibana side. Are there ways to configure apm-server.kibana to work with SSO?

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.