Apm-server configuration problem

Elastic Observability APM
1

Dears,

I need your help. I have problem with configuration apm-server. This is my configuration file:

apm-server:
  host: "node-01:8200"
output.elasticsearch:
  hosts: ["https://node-01:9200", "https://node-02:9200", "https://node-03:9200"]
  username: "elastic"
  password: "${ES_PWD}"
  ssl.certificate: "/etc/apm-server/certs/node-01.crt"
  ssl.key: "/etc/apm-server/certs/node-01.key"
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/apm-server
  name: apm-server
  keepfiles: 7
  permissions: 0600

Log file show errors:

2020-08-13T09:14:44.538+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:99   Attempting to reconnect to backoff(elasticsearch(https://node-02:9200)) with 11 reconnect attempt(s)
2020-08-13T09:14:44.538+0200    DEBUG   [esclientleg]   eslegclient/connection.go:239   ES Ping(url=https://node-02:9200)
2020-08-13T09:14:44.542+0200    DEBUG   [esclientleg]   eslegclient/connection.go:243   Ping request failed with: Get https://node-02:9200: x509: certificate signed by unknown authority
2020-08-13T09:14:58.458+0200    ERROR   [publisher_pipeline_output]     pipeline/output.go:106  Failed to connect to backoff(elasticsearch(https://node-03:9200)): Get https://node-03:9200: x509: certificate signed by unknown authority
2020-08-13T09:14:58.458+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:99   Attempting to reconnect to backoff(elasticsearch(https://node-03:9200)) with 11 reconnect attempt(s)
2020-08-13T09:14:58.458+0200    DEBUG   [esclientleg]   eslegclient/connection.go:239   ES Ping(url=https://node-03:9200)
2020-08-13T09:14:58.474+0200    DEBUG   [esclientleg]   eslegclient/connection.go:243   Ping request failed with: Get https://node-03:9200: x509: certificate signed by unknown authority

Certs are OK. I use them for nodes communication, logstash configuration, metricbeat configuration.

I do not have any idea what is wrong. Do you have any idea what is wrong? Any help?

Best Regards,
Dan

2

It looks to me like you're using self-signed certificates. If that is the case, you should also set output.elasticsearch.ssl.certificate_authorities, as shown in https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#elasticsearch-output

1 Like
3

@axw

Thanks for reply.

I changed the configuration:

apm-server:
  host: "node-01:8200"
output.elasticsearch:
  hosts: ["https://node-01:9200"]
  username: "elastic"
  password: "${ES_PWD}"
  ssl.certificate_authorities: "/etc/apm-server/certs/ca.crt"
  ssl.certificate: "/etc/apm-server/certs/node-01.crt"
  ssl.key: "/etc/apm-server/certs/node-01.key"
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/apm-server
  name: apm-server
  keepfiles: 7
  permissions: 0600

but errors there are still:

2020-08-13T10:01:07.618+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:99   Attempting to reconnect to backoff(elasticsearch(https://node-01:9200)) with 5 reconnect attempt(s)
2020-08-13T10:01:07.618+0200    DEBUG   [esclientleg]   eslegclient/connection.go:239   ES Ping(url=https://node-01:9200)
2020-08-13T10:01:07.623+0200    DEBUG   [esclientleg]   eslegclient/connection.go:243   Ping request failed with: Get https://node-01:9200: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Elastic Certificate Tool Autogenerated CA")
2020-08-13T10:01:59.626+0200    ERROR   [publisher_pipeline_output]     pipeline/output.go:106  Failed to connect to backoff(elasticsearch(https://node-01:9200)): Get https://node-01:9200: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Elastic Certificate Tool Autogenerated CA")
2020-08-13T10:01:59.626+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:99   Attempting to reconnect to backoff(elasticsearch(https://node-01:9200)) with 6 reconnect attempt(s)
2020-08-13T10:01:59.626+0200    DEBUG   [esclientleg]   eslegclient/connection.go:239   ES Ping(url=https://node-01:9200)
2020-08-13T10:01:59.629+0200    DEBUG   [esclientleg]   eslegclient/connection.go:243   Ping request failed with: Get https://node-01:9200: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Elastic Certificate Tool Autogenerated CA")

Dan

4

Is that the ca.crt the one that you used to sign node-01.crt?

I suggest taking apm-server out of the equation for a moment, and verifying the certificate with curl or similar. What happens if you run this?

curl --cacert /etc/apm-server/certs/ca.crt https://node-01:9200
5

@axw

This is very important notice in log:

2020-08-13T10:01:07.623+0200    DEBUG   [esclientleg]   eslegclient/connection.go:243   Ping request failed with: Get https://node-01:9200: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Elastic Certificate Tool Autogenerated CA")

I comparated logstash and apm-server certs again and find that weren't the same. Creation date was different. I copied all certs from Logstash catalog to apm-server catalog and now every thing works well.

Thanks for your help.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.