Kibana version: 8.5.0
Elasticsearch version: 8.5.0
APM Server version: 8.5.0
APM Agent language and version: Java, 1.35.0
Browser version: Chrome, 105.0.5195.102
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Using ECK Operator 2.5.0
Fresh install or upgraded from other version?
Fresh install
Is there anything special in your setup?
-
This is deployed in kubernetes. No loadbalancers, but exposed node port for apm agents to connect to apm server.
-
No fleet server was configured and no elastic agent deployed
-
Enabled APM integration from the kibana to create ingest pipelines and templates.
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
The apm server was deployed using the eck operator and expected to start and ingest traces. But it fails in pre-condition with following error: message":"precondition failed: error querying cluster_uuid: status_code=401
Steps to reproduce:
- Install Elasticsearch using eck with following settings enabled
-
xpack.security.enabled: true
-
xpack.security.authc.api_key.enabled: true
- Installed Kibana using eck with following settings enabled
- xpack.encryptedSavedObjects.encryptionKey: "aaaa_bbbb_cccc_dddd_eeee_aaaa_bbbb_cccc"
- Installed apm using following definition:
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server
namespace: monitoring
spec:
version: 8.5.0
count: 1
secureSettings:
- secretName: elasticsearch-es-elastic-user
http:
tls:
selfSignedCertificate:
disabled: true
config:
output:
elasticsearch:
hosts: ["elasticsearch-es-http.monitoring:9200"]
protocol: "https"
api_key: "5uGxo-V6TEans0vL4G0yqA"
ssl.certificate_verification: false
ssl.certificate_authorities: ["/usr/share/apm-server/config/elasticsearch-ca/tls.crt"]
kibana:
host: kibana-kb-http.monitoring.svc:5601
protocol: http
path: /kibana
podTemplate:
spec:
containers:
- name: apm-server
volumeMounts:
- mountPath: /usr/share/apm-server/config/elasticsearch-ca
name: elasticsearch-ca
readOnly: true
volumes:
- name: elasticsearch-ca
secret:
defaultMode: 420
optional: false
secretName: elasticsearch-es-http-ca-internal
- Have also tried with username and password in outpu.elasticsearch but same error
Errors in browser console (if relevant): NA
Provide logs and/or server output (if relevant):
APM Server Logs:
{"log.level":"error","@timestamp":"2023-01-13T06:55:17.053Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}
Logs from Agent:
{"log.level":"info","@timestamp":"2023-01-13T06:40:34.356Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (platform-service)","source.address":"192.168.224.56","http.request.id":"52db6ef7-22af-4275-9fc8-b46372e47ad0","event.duration":10025646449,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-13T06:40:43.175Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (platform-service)","source.address":"192.168.224.56","http.request.id":"f60243db-c017-4d7c-a385-620f24d90e43","event.duration":276653,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}