Kibana version:
n/a
Elasticsearch version:
openshift operator 4.6.0-202112140939
APM Server version:
7.16.2
APM Agent language and version:
n/a
Browser version:
n/a
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
ECK operator's APM or standalone APM server
Fresh install or upgraded from other version?
fresh install
Is there anything special in your setup?
Talking to openshift's Elasticsearch in openshift-logging
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I'm trying to get elastic's APM CRD or apm server to talk to openshift's Elasticsearch operator 4.6.0-202112140939. In most cases, I get a compatibility error. If I specify certain older version of APM, I get a compatibility error, which recommends using the OSS version. But the OSS version seems to still give an error. Am I missing something?
With APM server 7.16.2, I get:
{"log.level":"error","@timestamp":"2022-01-04T22:51:17.166Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.openshift-logging.svc:9200)): Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [_license], must not start with '_'.\",\"index_uuid\":\"_na_\",\"index\":\"_license\"}],\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [_license], must not start with '_'.\",\"index_uuid\":\"_na_\",\"index\":\"_license\"},\"status\":400}","service.name":"apm-server","ecs.version":"1.6.0"}
With the APM CRD, specifying older versions, I get similar errors unless I go back to 7.12.1 or certain earlier versions. With those, I get:
{"log.level":"error","@timestamp":"2021-12-06T23:14:37.387Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.openshift-logging.svc:9200)): Connection marked as failed because the onConnect callback failed: Apm-Server requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Apm-Server.","ecs.version":"1.6.0"}
Is there any way to make this work? Either using the OSS or the COTS versions of the APM server? Or maybe we just can't get there from here.
Steps to reproduce:
- See above. Issue happens at startup.
Errors in browser console (if relevant):
n/a
Provide logs and/or server output (if relevant):
See above