APM Server version:
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Fresh install or upgraded from other version?
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I am running ECK
eck/eck-operator:1.0.0-beta1 and have a elastic/apm/kibana stack setup across approximately 30 clusters.
On some of these everything seems to be working fine:
On others it looks like so:
and I start seeing
illegal_argument_exception: index.lifecycle.rollover_alias [apm-7.5.0-error] does not point to index [apm-7.5.0-error] in kibana
and the following in the apm server logs:
2019-12-19T02:49:56.716Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://monitoring-es-http.client-env-monitoring.svc:9200)): Connection marked as failed because the onConnect callback failed: resource 'apm-7.5.0-error' exists, but it is not an alias 2019-12-19T02:49:56.716Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://monitoring-es-http.client-env-monitoring.svc:9200)) with 3734 reconnect attempt(s)
I'm assuming these aliases should be getting creating but I am not completely sure on that so I guess that's my first question, do I need to create these aliases manually? If not, I could use a bit of direction in figuring out what is going on.
Steps to reproduce: