First post, so please feel free to let me know how things work around here. 
For the past several months I have been using OSS Elasticsearch + Beats to pull metrics from a few systems. The beats run in docker on the systems and send data to a small (3 node) Elasticsearch cluster also running in containers (Openshift).
This week I upgraded everything to standard Elasticsearch (basic license) and the latest 7.12.1. While cluster and beats all seem to be running happily, I am not getting data into system anymore.
I am an ELK novice, but here are a few of the things I have observed:
-
shards and indexes are not being created
-
I am seeing some 404 errors in the proxy server that passes traffic into the cluster
-
I also saw some strangely formatted URL in the logs
-
I exported/imported my configuration when rebuilding the cluster (via Kibana)
-
I add an authentication header to beats data and use a proxy in front of my ES cluster to allow/deny traffic based on this header
Here are some of the proxy logs that may provide insight.
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/metricbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"GET /_cat/templates/auditbeat-7.12.1 HTTP/1.1" 200 116 "-" "Go-http-client/1.1"
"GET /_cat/templates/packetbeat-7.12.1 HTTP/1.1" 200 117 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"PUT /%3Cpacketbeat-7.12.1-%7Bnow%2Fd%7D-000001%3E HTTP/1.1" 404 196 "-" "Go-http-client/1.1"
"GET /_ilm/policy/auditbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /auditbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/auditbeat-7.12.1 HTTP/1.1" 404 83 "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/packetbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/auditbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /auditbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/metricbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
And here is a snippet from one of the ES instances:
"Caused by: org.elasticsearch.index.IndexNotFoundException: no such index [metricbeat-7.12.1]",
And here is a recurring chunk of logs from one of the metricbeat instances, with the URL redacted:
2021-04-30T16:23:05.218Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":35780,"time":{"ms":4}},"total":{"ticks":163820,"time":{"ms":19},"value":163820},"user":{"ticks":128040,"time":{"ms":15}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":22},"info":{"ephemeral_id":"c3356daa-6b86-4255-b336-ca137352d1f7","uptime":{"ms":70920286}},"memstats":{"gc_next":113093376,"memory_alloc":59564168,"memory_total":4122005024,"rss":151179264},"runtime":{"goroutines":122}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"read":{"errors":1}},"pipeline":{"clients":17,"events":{"active":4132}}},"system":{"load":{"1":1.02,"15":1.04,"5":1.1,"norm":{"1":0.255,"15":0.26,"5":0.275}}}}}}
2021-04-30T16:23:11.930Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://xxxxxxxxxxxxxxxxxxxxxxxxx:443)): Connection marked as failed because the onConnect callback failed: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
2021-04-30T16:23:11.930Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://xxxxxxxxxxxxxxxxxxxxxxxxxx:443)) with 1560 reconnect attempt(s)
2021-04-30T16:23:11.930Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
2021-04-30T16:23:11.930Z INFO [publisher] pipeline/retry.go:217 done
2021-04-30T16:23:12.041Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.12.1
2021-04-30T16:23:12.080Z INFO [license] licenser/es_callback.go:51 Elasticsearch license: Basic
2021-04-30T16:23:12.080Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-04-30T16:23:12.091Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-7.12.1 {now/d}-000001}' as ILM is enabled.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-7.12.1-*' as ILM is enabled.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.12.1 {now/d}-000001} as ILM is enabled.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-30T16:23:12.124Z INFO template/load.go:97 Template metricbeat-7.12.1 already exists and will not be overwritten.
2021-04-30T16:23:12.125Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
So it seems like I am not creating the expected indexes after the upgrade. Not clear if this is due to the 7.9 => 7.12 upgrade or the switch from OSS to basic.
I would appreciate ideas about how to troubleshoot this issue, as well as any insight on migrating from OSS to basic license.