Beats indexes not being created after upgrading stack from oss-7.9.3 => basic-7.12.1

danpatdav · April 30, 2021, 2:54pm

First post, so please feel free to let me know how things work around here.

For the past several months I have been using OSS Elasticsearch + Beats to pull metrics from a few systems. The beats run in docker on the systems and send data to a small (3 node) Elasticsearch cluster also running in containers (Openshift).

This week I upgraded everything to standard Elasticsearch (basic license) and the latest 7.12.1. While cluster and beats all seem to be running happily, I am not getting data into system anymore.

I am an ELK novice, but here are a few of the things I have observed:

shards and indexes are not being created
I am seeing some 404 errors in the proxy server that passes traffic into the cluster
I also saw some strangely formatted URL in the logs
I exported/imported my configuration when rebuilding the cluster (via Kibana)
I add an authentication header to beats data and use a proxy in front of my ES cluster to allow/deny traffic based on this header

Here are some of the proxy logs that may provide insight.

"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/metricbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"GET /_cat/templates/auditbeat-7.12.1 HTTP/1.1" 200 116 "-" "Go-http-client/1.1"
"GET /_cat/templates/packetbeat-7.12.1 HTTP/1.1" 200 117 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"PUT /%3Cpacketbeat-7.12.1-%7Bnow%2Fd%7D-000001%3E HTTP/1.1" 404 196 "-" "Go-http-client/1.1"
"GET /_ilm/policy/auditbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /auditbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/auditbeat-7.12.1 HTTP/1.1" 404 83 "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/packetbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/auditbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /auditbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/metricbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"

And here is a snippet from one of the ES instances:

"Caused by: org.elasticsearch.index.IndexNotFoundException: no such index [metricbeat-7.12.1]",

And here is a recurring chunk of logs from one of the metricbeat instances, with the URL redacted:

2021-04-30T16:23:05.218Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":35780,"time":{"ms":4}},"total":{"ticks":163820,"time":{"ms":19},"value":163820},"user":{"ticks":128040,"time":{"ms":15}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":22},"info":{"ephemeral_id":"c3356daa-6b86-4255-b336-ca137352d1f7","uptime":{"ms":70920286}},"memstats":{"gc_next":113093376,"memory_alloc":59564168,"memory_total":4122005024,"rss":151179264},"runtime":{"goroutines":122}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"read":{"errors":1}},"pipeline":{"clients":17,"events":{"active":4132}}},"system":{"load":{"1":1.02,"15":1.04,"5":1.1,"norm":{"1":0.255,"15":0.26,"5":0.275}}}}}}
2021-04-30T16:23:11.930Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(elasticsearch(https://xxxxxxxxxxxxxxxxxxxxxxxxx:443)): Connection marked as failed because the onConnect callback failed: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2021-04-30T16:23:11.930Z        INFO    [publisher_pipeline_output]     pipeline/output.go:145  Attempting to reconnect to backoff(elasticsearch(https://xxxxxxxxxxxxxxxxxxxxxxxxxx:443)) with 1560 reconnect attempt(s)
2021-04-30T16:23:11.930Z        INFO    [publisher]     pipeline/retry.go:213   retryer: send wait signal to consumer
2021-04-30T16:23:11.930Z        INFO    [publisher]     pipeline/retry.go:217     done
2021-04-30T16:23:12.041Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.12.1
2021-04-30T16:23:12.080Z        INFO    [license]       licenser/es_callback.go:51      Elasticsearch license: Basic
2021-04-30T16:23:12.080Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2021-04-30T16:23:12.091Z        INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2021-04-30T16:23:12.091Z        INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2021-04-30T16:23:12.091Z        INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{metricbeat-7.12.1 {now/d}-000001}' as ILM is enabled.
2021-04-30T16:23:12.091Z        INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'metricbeat-7.12.1-*' as ILM is enabled.
2021-04-30T16:23:12.091Z        INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.12.1 {now/d}-000001} as ILM is enabled.
2021-04-30T16:23:12.091Z        INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-30T16:23:12.124Z        INFO    template/load.go:97     Template metricbeat-7.12.1 already exists and will not be overwritten.
2021-04-30T16:23:12.125Z        INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.

So it seems like I am not creating the expected indexes after the upgrade. Not clear if this is due to the 7.9 => 7.12 upgrade or the switch from OSS to basic.

I would appreciate ideas about how to troubleshoot this issue, as well as any insight on migrating from OSS to basic license.

danpatdav · May 11, 2021, 12:44pm

Reverted to OSS 7.10.2 and everything works fine.

leandrojmp · May 11, 2021, 1:55pm

It looks like a conflict with the ILM feature that does not exist in the OSS version, but is enabled by default when you use the Basic license.

You can disable it in beats using:

setup.ilm.enabled: false

system · June 8, 2021, 3:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.