# Beats indexes not being created after upgrading stack from oss-7.9.3 =\> basic-7.12.1

**URL:** <https://discuss.elastic.co/t/beats-indexes-not-being-created-after-upgrading-stack-from-oss-7-9-3-basic-7-12-1/271798>\
**Category:** Beats\
**Tags:** docker, metricbeat\
**Created:** [April 30, 2021, 2:54pm UTC](https://discuss.elastic.co/t/beats-indexes-not-being-created-after-upgrading-stack-from-oss-7-9-3-basic-7-12-1/271798 "2021-04-30T14:54:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![danpatdav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danpatdav/32/83222_2.png) [@danpatdav](https://discuss.elastic.co/u/danpatdav)\
**Post date:** [April 30, 2021, 2:54pm UTC](https://discuss.elastic.co/t/beats-indexes-not-being-created-after-upgrading-stack-from-oss-7-9-3-basic-7-12-1/271798/1 "2021-04-30T14:54:00Z")

</div>

First post, so please feel free to let me know how things work around here. 🙂

For the past several months I have been using OSS Elasticsearch + Beats to pull metrics from a few systems. The beats run in docker on the systems and send data to a small (3 node) Elasticsearch cluster also running in containers (Openshift).

This week I upgraded everything to standard Elasticsearch (basic license) and the latest 7.12.1. While cluster and beats all seem to be running happily, I am not getting data into system anymore.

I am an ELK novice, but here are a few of the things I have observed:

- shards and indexes are not being created

- I am seeing some 404 errors in the proxy server that passes traffic into the cluster

- I also saw some strangely formatted URL in the logs

- I exported/imported my configuration when rebuilding the cluster (via Kibana)

- I add an authentication header to beats data and use a proxy in front of my ES cluster to allow/deny traffic based on this header

Here are some of the proxy logs that may provide insight.

```auto
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/metricbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"GET /_cat/templates/auditbeat-7.12.1 HTTP/1.1" 200 116 "-" "Go-http-client/1.1"
"GET /_cat/templates/packetbeat-7.12.1 HTTP/1.1" 200 117 "-" "Go-http-client/1.1"
"GET / HTTP/1.1" 200 331 "-" "Go-http-client/1.1"
"PUT /%3Cpacketbeat-7.12.1-%7Bnow%2Fd%7D-000001%3E HTTP/1.1" 404 196 "-" "Go-http-client/1.1"
"GET /_ilm/policy/auditbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /auditbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/auditbeat-7.12.1 HTTP/1.1" 404 83 "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/packetbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/auditbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /auditbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_license?human=false HTTP/1.1" 200 233 "-" "Go-http-client/1.1"
"GET /_alias/metricbeat-7.12.1 HTTP/1.1" 404 84 "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/metricbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /metricbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"
"GET /_ilm/policy/packetbeat HTTP/1.1" 200 159 "-" "Go-http-client/1.1"
"HEAD /packetbeat-7.12.1 HTTP/1.1" 404 - "-" "Go-http-client/1.1"

```

And here is a snippet from one of the ES instances:

```auto
"Caused by: org.elasticsearch.index.IndexNotFoundException: no such index [metricbeat-7.12.1]",

```

And here is a recurring chunk of logs from one of the metricbeat instances, with the URL redacted:

```auto
2021-04-30T16:23:05.218Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":35780,"time":{"ms":4}},"total":{"ticks":163820,"time":{"ms":19},"value":163820},"user":{"ticks":128040,"time":{"ms":15}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":22},"info":{"ephemeral_id":"c3356daa-6b86-4255-b336-ca137352d1f7","uptime":{"ms":70920286}},"memstats":{"gc_next":113093376,"memory_alloc":59564168,"memory_total":4122005024,"rss":151179264},"runtime":{"goroutines":122}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"read":{"errors":1}},"pipeline":{"clients":17,"events":{"active":4132}}},"system":{"load":{"1":1.02,"15":1.04,"5":1.1,"norm":{"1":0.255,"15":0.26,"5":0.275}}}}}}
2021-04-30T16:23:11.930Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://xxxxxxxxxxxxxxxxxxxxxxxxx:443)): Connection marked as failed because the onConnect callback failed: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2021-04-30T16:23:11.930Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://xxxxxxxxxxxxxxxxxxxxxxxxxx:443)) with 1560 reconnect attempt(s)
2021-04-30T16:23:11.930Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
2021-04-30T16:23:11.930Z INFO [publisher] pipeline/retry.go:217 done
2021-04-30T16:23:12.041Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.12.1
2021-04-30T16:23:12.080Z INFO [license] licenser/es_callback.go:51 Elasticsearch license: Basic
2021-04-30T16:23:12.080Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-04-30T16:23:12.091Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-7.12.1 {now/d}-000001}' as ILM is enabled.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-7.12.1-*' as ILM is enabled.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.12.1 {now/d}-000001} as ILM is enabled.
2021-04-30T16:23:12.091Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-30T16:23:12.124Z INFO template/load.go:97 Template metricbeat-7.12.1 already exists and will not be overwritten.
2021-04-30T16:23:12.125Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.

```

So it seems like I am not creating the expected indexes after the upgrade. Not clear if this is due to the 7.9 =\> 7.12 upgrade or the switch from OSS to basic.

I would appreciate ideas about how to troubleshoot this issue, as well as any insight on migrating from OSS to basic license.

---

<div class="post-metadata">

**Author:** ![danpatdav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danpatdav/32/83222_2.png) [@danpatdav](https://discuss.elastic.co/u/danpatdav)\
**Post date:** [May 11, 2021, 12:44pm UTC](https://discuss.elastic.co/t/beats-indexes-not-being-created-after-upgrading-stack-from-oss-7-9-3-basic-7-12-1/271798/2 "2021-05-11T12:44:15Z")

</div>

Reverted to OSS 7.10.2 and everything works fine.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 11, 2021, 1:55pm UTC](https://discuss.elastic.co/t/beats-indexes-not-being-created-after-upgrading-stack-from-oss-7-9-3-basic-7-12-1/271798/3 "2021-05-11T13:55:08Z")

</div>

It looks like a conflict with the ILM feature that does not exist in the OSS version, but is enabled by default when you use the Basic license.

You can disable it in beats using:

```auto
setup.ilm.enabled: false

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2021, 3:55pm UTC](https://discuss.elastic.co/t/beats-indexes-not-being-created-after-upgrading-stack-from-oss-7-9-3-basic-7-12-1/271798/4 "2021-06-08T15:55:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.