Elasticsearch Multi-Tier Architecture – Master/Hot/Frozen

Hello,

I want deploy Elasticsearch Multi-Tier Architecture – Master/Hot/Frozen on prime.

Please could you help me with documents or blog.

Best regards.

I think you can start from this page: Data tiers | Elasticsearch Guide [8.13] | Elastic

Thank you,
I have three nodes on premise.

  1. master: I put node.roles: [ data, master, voting_only ] I can start it.
  2. Hot: I put node.roles: [ data_hot ] I can start it.
  3. Frozen: when I put node.roles: [ data_frozen ] I can't start it.

I do no where is a problem.

Regards.

I have a few comments:

  • You need at least one master eligible node that is not voting_only, so change this.
  • If you are going to have 3 nodes in the cluster, make all of them master eligible. If you have 3 master eligible nodes you can handle one of the nodes going down without losing the cluster.

I believe frozen nodes require a commercial license, so if you want tiers within such a small cluster, instead make it hot-warm.

What is the rationale behind setting up a tiered architecture? Are there any difference in hardware between the nodes?

Thank you,
Juste for test.
I prepare a big cluster.

Three nodes Master.
Three nodes Hot.
Three nodes Frozen.

If you have recommendation or docs please let me know.

Regards.

OK, that makes more sense. Make sure you make the 3 dedicated master nodes proper maters and not voting only.

Can you answer this?

I would recommend setting up a hot-warm architecture unless you are going to have a commercial license.

Just for HA and critical environment.

That still does not answer my questions.

What is the rationale behind setting up a tiered architecture?

Just for HA and critical environment.

Are there any difference in hardware between the nodes?

No difference in hardware between the nodes.

The main benefit of a tiered architecture is that you can optimise the cluster by having different hardware for the different tiers. If all your nodes have the same hardware profile I would probably recommend deploying 3 dedicated master nodes (or master/data if the hardware is the same for all 9 nodes) together with 6 regular data nodes (no hot/warm tiering). This will give you HA, which is what you are looking for and be easy to manage.

Just because Elasticsearch supports voting only master nodes and flexible tierings it does not mean that this is something you should use for a specific use case/deployment.

Thank you,
Please do you have deployment step by step with three tier Master, Hot, Frozen.

Just create all your nodes with the right roles. Once they have all joined the cluster, you can define your ilm policies.

I created with elasticsearch.yml
1)Master node

role.nodes: [ master ]

  1. Hot node

node.roles: [ data_hot ]

  1. Frozen node

node.roles: [ data_frozen ]. I can't start service

Have a look at these docs. I believe the frozen tier require a commercial license (do you have this?), so that may be why that node does not come up. I would expect there to be something in the logs though, so look for that. Try changing it to a warm node instead and see if that works.

I want to use the trail version, if it is good we will buy it.

With Hot node it's fine.

According to this page the trial license should give access to all subscription features. According to the subscriptions page searchable snapshots, which the frozen tier relies on, requires an enterprise level license, so it should work. Have you activated the trial license before you start up the frozen node?

I am not sure this interpretation is correct though, so hope someone from Elastic can confim or clarify.

1 Like

Could you share the full logs of the frozen node please?

[2024-03-31T14:03:28,817][INFO ][o.e.n.NativeAccess ] [elk3] Using [jdk] native provider and native methods for [Linux]
[2024-03-31T14:03:29,533][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [elk3] Java vector incubator API enabled; uses preferr
[2024-03-31T14:03:30,831][INFO ][o.e.n.Node ] [elk3] version[8.13.0], pid[1748], build[rpm/09df99393193b2c53d9283-22T03:35:46.757803203Z], OS[Linux/5.14.0-362.24.1.el9_3.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/22/22
[2024-03-31T14:03:30,836][INFO ][o.e.n.Node ] [elk3] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK
[2024-03-31T14:03:30,836][INFO ][o.e.n.Node ] [elk3] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.netwotl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosyeInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlolse, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/jaeallocate, --enable-native-access=org.elasticsearch.nativeaccess, -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log, , -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-5907291211499219004, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOunOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,g=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms1814m, -Xmx1814m, -XX:MaxDirectMemorySizeonSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, --module-path=/usr/share/elasticsearch/lib, --add-mles=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2024-03-31T14:03:30,837][INFO ][o.e.n.Node ] [elk3] Default Locale [en]
[2024-03-31T14:03:37,103][INFO ][o.e.p.PluginsService ] [elk3] loaded module [repository-url]
[2024-03-31T14:03:37,104][INFO ][o.e.p.PluginsService ] [elk3] loaded module [rest-root]
[2024-03-31T14:03:37,104][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-core]
[2024-03-31T14:03:37,104][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-redact]
[2024-03-31T14:03:37,105][INFO ][o.e.p.PluginsService ] [elk3] loaded module [ingest-user-agent]
[2024-03-31T14:03:37,105][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-async-search]
[2024-03-31T14:03:37,105][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-monitoring]
[2024-03-31T14:03:37,106][INFO ][o.e.p.PluginsService ] [elk3] loaded module [repository-s3]
[2024-03-31T14:03:37,106][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-analytics]
[2024-03-31T14:03:37,107][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-ent-search]
[2024-03-31T14:03:37,107][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-autoscaling]
[2024-03-31T14:03:37,107][INFO ][o.e.p.PluginsService ] [elk3] loaded module [lang-painless]
[2024-03-31T14:03:37,108][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-ml]
[2024-03-31T14:03:37,108][INFO ][o.e.p.PluginsService ] [elk3] loaded module [lang-mustache]
[2024-03-31T14:03:37,109][INFO ][o.e.p.PluginsService ] [elk3] loaded module [legacy-geo]
[2024-03-31T14:03:37,109][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-ql]
[2024-03-31T14:03:37,110][INFO ][o.e.p.PluginsService ] [elk3] loaded module [rank-rrf]
[2024-03-31T14:03:37,110][INFO ][o.e.p.PluginsService ] [elk3] loaded module [analysis-common]
[2024-03-31T14:03:37,111][INFO ][o.e.p.PluginsService ] [elk3] loaded module [health-shards-availability]
[2024-03-31T14:03:37,111][INFO ][o.e.p.PluginsService ] [elk3] loaded module [transport-netty4]
[2024-03-31T14:03:37,112][INFO ][o.e.p.PluginsService ] [elk3] loaded module [aggregations]
[2024-03-31T14:03:37,113][INFO ][o.e.p.PluginsService ] [elk3] loaded module [ingest-common]
[2024-03-31T14:03:37,113][INFO ][o.e.p.PluginsService ] [elk3] loaded module [frozen-indices]
[2024-03-31T14:03:37,113][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-identity-provider]
[2024-03-31T14:03:37,114][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-shutdown]
[2024-03-31T14:03:37,114][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-text-structure]
[2024-03-31T14:03:37,114][INFO ][o.e.p.PluginsService ] [elk3] loaded module [snapshot-repo-test-kit]
[2024-03-31T14:03:37,115][INFO ][o.e.p.PluginsService ] [elk3] loaded module [ml-package-loader]
[2024-03-31T14:03:37,115][INFO ][o.e.p.PluginsService ] [elk3] loaded module [kibana]
[2024-03-31T14:03:37,119][INFO ][o.e.p.PluginsService ] [elk3] loaded module [constant-keyword]
[2024-03-31T14:03:37,120][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-logstash]
[2024-03-31T14:03:37,121][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-ccr]
[2024-03-31T14:03:37,121][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-graph]
[2024-03-31T14:03:37,122][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-esql]
[2024-03-31T14:03:37,123][INFO ][o.e.p.PluginsService ] [elk3] loaded module [parent-join]
[2024-03-31T14:03:37,123][INFO ][o.e.p.PluginsService ] [elk3] loaded module [counted-keyword]
[2024-03-31T14:03:37,125][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-enrich]
[2024-03-31T14:03:37,127][INFO ][o.e.p.PluginsService ] [elk3] loaded module [repositories-metering-api]
[2024-03-31T14:03:37,128][INFO ][o.e.p.PluginsService ] [elk3] loaded module [transform]
[2024-03-31T14:03:37,128][INFO ][o.e.p.PluginsService ] [elk3] loaded module [repository-azure]
[2024-03-31T14:03:37,128][INFO ][o.e.p.PluginsService ] [elk3] loaded module [repository-gcs]
[2024-03-31T14:03:37,129][INFO ][o.e.p.PluginsService ] [elk3] loaded module [spatial]
[2024-03-31T14:03:37,130][INFO ][o.e.p.PluginsService ] [elk3] loaded module [apm]
[2024-03-31T14:03:37,132][INFO ][o.e.p.PluginsService ] [elk3] loaded module [mapper-extras]
[2024-03-31T14:03:37,132][INFO ][o.e.p.PluginsService ] [elk3] loaded module [mapper-version]
[2024-03-31T14:03:37,133][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-rollup]
[2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService ] [elk3] loaded module [percolator]
[2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService ] [elk3] loaded module [data-streams]
[2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-stack]
[2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService ] [elk3] loaded module [rank-eval]
[2024-03-31T14:03:37,135][INFO ][o.e.p.PluginsService ] [elk3] loaded module [reindex]
[2024-03-31T14:03:37,136][INFO ][o.e.p.PluginsService ] [elk3] loaded module [systemd]
[2024-03-31T14:03:37,138][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-security]
[2024-03-31T14:03:37,138][INFO ][o.e.p.PluginsService ] [elk3] loaded module [blob-cache]
[2024-03-31T14:03:37,140][INFO ][o.e.p.PluginsService ] [elk3] loaded module [searchable-snapshots]
[2024-03-31T14:03:37,142][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-slm]
[2024-03-31T14:03:37,142][INFO ][o.e.p.PluginsService ] [elk3] loaded module [snapshot-based-recoveries]
[2024-03-31T14:03:37,149][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-watcher]
[2024-03-31T14:03:37,152][INFO ][o.e.p.PluginsService ] [elk3] loaded module [old-lucene-versions]
[2024-03-31T14:03:37,153][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-ilm]
[2024-03-31T14:03:37,153][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-inference]
[2024-03-31T14:03:37,153][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-voting-only-node]
[2024-03-31T14:03:37,154][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-deprecation]
[2024-03-31T14:03:37,159][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-fleet]
[2024-03-31T14:03:37,159][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-aggregate-metric]
[2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-downsample]
[2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-profiling]
[2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService ] [elk3] loaded module [ingest-geoip]
[2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-write-load-forecaster]
[2024-03-31T14:03:37,161][INFO ][o.e.p.PluginsService ] [elk3] loaded module [search-business-rules]
[2024-03-31T14:03:37,161][INFO ][o.e.p.PluginsService ] [elk3] loaded module [ingest-attachment]
[2024-03-31T14:03:37,162][INFO ][o.e.p.PluginsService ] [elk3] loaded module [wildcard]
[2024-03-31T14:03:37,162][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-apm-data]
[2024-03-31T14:03:37,162][INFO ][o.e.p.PluginsService ] [elk3] loaded module [unsigned-long]
[2024-03-31T14:03:37,163][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-sql]
[2024-03-31T14:03:37,163][INFO ][o.e.p.PluginsService ] [elk3] loaded module [runtime-fields-common]
[2024-03-31T14:03:37,163][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-async]
[2024-03-31T14:03:37,164][INFO ][o.e.p.PluginsService ] [elk3] loaded module [vector-tile]
[2024-03-31T14:03:37,164][INFO ][o.e.p.PluginsService ] [elk3] loaded module [lang-expression]
[2024-03-31T14:03:37,165][INFO ][o.e.p.PluginsService ] [elk3] loaded module [x-pack-eql]
[2024-03-31T14:03:38,675][INFO ][o.e.e.NodeEnvironment ] [elk3] using [1] data paths, mounts [[/ (/dev/mapper/almalinux-roo.3gb], net total_space [16.9gb], types [xfs]
[2024-03-31T14:03:38,676][INFO ][o.e.e.NodeEnvironment ] [elk3] heap size [1.7gb], compressed ordinary object pointers [tru
[2024-03-31T14:03:38,686][INFO ][o.e.n.Node ] [elk3] node name [elk3], node ID [P3QUmeLjQYa7s13tD1OPPA], cluster, roles [data_frozen]
[2024-03-31T14:03:46,698][INFO ][o.e.f.FeatureService ] [elk3] Registered local node features [data_stream.rollover.lazy, cated, features_supported, health.dsl.info, health.extended_repository_indicator, usage.data_tiers.precalculate_stats]
[2024-03-31T14:03:47,835][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk3] [controller/1767] [Main.cc@123] controller (64 bit):6ce29513f20) Copyright (c) 2024 Elasticsearch BV
[2024-03-31T14:03:48,288][INFO ][o.e.t.a.APM ] [elk3] Sending apm metrics is disabled
[2024-03-31T14:03:48,291][INFO ][o.e.t.a.APM ] [elk3] Sending apm tracing is disabled
[2024-03-31T14:03:48,351][INFO ][o.e.x.s.Security ] [elk3] Security is enabled
[2024-03-31T14:03:49,545][INFO ][o.e.x.s.a.s.FileRolesStore] [elk3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-03-31T14:03:50,255][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [elk3] Auto-configuration will not generate a pilt-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the bin/ed tool to set the password for the elastic user.
[2024-03-31T14:03:50,462][ERROR][o.e.b.Elasticsearch ] [elk3] fatal exception while booting Elasticsearch
java.io.UncheckedIOException: java.io.IOException: Not enough free space [15372410880] for cache file of size [16357785600] inrch]
at org.elasticsearch.blobcache.shared.SharedBlobCacheService.(SharedBlobCacheService.java:374) ~[?:?]
at org.elasticsearch.blobcache.shared.SharedBlobCacheService.(SharedBlobCacheService.java:333) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:335) ~[?:
at org.elasticsearch.node.NodeConstruction.lambda$construct$12(NodeConstruction.java:804) ~[elasticsearch-8.13.0.jar:?
at org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:253) ~[elasticsearch-8.13.0.jar:?]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:288) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) ~[?:?]
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:622) ~[?:?]
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:631) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:637) ~[?:?]
at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:642) ~[?:?]
at org.elasticsearch.node.NodeConstruction.construct(NodeConstruction.java:804) ~[elasticsearch-8.13.0.jar:?]
at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:262) ~[elasticsearch-8.13.0.jar:?
at org.elasticsearch.node.Node.(Node.java:192) ~[elasticsearch-8.13.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:237) ~[elasticsearch-8.13.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:237) ~[elasticsearch-8.13.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74) ~[elasticsearch-8.13.0.jar:?]
Caused by: java.io.IOException: Not enough free space [15372410880] for cache file of size [16357785600] in path [/var/lib/ela
at org.elasticsearch.blobcache.shared.SharedBytes.findCacheSnapshotCacheFilePath(SharedBytes.java:137) ~[?:?]
at org.elasticsearch.blobcache.shared.SharedBytes.(SharedBytes.java:79) ~[?:?]
at org.elasticsearch.blobcache.shared.SharedBlobCacheService.(SharedBlobCacheService.java:371) ~[?:?]

Thank you so much.
The problem is Not enough free space

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.