Elasticsearch Multi-Tier Architecture – Master/Hot/Frozen

bouzeghoub · March 28, 2024, 10:17am

Hello,

I want deploy Elasticsearch Multi-Tier Architecture – Master/Hot/Frozen on prime.

Please could you help me with documents or blog.

Best regards.

dadoonet · March 28, 2024, 10:35am

I think you can start from this page: Data tiers | Elasticsearch Guide [8.13] | Elastic

bouzeghoub · March 29, 2024, 10:37am

Thank you,
I have three nodes on premise.

master: I put node.roles: [ data, master, voting_only ] I can start it.
Hot: I put node.roles: [ data_hot ] I can start it.
Frozen: when I put node.roles: [ data_frozen ] I can't start it.

I do no where is a problem.

Regards.

Christian_Dahlqvist · March 29, 2024, 10:46am

I have a few comments:

You need at least one master eligible node that is not voting_only, so change this.
If you are going to have 3 nodes in the cluster, make all of them master eligible. If you have 3 master eligible nodes you can handle one of the nodes going down without losing the cluster.

I believe frozen nodes require a commercial license, so if you want tiers within such a small cluster, instead make it hot-warm.

What is the rationale behind setting up a tiered architecture? Are there any difference in hardware between the nodes?

bouzeghoub · March 29, 2024, 10:57am

Thank you,
Juste for test.
I prepare a big cluster.

Three nodes Master.
Three nodes Hot.
Three nodes Frozen.

If you have recommendation or docs please let me know.

Regards.

Christian_Dahlqvist · March 29, 2024, 11:07am

OK, that makes more sense. Make sure you make the 3 dedicated master nodes proper maters and not voting only.

Can you answer this?

I would recommend setting up a hot-warm architecture unless you are going to have a commercial license.

bouzeghoub · March 29, 2024, 11:32am

Just for HA and critical environment.

Christian_Dahlqvist · March 29, 2024, 11:34am

That still does not answer my questions.

bouzeghoub · March 29, 2024, 1:20pm

What is the rationale behind setting up a tiered architecture?

Just for HA and critical environment.

Are there any difference in hardware between the nodes?

No difference in hardware between the nodes.

Christian_Dahlqvist · March 29, 2024, 1:43pm

The main benefit of a tiered architecture is that you can optimise the cluster by having different hardware for the different tiers. If all your nodes have the same hardware profile I would probably recommend deploying 3 dedicated master nodes (or master/data if the hardware is the same for all 9 nodes) together with 6 regular data nodes (no hot/warm tiering). This will give you HA, which is what you are looking for and be easy to manage.

Just because Elasticsearch supports voting only master nodes and flexible tierings it does not mean that this is something you should use for a specific use case/deployment.

bouzeghoub · March 30, 2024, 5:38am

Thank you,
Please do you have deployment step by step with three tier Master, Hot, Frozen.

dadoonet · March 30, 2024, 8:16am

Just create all your nodes with the right roles. Once they have all joined the cluster, you can define your ilm policies.

bouzeghoub · March 30, 2024, 9:58am

I created with elasticsearch.yml
1)Master node

role.nodes: [ master ]

Hot node

node.roles: [ data_hot ]

Frozen node

node.roles: [ data_frozen ]. I can't start service

Christian_Dahlqvist · March 30, 2024, 11:25am

Have a look at these docs. I believe the frozen tier require a commercial license (do you have this?), so that may be why that node does not come up. I would expect there to be something in the logs though, so look for that. Try changing it to a warm node instead and see if that works.

bouzeghoub · March 30, 2024, 11:34am

I want to use the trail version, if it is good we will buy it.

With Hot node it's fine.

Christian_Dahlqvist · March 30, 2024, 11:42am

According to this page the trial license should give access to all subscription features. According to the subscriptions page searchable snapshots, which the frozen tier relies on, requires an enterprise level license, so it should work. Have you activated the trial license before you start up the frozen node?

I am not sure this interpretation is correct though, so hope someone from Elastic can confim or clarify.

dadoonet · March 30, 2024, 12:48pm

Could you share the full logs of the frozen node please?

bouzeghoub · April 1, 2024, 1:01pm

[2024-03-31T14:03:28,817][INFO ][o.e.n.NativeAccess [2024-03-31T14:03:29,533][INFO ][o.a.l.i.v.PanamaVect [2024-03-31T14:03:30,831][INFO ][o.e.n.Node [2024-03-31T14:03:30,836][INFO ][o.e.n.Node [2024-03-31T14:03:30,836][INFO ][o.e.n.Node [2024-03-31T14:03:30,837][INFO ][o.e.n.Node [2024-03-31T14:03:37,103][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,104][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,104][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,104][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,105][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,105][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,105][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,106][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,106][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,107][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,107][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,107][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,108][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,108][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,109][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,109][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,110][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,110][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,111][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,111][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,112][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,113][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,113][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,113][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,114][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,114][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,114][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,115][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,115][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,119][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,120][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,121][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,121][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,122][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,123][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,123][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,125][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,127][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,128][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,128][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,128][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,129][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,130][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,132][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,132][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,133][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,134][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,135][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,136][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,138][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,138][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,140][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,142][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,142][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,149][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,152][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,153][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,153][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,153][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,154][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,159][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,159][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,160][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,161][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,161][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,162][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,162][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,162][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,163][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,163][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,163][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,164][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,164][INFO ][o.e.p.PluginsService [2024-03-31T14:03:37,165][INFO ][o.e.p.PluginsService [2024-03-31T14:03:38,675][INFO ][o.e.e.NodeEnvironment [2024-03-31T14:03:38,676][INFO ][o.e.e.NodeEnvironment [2024-03-31T14:03:38,686][INFO ][o.e.n.Node [2024-03-31T14:03:46,698][INFO ][o.e.f.FeatureService [2024-03-31T14:03:47,835][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [2024-03-31T14:03:48,288][INFO ][o.e.t.a.APM [2024-03-31T14:03:48,291][INFO ][o.e.t.a.APM [2024-03-31T14:03:48,351][INFO ][o.e.x.s.Security [2024-03-31T14:03:49,545][INFO ][o.e.x.s.a.s.FileRolesStore] [2024-03-31T14:03:50,255][INFO ][o.e.x.s.InitialNodeS [2024-03-31T14:03:50,462][ERROR][o.e.b.Elasticsearch java.io.UncheckedIOException: java.io.IOException: at org.elasticsearch.blobcache.shared.SharedBlobCache at org.elasticsearch.blobcache.shared.SharedBlobCache at org.elasticsearch.xpack.searchablesnapshots.Search at org.elasticsearch.node.NodeConstruction.lambda$con at org.elasticsearch.plugins.PluginsService.lambda$fl at java.util.stream.ReferencePipeline$7$1.accept(Refe at java.util.stream.ReferencePipeline$3$1.accept(Refe at java.util.AbstractList$RandomAccessSpliterator.for at java.util.stream.AbstractPipeline.copyInto(Abstrac at java.util.stream.AbstractPipeline.wrapAndCopyInto( at java.util.stream.AbstractPipeline.evaluate(Abstrac at java.util.stream.AbstractPipeline.evaluateToArrayN at java.util.stream.ReferencePipeline.toArray(Referen at java.util.stream.ReferencePipeline.toArray(Referen at java.util.stream.ReferencePipeline.toList(Referenc at org.elasticsearch.node.NodeConstruction.construct( at org.elasticsearch.node.NodeConstruction.prepareCon at org.elasticsearch.node.Node.(Node.java:192) at org.elasticsearch.bootstrap.Elasticsearch$2.(Elast at org.elasticsearch.bootstrap.Elasticsearch.initPhas at org.elasticsearch.bootstrap.Elasticsearch.main(Ela Caused by: java.io.IOException: Not enough free at org.elasticsearch.blobcache.shared.SharedBytes.fin at org.elasticsearch.blobcache.shared.SharedBytes.(Sh at org.elasticsearch.blobcache.shared.SharedBlobCache ] [elk3] Using [jdk] native provider and native methods for [Linux]
orizationProvider] [elk3] Java vector incubator API enabled; uses preferr
] [elk3] version[8.13.0], pid[1748], build[rpm/09df99393193b2c53d9283-22T03:35:46.757803203Z], OS[Linux/5.14.0-362.24.1.el9_3.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/22/22
] [elk3] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK
] [elk3] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.netwotl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosyeInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlolse, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/jaeallocate, --enable-native-access=org.elasticsearch.nativeaccess, -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log, , -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-5907291211499219004, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOunOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,g=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms1814m, -Xmx1814m, -XX:MaxDirectMemorySizeonSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, --module-path=/usr/share/elasticsearch/lib, --add-mles=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
] [elk3] Default Locale [en]
] [elk3] loaded module [repository-url]
] [elk3] loaded module [rest-root]
] [elk3] loaded module [x-pack-core]
] [elk3] loaded module [x-pack-redact]
] [elk3] loaded module [ingest-user-agent]
] [elk3] loaded module [x-pack-async-search]
] [elk3] loaded module [x-pack-monitoring]
] [elk3] loaded module [repository-s3]
] [elk3] loaded module [x-pack-analytics]
] [elk3] loaded module [x-pack-ent-search]
] [elk3] loaded module [x-pack-autoscaling]
] [elk3] loaded module [lang-painless]
] [elk3] loaded module [x-pack-ml]
] [elk3] loaded module [lang-mustache]
] [elk3] loaded module [legacy-geo]
] [elk3] loaded module [x-pack-ql]
] [elk3] loaded module [rank-rrf]
] [elk3] loaded module [analysis-common]
] [elk3] loaded module [health-shards-availability]
] [elk3] loaded module [transport-netty4]
] [elk3] loaded module [aggregations]
] [elk3] loaded module [ingest-common]
] [elk3] loaded module [frozen-indices]
] [elk3] loaded module [x-pack-identity-provider]
] [elk3] loaded module [x-pack-shutdown]
] [elk3] loaded module [x-pack-text-structure]
] [elk3] loaded module [snapshot-repo-test-kit]
] [elk3] loaded module [ml-package-loader]
] [elk3] loaded module [kibana]
] [elk3] loaded module [constant-keyword]
] [elk3] loaded module [x-pack-logstash]
] [elk3] loaded module [x-pack-ccr]
] [elk3] loaded module [x-pack-graph]
] [elk3] loaded module [x-pack-esql]
] [elk3] loaded module [parent-join]
] [elk3] loaded module [counted-keyword]
] [elk3] loaded module [x-pack-enrich]
] [elk3] loaded module [repositories-metering-api]
] [elk3] loaded module [transform]
] [elk3] loaded module [repository-azure]
] [elk3] loaded module [repository-gcs]
] [elk3] loaded module [spatial]
] [elk3] loaded module [apm]
] [elk3] loaded module [mapper-extras]
] [elk3] loaded module [mapper-version]
] [elk3] loaded module [x-pack-rollup]
] [elk3] loaded module [percolator]
] [elk3] loaded module [data-streams]
] [elk3] loaded module [x-pack-stack]
] [elk3] loaded module [rank-eval]
] [elk3] loaded module [reindex]
] [elk3] loaded module [systemd]
] [elk3] loaded module [x-pack-security]
] [elk3] loaded module [blob-cache]
] [elk3] loaded module [searchable-snapshots]
] [elk3] loaded module [x-pack-slm]
] [elk3] loaded module [snapshot-based-recoveries]
] [elk3] loaded module [x-pack-watcher]
] [elk3] loaded module [old-lucene-versions]
] [elk3] loaded module [x-pack-ilm]
] [elk3] loaded module [x-pack-inference]
] [elk3] loaded module [x-pack-voting-only-node]
] [elk3] loaded module [x-pack-deprecation]
] [elk3] loaded module [x-pack-fleet]
] [elk3] loaded module [x-pack-aggregate-metric]
] [elk3] loaded module [x-pack-downsample]
] [elk3] loaded module [x-pack-profiling]
] [elk3] loaded module [ingest-geoip]
] [elk3] loaded module [x-pack-write-load-forecaster]
] [elk3] loaded module [search-business-rules]
] [elk3] loaded module [ingest-attachment]
] [elk3] loaded module [wildcard]
] [elk3] loaded module [x-pack-apm-data]
] [elk3] loaded module [unsigned-long]
] [elk3] loaded module [x-pack-sql]
] [elk3] loaded module [runtime-fields-common]
] [elk3] loaded module [x-pack-async]
] [elk3] loaded module [vector-tile]
] [elk3] loaded module [lang-expression]
] [elk3] loaded module [x-pack-eql]
] [elk3] using [1] data paths, mounts [[/ (/dev/mapper/almalinux-roo.3gb], net total_space [16.9gb], types [xfs]
] [elk3] heap size [1.7gb], compressed ordinary object pointers [tru
] [elk3] node name [elk3], node ID [P3QUmeLjQYa7s13tD1OPPA], cluster, roles [data_frozen]
] [elk3] Registered local node features [data_stream.rollover.lazy, cated, features_supported, health.dsl.info, health.extended_repository_indicator, usage.data_tiers.precalculate_stats]
[elk3] [controller/1767] [Main.cc@123] controller (64 bit):6ce29513f20) Copyright (c) 2024 Elasticsearch BV
] [elk3] Sending apm metrics is disabled
] [elk3] Sending apm tracing is disabled
] [elk3] Security is enabled
[elk3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
ecurityAutoConfiguration] [elk3] Auto-configuration will not generate a pilt-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the bin/ed tool to set the password for the elastic user.
] [elk3] fatal exception while booting Elasticsearch
Not enough free space [15372410880] for cache file of size [16357785600] inrch]
Service.(SharedBlobCacheService.java:374) ~[?:?]
Service.(SharedBlobCacheService.java:333) ~[?:?]
ableSnapshots.createComponents(SearchableSnapshots.java:335) ~[?:
struct$12(NodeConstruction.java:804) ~[elasticsearch-8.13.0.jar:?
atMap$1(PluginsService.java:253) ~[elasticsearch-8.13.0.jar:?]
rencePipeline.java:288) ~[?:?]
rencePipeline.java:212) ~[?:?]
EachRemaining(AbstractList.java:722) ~[?:?]
tPipeline.java:556) ~[?:?]
AbstractPipeline.java:546) ~[?:?]
tPipeline.java:622) ~[?:?]
ode(AbstractPipeline.java:291) ~[?:?]
cePipeline.java:631) ~[?:?]
cePipeline.java:637) ~[?:?]
ePipeline.java:642) ~[?:?]
NodeConstruction.java:804) ~[elasticsearch-8.13.0.jar:?]
struction(NodeConstruction.java:262) ~[elasticsearch-8.13.0.jar:?
~[elasticsearch-8.13.0.jar:?]
icsearch.java:237) ~[elasticsearch-8.13.0.jar:?]
e3(Elasticsearch.java:237) ~[elasticsearch-8.13.0.jar:?]
sticsearch.java:74) ~[elasticsearch-8.13.0.jar:?]
space [15372410880] for cache file of size [16357785600] in path [/var/lib/ela
dCacheSnapshotCacheFilePath(SharedBytes.java:137) ~[?:?]
aredBytes.java:79) ~[?:?]
Service.(SharedBlobCacheService.java:371) ~[?:?]

bouzeghoub · April 2, 2024, 12:00pm

Thank you so much.
The problem is Not enough free space