Hello,
So, I put about 500Go of data in elasticsearch, snapshoted it and went on to another project for about two weeks. I now have to go back to my elasticsearch data, but every indices I created, as well as the snapshot, disapeared from the server. I don't know when or how.
Here are what the logs look like:
sudo zcat /mnt/node/log/elasticsearch/elastinoe7-2020-03-17-1.log.gz | more
[2020-03-17T01:15:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] [node-1] triggering scheduled [ML] maintenance tasks
[2020-03-17T01:15:00,007][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [node-1] Deleting expired data
[2020-03-17T01:15:00,012][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [node-1] Completed deletion of expired ML data
[2020-03-17T01:15:00,013][INFO ][o.e.x.m.MlDailyMaintenanceService] [node-1] Successfully completed [ML] maintenance tasks
[2020-03-17T02:30:00,001][INFO ][o.e.x.s.SnapshotRetentionTask] [node-1] starting SLM retention snapshot cleanup task
And the last log that does not look like this is:
sudo zcat /mnt/node/log/elasticsearch/elastinoe7-2020-03-06-1.log.gz | more
[2020-03-06T13:05:07,548][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [16.5gb], net total_space [22.7gb], types [ext4]
[2020-03-06T13:05:07,553][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [7.8gb], compressed ordinary object pointers [true]
[2020-03-06T13:05:07,556][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [011u_HJdSbqhMEXXcIiNmg], cluster name [elastinoe7]
[2020-03-06T13:05:07,556][INFO ][o.e.n.Node ] [node-1] version[7.5.1], pid[969], build[default/deb/3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96/2019-12-16T22:57:37.835892Z], OS[Linux/4.19.0-6-amd64/amd64], JVM[AdoptOpenJDK/Open
JDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2020-03-06T13:05:07,557][INFO ][o.e.n.Node ] [node-1] JVM home [/usr/share/elasticsearch/jdk]
[2020-03-06T13:05:07,557][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8
, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -D
log4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-281080279159202987, -XX:+Hea
pDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:
MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-03-06T13:05:11,938][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2020-03-06T13:05:11,940][INFO ][o.e.p.PluginsService ] [node-1] loaded module [analysis-common]
[2020-03-06T13:05:11,940][INFO ][o.e.p.PluginsService ] [node-1] loaded module [flattened]
[2020-03-06T13:05:11,940][INFO ][o.e.p.PluginsService ] [node-1] loaded module [frozen-indices]
[2020-03-06T13:05:11,941][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
[2020-03-06T13:05:11,941][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2020-03-06T13:05:11,941][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2020-03-06T13:05:11,942][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
[2020-03-06T13:05:11,942][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2020-03-06T13:05:11,942][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
[2020-03-06T13:05:11,943][INFO ][o.e.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2020-03-06T13:05:11,943][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
[2020-03-06T13:05:11,943][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
[2020-03-06T13:05:11,944][INFO ][o.e.p.PluginsService ] [node-1] loaded module [rank-eval]
[2020-03-06T13:05:11,944][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
[2020-03-06T13:05:11,944][INFO ][o.e.p.PluginsService ] [node-1] loaded module [repository-url]
[2020-03-06T13:05:11,945][INFO ][o.e.p.PluginsService ] [node-1] loaded module [search-business-rules]
[2020-03-06T13:05:11,945][INFO ][o.e.p.PluginsService ] [node-1] loaded module [spatial]
[2020-03-06T13:05:11,945][INFO ][o.e.p.PluginsService ] [node-1] loaded module [systemd]
[2020-03-06T13:05:11,946][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transform]
[2020-03-06T13:05:11,946][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2020-03-06T13:05:11,946][INFO ][o.e.p.PluginsService ] [node-1] loaded module [vectors]
[2020-03-06T13:05:11,946][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-analytics]
[2020-03-06T13:05:11,947][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ccr]
[2020-03-06T13:05:11,947][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-core]
[2020-03-06T13:05:11,947][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-deprecation]
[2020-03-06T13:05:11,948][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-enrich]
[2020-03-06T13:05:11,948][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-graph]
[2020-03-06T13:05:11,948][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ilm]
[2020-03-06T13:05:11,949][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-logstash]
[2020-03-06T13:05:11,949][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ml]
[2020-03-06T13:05:11,949][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-monitoring]
[2020-03-06T13:05:11,949][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-rollup]
[2020-03-06T13:05:11,950][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-security]
[2020-03-06T13:05:11,950][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-sql]
[2020-03-06T13:05:11,950][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-voting-only-node]
[2020-03-06T13:05:11,950][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-watcher]
[2020-03-06T13:05:11,951][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2020-03-06T13:05:17,678][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-03-06T13:05:18,489][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/1170] [Main.cc@110] controller (64 bit): Version 7.5.1 (Build ae3c3c51b849be) Copyright (c) 2019 Elasticsearch BV
[2020-03-06T13:05:19,377][DEBUG][o.e.a.ActionModule ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-03-06T13:05:19,608][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-03-06T13:05:20,797][INFO ][o.e.n.Node ] [node-1] initialized
[2020-03-06T13:05:20,799][INFO ][o.e.n.Node ] [node-1] starting ...
[2020-03-06T13:05:21,025][INFO ][o.e.t.TransportService ] [node-1] publish_address {192.168.10.122:9300}, bound_addresses {192.168.10.122:9300}
[2020-03-06T13:05:21,266][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-03-06T13:05:21,284][INFO ][o.e.c.c.Coordinator ] [node-1] setting initial configuration to VotingConfiguration{011u_HJdSbqhMEXXcIiNmg}
[2020-03-06T13:05:21,549][INFO ][o.e.c.s.MasterService ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{011u_HJdSbqhMEXXcIiNmg}{_hHBwbkVQRypnxY3cwfFoA}{192.168.10.122}{192.168.10.122:9300}{dilm}{ml.machine_memory=50628943872,
xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node-1}{011u_HJdSbqhMEXXcIiNmg}{_hHBwbkVQRypnxY3cwfFoA}{192.168.10
.122}{192.168.10.122:9300}{dilm}{ml.machine_memory=50628943872, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-03-06T13:05:21,634][INFO ][o.e.c.c.CoordinationState] [node-1] cluster UUID set to [pzt3rv0cR0qPqxG-XlS9eg]
[2020-03-06T13:05:21,697][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{011u_HJdSbqhMEXXcIiNmg}{_hHBwbkVQRypnxY3cwfFoA}{192.168.10.122}{192.168.10.122:9300}{dilm}{ml.machine_memory=506
28943872, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-03-06T13:05:21,763][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {192.168.10.122:9200}, bound_addresses {192.168.10.122:9200}
[2020-03-06T13:05:21,764][INFO ][o.e.n.Node ] [node-1] started
[2020-03-06T13:05:21,886][INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state
[2020-03-06T13:05:22,094][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2020-03-06T13:05:22,181][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watches] for index patterns [.watches*]
[2020-03-06T13:05:22,295][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watch-history-10] for index patterns [.watcher-history-10*]
[2020-03-06T13:05:22,376][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.slm-history] for index patterns [.slm-history-1*]
[2020-03-06T13:05:22,469][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]
[2020-03-06T13:05:22,576][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2020-03-06T13:05:22,659][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2020-03-06T13:05:22,744][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2020-03-06T13:05:22,818][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2020-03-06T13:05:22,889][INFO ][o.e.a.s.m.TransportMasterNodeAction] [node-1] adding index lifecycle policy [slm-history-ilm-policy]
[2020-03-06T13:05:22,970][INFO ][o.e.a.s.m.TransportMasterNodeAction] [node-1] adding index lifecycle policy [watch-history-ilm-policy]
[2020-03-06T13:05:23,191][INFO ][o.e.l.LicenseService ] [node-1] license [27eb44c0-e750-4111-8fc7-51a78ee900fa] mode [basic] - valid
[2020-03-06T13:05:23,193][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled
PS: Sorry for the format, or if I forgot to say something, this is my first post here. Thank you for understanding.