I have an Elasticsearch snapshot repository configured to folder on NFS share, and i'm unable to do list snapshots because of strange errors
curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"
{
"error" : {
"root_cause" : [
{
"type" : "parse_exception",
"reason" : "start object expected"
}
]
"type" : "parse_exception",
"reason" : "start object expected"
},
"status" : 400
}
What version are you running?
What is the output from POST /_snapshot/temp_elastic_backup/_verify?
i'm using 7.4.1 version
and here is output
{
"nodes" : {
"1EUlsWD5QUSNH8cxQFfu0w" : {
"name" : "elastic-04"
},
"YHKzDouIQ5G9z0Wpje-R0g" : {
"name" : "elastic-07"
},
"-8IDzxOqTWyLj1R4UFBakA" : {
"name" : "elastic-03"
},
"xTgm8LF0TPSdS41hdJRVZg" : {
"name" : "elastic-13"
},
"poONWL6pQ3mfIDDavCTNeQ" : {
"name" : "elastic-02"
},
"cyCDSKetQ7ej_JjUI73Zyg" : {
"name" : "elastic-12"
},
"NzngSLcVR56VdbMkJo7L_g" : {
"name" : "elastic-08"
},
"RDnI0eLXR3WWd8B8GgsWww" : {
"name" : "elastic-14"
},
"8I30XzoFTLiLiGPbag8rxQ" : {
"name" : "elastic-10"
},
"XdxY0H1qSrmd0FUF6CO3Aw" : {
"name" : "elastic-01"
},
"S9k3VyBKQd61jK_KoM5EPQ" : {
"name" : "elastic-11"
},
"pDQ7XQteR7SweaWqLIoCpg" : {
"name" : "elastic-09"
},
"iwxVxbM5TBiksOAIi3IHUQ" : {
"name" : "elastic-05"
},
"PUoA_SbUQm2KYeJTzucxxg" : {
"name" : "elastic-06"
}
}
}
im create another vm and install the latest version of ES (7.12.1) and get this error
curl -X PUT "localhost:9200/_snapshot/old_anthill_efk_nopci?pretty" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021",
"compress": true
}
}
'
{
"error" : {
"root_cause" : [
{
"type" : "parsing_exception",
"reason" : "Failed to parse object: expecting token of type [START_OBJECT] but found [null]",
"line" : 1,
"col" : 0
}
],
"type" : "repository_exception",
"reason" : "[old_anthill_efk_nopci] Unexpected exception when loading repository data",
"caused_by" : {
"type" : "parsing_exception",
"reason" : "Failed to parse object: expecting token of type [START_OBJECT] but found [null]",
"line" : 1,
"col" : 0
}
},
"status" : 500
}
You need to remove the > characters.
excuse me, from where?
From here
fixed
So did you exactly run that?
curl -XPUT "localhost:9200/_snapshot/old_anthill_efk_nopci?pretty" -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location": "/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021",
"compress": true
}
}'
Or something else?
yeah! i run this on the new machine, and it throws error
and i have another snapshot repository ( separate dir on NFS share) that thorws this error when i'm try to list snaps
and error the same as
So you are probably doing something which is not exactly that but I can't tell...
May be do a screenshot?
Here is what I have when I run this exact command locally:
{
"error" : {
"root_cause" : [
{
"type" : "repository_exception",
"reason" : "[old_anthill_efk_nopci] location [/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021] doesn't match any of the locations specified by path.repo because this setting is empty"
}
],
"type" : "repository_exception",
"reason" : "[old_anthill_efk_nopci] failed to create repository",
"caused_by" : {
"type" : "repository_exception",
"reason" : "[old_anthill_efk_nopci] location [/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021] doesn't match any of the locations specified by path.repo because this setting is empty"
}
},
"status" : 500
}
Which is expected. But at least my request is correct.
This errors happens because you don't specify
path.repo setting in your elasticsearch.yml file
I know that.
I'm just showing you that the request is correctly parsed and then it fails.
In your case, the request can not be read correctly.
Sorry for my bad english, this is my scenario
I have 6 node elasticsearch cluster for logs, and i have a cronJob that executes snapshots all of my indicex to NFS share ( Folder on NFS ) so when i'm try to list snapshots this error appears
curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"
{
"error" : {
"root_cause" : [
{
"type" : "parse_exception",
"reason" : "start object expected"
}
]
"type" : "parse_exception",
"reason" : "start object expected"
},
"status" : 400
}
and on this cluster if i
POST /_snapshot/temp_elastic_backup/_verify
it will execute for hours and no reply
but i create another fresh vm and install elasticsearch ( 7.4.1 version on it) and registered this repo on this vm
so register completed without errors, the next step is verify this repo
curl -X POST "localhost:9200/_snapshot/temp_elastic_backup/_verify?pretty"
{
"nodes" : {
"WNjRRRJXTuajOXHp542R9w" : {
"name" : "loges-prod-15-uv03"
}
}
}
so as you can see the snapshot verifies without errors, the next step is to list snapshots
curl -X GET "localhost:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"
{
"error" : {
"root_cause" : [
{
"type" : "parse_exception",
"reason" : "start object expected"
}
],
"type" : "parse_exception",
"reason" : "start object expected"
},
"status" : 400
}
same error appeared...
The first error is weird.
curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"
Does not have any body to parse. So I don't see what is happening.
Could you share the full elasticsearch logs?
Here is the logZ
cat elasticsearch.log
[2021-04-30T13:30:45,944][INFO ][o.e.e.NodeEnvironment ] [loges-prod-15-uv03.] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vg1-data)]], net usable_space [1.9tb], net total_space [1.9tb], types [xfs]
[2021-04-30T13:30:45,948][INFO ][o.e.e.NodeEnvironment ] [loges-prod-15-uv03.] heap size [990.7mb], compressed ordinary object pointers [true]
[2021-04-30T13:30:45,950][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] node name [loges-prod-15-uv03.], node ID [3HZeqohlSSS73Fl0gPY-bA], cluster name [elasticsearch]
[2021-04-30T13:30:45,951][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] version[7.4.1], pid[2866], build[default/rpm/fc0eeb6e2c25915d63d871d344e3d0b45ea0ea1e/2019-10-22T17:16:35.176724Z], OS[Linux/3.10.0-1160.11.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]
[2021-04-30T13:30:45,951][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] JVM home [/usr/share/elasticsearch/jdk]
[2021-04-30T13:30:45,952][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12745454016527767902, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [aggs-matrix-stats]
[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [analysis-common]
[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [data-frame]
[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [flattened]
[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [frozen-indices]
[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [ingest-common]
[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [ingest-geoip]
[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [ingest-user-agent]
[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [lang-expression]
[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [lang-mustache]
[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [lang-painless]
[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [mapper-extras]
[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [parent-join]
[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [percolator]
[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [rank-eval]
[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [reindex]
[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [repository-url]
[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [search-business-rules]
[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [spatial]
[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [systemd]
[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [transport-netty4]
[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [vectors]
[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-analytics]
[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-ccr]
[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-core]
[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-deprecation]
[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-graph]
[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-ilm]
[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-logstash]
[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-ml]
[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-monitoring]
[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-rollup]
[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-security]
[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-sql]
[2021-04-30T13:30:47,618][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-voting-only-node]
[2021-04-30T13:30:47,618][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] loaded module [x-pack-watcher]
[2021-04-30T13:30:47,618][INFO ][o.e.p.PluginsService ] [loges-prod-15-uv03.] no plugins loaded
[2021-04-30T13:30:51,036][INFO ][o.e.x.s.a.s.FileRolesStore] [loges-prod-15-uv03.] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-04-30T13:30:51,506][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [loges-prod-15-uv03.] [controller/2960] [Main.cc@110] controller (64 bit): Version 7.4.1 (Build 973380bdacc5e8) Copyright (c) 2019 Elasticsearch BV
[2021-04-30T13:30:52,061][DEBUG][o.e.a.ActionModule ] [loges-prod-15-uv03.] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2021-04-30T13:30:52,359][INFO ][o.e.d.DiscoveryModule ] [loges-prod-15-uv03.] using discovery type [zen] and seed hosts providers [settings]
[2021-04-30T13:30:53,112][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] initialized
[2021-04-30T13:30:53,113][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] starting ...
[2021-04-30T13:30:53,254][INFO ][o.e.t.TransportService ] [loges-prod-15-uv03.] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-04-30T13:30:53,287][WARN ][o.e.b.BootstrapChecks ] [loges-prod-15-uv03.] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-04-30T13:30:53,292][INFO ][o.e.c.c.Coordinator ] [loges-prod-15-uv03.] cluster UUID [JQASlHJoQlO737P9hPV6cg]
[2021-04-30T13:30:53,300][INFO ][o.e.c.c.ClusterBootstrapService] [loges-prod-15-uv03.] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-04-30T13:30:53,459][INFO ][o.e.c.s.MasterService ] [loges-prod-15-uv03.] elected-as-master ([1] nodes joined)[{loges-prod-15-uv03.}{3HZeqohlSSS73Fl0gPY-bA}{CcCuyed4Qqq_zAvf00gNPw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16637538304, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 21, reason: master node changed {previous [], current [{loges-prod-15-uv03.}{3HZeqohlSSS73Fl0gPY-bA}{CcCuyed4Qqq_zAvf00gNPw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16637538304, xpack.installed=true, ml.max_open_jobs=20}]}
[2021-04-30T13:30:53,507][INFO ][o.e.c.s.ClusterApplierService] [loges-prod-15-uv03.] master node changed {previous [], current [{loges-prod-15-uv03.}{3HZeqohlSSS73Fl0gPY-bA}{CcCuyed4Qqq_zAvf00gNPw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16637538304, xpack.installed=true, ml.max_open_jobs=20}]}, term: 3, version: 21, reason: Publication{term=3, version=21}
[2021-04-30T13:30:53,577][INFO ][o.e.h.AbstractHttpServerTransport] [loges-prod-15-uv03.] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-04-30T13:30:53,577][INFO ][o.e.n.Node ] [loges-prod-15-uv03.] started
[2021-04-30T13:30:53,732][INFO ][o.e.l.LicenseService ] [loges-prod-15-uv03.] license [65da0ec5-43a4-429e-ab7a-d915d2a70d23] mode [basic] - valid
[2021-04-30T13:30:53,733][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [loges-prod-15-uv03.] Active license is now [BASIC]; Security is disabled
[2021-04-30T13:30:53,742][INFO ][o.e.g.GatewayService ] [loges-prod-15-uv03.] recovered [0] indices into cluster_state
[2021-04-30T13:31:23,555][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]
[2021-04-30T13:31:53,561][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]
[2021-04-30T13:32:23,565][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]
[2021-04-30T13:32:53,568][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]
Nothing wrong here. Is there any reason why I can see blank lines in every single post you are posting?
Could you run again:
curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"
And do a screenshot and attach it here?
Thank you so much for you help! sorry blank lines appears bcoz i'm stupid, the next posts will be without blank lines, sure i can run it and post is right now
thanks again
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
