Unnable to list snapshots in Elasticsearch repository

KA3AXCTAH · April 27, 2021, 5:57pm

I have an Elasticsearch snapshot repository configured to folder on NFS share, and i'm unable to do list snapshots because of strange errors

curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "start object expected"   
      }
    ]
    "type" : "parse_exception",
    "reason" : "start object expected"
  },
  "status" : 400
}

warkolm · April 28, 2021, 1:24am

What version are you running?
What is the output from POST /_snapshot/temp_elastic_backup/_verify?

KA3AXCTAH · April 28, 2021, 7:14am

i'm using 7.4.1 version
and here is output

 
 {
 
   "nodes" : {
 
     "1EUlsWD5QUSNH8cxQFfu0w" : {
 
       "name" : "elastic-04"
 
     },
 
     "YHKzDouIQ5G9z0Wpje-R0g" : {
 
       "name" : "elastic-07"
 
     },

     "-8IDzxOqTWyLj1R4UFBakA" : {

       "name" : "elastic-03"
 
     },
 
     "xTgm8LF0TPSdS41hdJRVZg" : {
 
       "name" : "elastic-13"

     },
 
     "poONWL6pQ3mfIDDavCTNeQ" : {
 
       "name" : "elastic-02"
 
     },

     "cyCDSKetQ7ej_JjUI73Zyg" : {
 
       "name" : "elastic-12"
 
     },
 
     "NzngSLcVR56VdbMkJo7L_g" : {
 
       "name" : "elastic-08"
 
     },
 
    "RDnI0eLXR3WWd8B8GgsWww" : {
 
       "name" : "elastic-14"
 
     },
 
     "8I30XzoFTLiLiGPbag8rxQ" : {
 
       "name" : "elastic-10"
 
     },
 
     "XdxY0H1qSrmd0FUF6CO3Aw" : {
 
       "name" : "elastic-01"
 
     },
 
     "S9k3VyBKQd61jK_KoM5EPQ" : {

      "name" : "elastic-11"
 
     },
 
     "pDQ7XQteR7SweaWqLIoCpg" : {
 
       "name" : "elastic-09"
 
     },
 
     "iwxVxbM5TBiksOAIi3IHUQ" : {
 
       "name" : "elastic-05"

     },

     "PUoA_SbUQm2KYeJTzucxxg" : {
 
       "name" : "elastic-06"
 
     }
 
   }
 
 }

KA3AXCTAH · April 28, 2021, 8:16am

im create another vm and install the latest version of ES (7.12.1) and get this error



curl -X PUT "localhost:9200/_snapshot/old_anthill_efk_nopci?pretty" -H 'Content-Type: application/json' -d'

 {

   "type": "fs",

   "settings": {

     "location": "/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021",

     "compress": true

   }

 }

 '

{

  "error" : {

    "root_cause" : [

      {

        "type" : "parsing_exception",

        "reason" : "Failed to parse object: expecting token of type [START_OBJECT] but found [null]",

        "line" : 1,

        "col" : 0

      }

    ],

    "type" : "repository_exception",

    "reason" : "[old_anthill_efk_nopci] Unexpected exception when loading repository data",

    "caused_by" : {

      "type" : "parsing_exception",

      "reason" : "Failed to parse object: expecting token of type [START_OBJECT] but found [null]",

      "line" : 1,

      "col" : 0

    }

  },

  "status" : 500

}

dadoonet · April 28, 2021, 8:29am

You need to remove the > characters.

KA3AXCTAH · April 28, 2021, 8:46am

excuse me, from where?

dadoonet · April 28, 2021, 9:14am

From here

KA3AXCTAH:

curl -X PUT "localhost:9200/_snapshot/old_anthill_efk_nopci?pretty" -H 'Content-Type: application/json' -d'

> {

>   "type": "fs",

>   "settings": {

>     "location": "/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021",

>     "compress": true

>   }

> }

> '

KA3AXCTAH · April 28, 2021, 9:55am

fixed

dadoonet · April 28, 2021, 10:05am

So did you exactly run that?

curl -XPUT "localhost:9200/_snapshot/old_anthill_efk_nopci?pretty" -H 'Content-Type: application/json' -d '{
   "type": "fs",
   "settings": {
     "location": "/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021",
     "compress": true
   }
}'

Or something else?

KA3AXCTAH · April 28, 2021, 5:19pm

yeah! i run this on the new machine, and it throws error

KA3AXCTAH · April 29, 2021, 9:46am

and i have another snapshot repository ( separate dir on NFS share) that thorws this error when i'm try to list snaps
and error the same as

dadoonet · April 29, 2021, 11:44am

So you are probably doing something which is not exactly that but I can't tell...
May be do a screenshot?

Here is what I have when I run this exact command locally:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[old_anthill_efk_nopci] location [/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021] doesn't match any of the locations specified by path.repo because this setting is empty"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[old_anthill_efk_nopci] failed to create repository",
    "caused_by" : {
      "type" : "repository_exception",
      "reason" : "[old_anthill_efk_nopci] location [/opt/elasticsearch-backup/efk-anthill-nopci-14.03.2021] doesn't match any of the locations specified by path.repo because this setting is empty"
    }
  },
  "status" : 500
}

Which is expected. But at least my request is correct.

KA3AXCTAH · April 30, 2021, 6:04am

This errors happens because you don't specify

path.repo setting in your elasticsearch.yml file

dadoonet · April 30, 2021, 6:39am

I know that.

I'm just showing you that the request is correctly parsed and then it fails.

In your case, the request can not be read correctly.

KA3AXCTAH · April 30, 2021, 6:40am

Sorry for my bad english, this is my scenario
I have 6 node elasticsearch cluster for logs, and i have a cronJob that executes snapshots all of my indicex to NFS share ( Folder on NFS ) so when i'm try to list snapshots this error appears

curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "start object expected"   
      }
    ]
    "type" : "parse_exception",
    "reason" : "start object expected"
  },
  "status" : 400
}

and on this cluster if i
POST /_snapshot/temp_elastic_backup/_verify
it will execute for hours and no reply

but i create another fresh vm and install elasticsearch ( 7.4.1 version on it) and registered this repo on this vm
so register completed without errors, the next step is verify this repo

curl -X POST "localhost:9200/_snapshot/temp_elastic_backup/_verify?pretty"

{

  "nodes" : {

    "WNjRRRJXTuajOXHp542R9w" : {

      "name" : "loges-prod-15-uv03"

    }

  }

}

so as you can see the snapshot verifies without errors, the next step is to list snapshots

curl -X GET "localhost:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"

{

  "error" : {

    "root_cause" : [

      {

        "type" : "parse_exception",

        "reason" : "start object expected"

      }

    ],

    "type" : "parse_exception",

    "reason" : "start object expected"

  },

  "status" : 400

}

same error appeared...

dadoonet · April 30, 2021, 7:00am

The first error is weird.

curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"

Does not have any body to parse. So I don't see what is happening.

Could you share the full elasticsearch logs?

KA3AXCTAH · April 30, 2021, 7:35am

Here is the logZ



cat elasticsearch.log

[2021-04-30T13:30:45,944][INFO ][o.e.e.NodeEnvironment    ] [loges-prod-15-uv03.] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vg1-data)]], net usable_space [1.9tb], net total_space [1.9tb], types [xfs]

[2021-04-30T13:30:45,948][INFO ][o.e.e.NodeEnvironment    ] [loges-prod-15-uv03.] heap size [990.7mb], compressed ordinary object pointers [true]

[2021-04-30T13:30:45,950][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] node name [loges-prod-15-uv03.], node ID [3HZeqohlSSS73Fl0gPY-bA], cluster name [elasticsearch]

[2021-04-30T13:30:45,951][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] version[7.4.1], pid[2866], build[default/rpm/fc0eeb6e2c25915d63d871d344e3d0b45ea0ea1e/2019-10-22T17:16:35.176724Z], OS[Linux/3.10.0-1160.11.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]

[2021-04-30T13:30:45,951][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] JVM home [/usr/share/elasticsearch/jdk]

[2021-04-30T13:30:45,952][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12745454016527767902, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]

[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [aggs-matrix-stats]

[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [analysis-common]

[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [data-frame]

[2021-04-30T13:30:47,612][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [flattened]

[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [frozen-indices]

[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [ingest-common]

[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [ingest-geoip]

[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [ingest-user-agent]

[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [lang-expression]

[2021-04-30T13:30:47,613][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [lang-mustache]

[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [lang-painless]

[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [mapper-extras]

[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [parent-join]

[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [percolator]

[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [rank-eval]

[2021-04-30T13:30:47,614][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [reindex]

[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [repository-url]

[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [search-business-rules]

[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [spatial]

[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [systemd]

[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [transport-netty4]

[2021-04-30T13:30:47,615][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [vectors]

[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-analytics]

[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-ccr]

[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-core]

[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-deprecation]

[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-graph]

[2021-04-30T13:30:47,616][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-ilm]

[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-logstash]

[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-ml]

[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-monitoring]

[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-rollup]

[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-security]

[2021-04-30T13:30:47,617][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-sql]

[2021-04-30T13:30:47,618][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-voting-only-node]

[2021-04-30T13:30:47,618][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] loaded module [x-pack-watcher]

[2021-04-30T13:30:47,618][INFO ][o.e.p.PluginsService     ] [loges-prod-15-uv03.] no plugins loaded

[2021-04-30T13:30:51,036][INFO ][o.e.x.s.a.s.FileRolesStore] [loges-prod-15-uv03.] parsed [0] roles from file [/etc/elasticsearch/roles.yml]

[2021-04-30T13:30:51,506][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [loges-prod-15-uv03.] [controller/2960] [Main.cc@110] controller (64 bit): Version 7.4.1 (Build 973380bdacc5e8) Copyright (c) 2019 Elasticsearch BV

[2021-04-30T13:30:52,061][DEBUG][o.e.a.ActionModule       ] [loges-prod-15-uv03.] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security

[2021-04-30T13:30:52,359][INFO ][o.e.d.DiscoveryModule    ] [loges-prod-15-uv03.] using discovery type [zen] and seed hosts providers [settings]

[2021-04-30T13:30:53,112][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] initialized

[2021-04-30T13:30:53,113][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] starting ...

[2021-04-30T13:30:53,254][INFO ][o.e.t.TransportService   ] [loges-prod-15-uv03.] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}

[2021-04-30T13:30:53,287][WARN ][o.e.b.BootstrapChecks    ] [loges-prod-15-uv03.] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

[2021-04-30T13:30:53,292][INFO ][o.e.c.c.Coordinator      ] [loges-prod-15-uv03.] cluster UUID [JQASlHJoQlO737P9hPV6cg]

[2021-04-30T13:30:53,300][INFO ][o.e.c.c.ClusterBootstrapService] [loges-prod-15-uv03.] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered

[2021-04-30T13:30:53,459][INFO ][o.e.c.s.MasterService    ] [loges-prod-15-uv03.] elected-as-master ([1] nodes joined)[{loges-prod-15-uv03.}{3HZeqohlSSS73Fl0gPY-bA}{CcCuyed4Qqq_zAvf00gNPw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16637538304, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 21, reason: master node changed {previous [], current [{loges-prod-15-uv03.}{3HZeqohlSSS73Fl0gPY-bA}{CcCuyed4Qqq_zAvf00gNPw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16637538304, xpack.installed=true, ml.max_open_jobs=20}]}

[2021-04-30T13:30:53,507][INFO ][o.e.c.s.ClusterApplierService] [loges-prod-15-uv03.] master node changed {previous [], current [{loges-prod-15-uv03.}{3HZeqohlSSS73Fl0gPY-bA}{CcCuyed4Qqq_zAvf00gNPw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16637538304, xpack.installed=true, ml.max_open_jobs=20}]}, term: 3, version: 21, reason: Publication{term=3, version=21}

[2021-04-30T13:30:53,577][INFO ][o.e.h.AbstractHttpServerTransport] [loges-prod-15-uv03.] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}

[2021-04-30T13:30:53,577][INFO ][o.e.n.Node               ] [loges-prod-15-uv03.] started

[2021-04-30T13:30:53,732][INFO ][o.e.l.LicenseService     ] [loges-prod-15-uv03.] license [65da0ec5-43a4-429e-ab7a-d915d2a70d23] mode [basic] - valid

[2021-04-30T13:30:53,733][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [loges-prod-15-uv03.] Active license is now [BASIC]; Security is disabled

[2021-04-30T13:30:53,742][INFO ][o.e.g.GatewayService     ] [loges-prod-15-uv03.] recovered [0] indices into cluster_state

[2021-04-30T13:31:23,555][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]

[2021-04-30T13:31:53,561][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]

[2021-04-30T13:32:23,565][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]

[2021-04-30T13:32:53,568][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [loges-prod-15-uv03.] resolving shards for [indices:monitor/stats] based on cluster state version [23]

dadoonet · April 30, 2021, 7:48am

Nothing wrong here. Is there any reason why I can see blank lines in every single post you are posting?

Could you run again:

curl -X GET "10.0.1.159:9200/_cat/snapshots/temp_elastic_backup?v&s=id&pretty"

And do a screenshot and attach it here?

KA3AXCTAH · April 30, 2021, 7:49am

Thank you so much for you help! sorry blank lines appears bcoz i'm stupid, the next posts will be without blank lines, sure i can run it and post is right now
thanks again

KA3AXCTAH · April 30, 2021, 8:01am

Topic		Replies	Views
Trying to recover snapshot Elasticsearch elastic-stack-searchable-snapshots , snapshot-and-restore	5	1306	February 15, 2023
Snapshot Error Elasticsearch	9	1694	January 10, 2017
Elasticsearch Create a snapshot repository error Elasticsearch	14	7483	November 4, 2022
Array index out of bound exception Elasticsearch snapshot-and-restore	15	2821	October 20, 2021
Unable to create the repository for snapshots after mounting nfs share in the container Elasticsearch docker , snapshot-and-restore	7	452	September 14, 2022

Unnable to list snapshots in Elasticsearch repository

Related Topics