Backup and restore elasticsearch

Elastic Stack Elasticsearch
1

Hello,

in-order to take backup while creating path.repo in /etc/elaticsearch/elasticsearch.yml, the elasticsearch is failing to start its beeing in stoped state dont know what is the issue

below config file

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.repo: /root/backup_repo

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: **********
network.host: 0.0.0.0

Set a custom port for HTTP:

#http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.seed_hosts: ["host1", "host2"]

Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: ["node-1", "node-2"]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true
discovery.type: single-node

any help,

Thanks,
Madhu

2

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

3 
in-order to take backup while creating path.repo in /etc/elaticsearch/elasticsearch.yml, the elasticsearch is failing to start its beeing in stoped state dont know what is the issue

below config file    


     # ======================== Elasticsearch Configuration =========================
        #
        # NOTE: Elasticsearch comes with reasonable defaults for most settings.
        #       Before you set out to tweak and tune the configuration, make sure you
        #       understand what are you trying to accomplish and the consequences.
        #
        # The primary way of configuring a node is via this file. This template lists
        # the most important settings you may want to configure for a production cluster.
        #
        # Please consult the documentation for further information on configuration options:
        # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
        #
        # ---------------------------------- Cluster -----------------------------------
        #
        # Use a descriptive name for your cluster:
        #
        #cluster.name: my-application
        #
        # ------------------------------------ Node ------------------------------------
        #
        # Use a descriptive name for the node:
        #
        #node.name: node-1
        #
        # Add custom attributes to the node:
        #
        #node.attr.rack: r1
        #
        # ----------------------------------- Paths ------------------------------------
        #
        # Path to directory where to store the data (separate multiple locations by comma):
        #
        path.data: /var/lib/elasticsearch
        #
        # Path to log files:
        path.repo: /root/backup_repo
        #
        path.logs: /var/log/elasticsearch
        #
        # ----------------------------------- Memory -----------------------------------
        #
        # Lock the memory on startup:
        #
        #bootstrap.memory_lock: true
        #
        # Make sure that the heap size is set to about half the memory available
        # on the system and that the owner of the process is allowed to use this
        # limit.
        #
        # Elasticsearch performs poorly when the system is swapping the memory.
        #
        # ---------------------------------- Network -----------------------------------
        #
        # Set the bind address to a specific IP (IPv4 or IPv6):
        #
        #network.host: **********
        network.host: 0.0.0.0
        #
        # Set a custom port for HTTP:
        #
        #http.port: 9200
        #
        # For more information, consult the network module documentation.
        #
        # --------------------------------- Discovery ----------------------------------
        #
        # Pass an initial list of hosts to perform discovery when this node is started:
        # The default list of hosts is ["127.0.0.1", "[::1]"]
        #
        #discovery.seed_hosts: ["host1", "host2"]
        #
        # Bootstrap the cluster using an initial set of master-eligible nodes:
        #
        #cluster.initial_master_nodes: ["node-1", "node-2"]
        #
        # For more information, consult the discovery and cluster formation module documentation.
        #
        # ---------------------------------- Gateway -----------------------------------
        #
        # Block initial recovery after a full cluster restart until N nodes are started:
        #
        #gateway.recover_after_nodes: 3
        #
        # For more information, consult the gateway module documentation.
        #
        # ---------------------------------- Various -----------------------------------
        #
        # Require explicit names when deleting indices:
        #
        #action.destructive_requires_name: true
        discovery.type: single-node

any help
4

The indentation is wrong. May be that's the issue.

5

i think its right only,

can you please send me the line

6

these are the logs after adding path.repo and restarted

[2020-04-06T08:43:11,509][ERROR][o.e.b.Bootstrap          ] [elastic7] Exception
java.lang.IllegalStateException: Unable to access 'path.repo' (/root/my_backup)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:314) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:206) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.1.jar:7.1.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.nio.file.AccessDeniedException: /root/my_backup
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:401) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-7.1.1.jar:7.1.1]
        ... 12 more
[2020-04-06T08:43:11,524][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elastic7] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/root/my_backup)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' (/root/my_backup)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:314) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:206) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /root/my_backup
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:401) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:314) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:253) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:122) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:206) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
        ... 6 more
[2020-04-06T08:43:37,806][INFO ][o.e.e.NodeEnvironment    ] [elastic7] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [144.6gb], net total_space [154.8gb], types [ext4]
[2020-04-06T08:43:37,811][INFO ][o.e.e.NodeEnvironment    ] [elastic7] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-04-06T08:43:37,879][INFO ][o.e.n.Node               ] [elastic7] node name [elastic7], node ID [3eScVBuNT5K0fveQzu-YEA], cluster name [elasticsearch]
[2020-04-06T08:43:37,880][INFO ][o.e.n.Node               ] [elastic7] version[7.1.1], pid[8847], build[default/deb/7a013de/2019-05-23T14:04:00.380842Z], OS[Linux/4.15.0-72-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2020-04-06T08:43:37,881][INFO ][o.e.n.Node               ] [elastic7] JVM home [/usr/share/elasticsearch/jdk]
[2020-04-06T08:43:37,881][INFO ][o.e.n.Node               ] [elastic7] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-17675785612206672785, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-04-06T08:43:39,775][INFO ][o.e.p.PluginsService     ] [elastic7] loaded module [aggs-matrix-stats]
7 
AccessDeniedException: /root/my_backup

It means that not all the nodes can access this dir from the elasticsearch process.

8

i need to restore all my indexes from dev to prod env what is the best way to do it any document to follow

Thanks

9

You can use snapshot and restore, you can use reindex from remote if your prod cluster has access to your dev cluster, you can "just" reindex the same way you did when you initially indexed in the dev cluster.

What is exactly the problem you have?

10

While i trying to index from dev to prod i got below error

 curl -X POST "***.**.***.**:9200/_snapshot/my_backup/snapshot_1/_restore?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_missing_exception",
        "reason" : "[my_backup] missing"
      }
    ],
    "type" : "repository_missing_exception",
    "reason" : "[my_backup] missing"
  },
  "status" : 404
}
11

i took backup on dev box and try to load it from dev to prod by executing below command from dev box

curl -X POST "Prod_URL:9200/_snapshot/my_backup/snapshot_1/_restore?pretty"

but i got below error, even through i added my_backup folder and added (path.repo: $path) on elastiseach.yml file on prod

what could be the issue

12

You need to create the repository in the destination cluster.

13

can we have quick zoom call please

14

I can't I'm afraid.

15

my question is cant we restore indexes from dev box to prod frm dev box only
??

16

because now its showing there is no snapshot after creating repo

{
  "error" : {
    "root_cause" : [
      {
        "type" : "snapshot_restore_exception",
        "reason" : "[my_backup:snapshot_1] snapshot does not exist"
      }
    ],
    "type" : "snapshot_restore_exception",
    "reason" : "[my_backup:snapshot_1] snapshot does not exist"
  },
  "status" : 500
}
17

so do i need to copy the snapshot manually from dev to prod and need to execute below command or i can execute from dev box itself ??

curl -X POST "Prod_URL:9200/_snapshot/my_backup/snapshot_1/_restore?pretty"

18

You need to have a repository on the dev machine which is able to save the data to a shared FS.
You need to create a repository on the prod machine which is able to access the same data on a shared FS.

So yes, once everything is backup on the dev shared FS, you can copy the files to the production system in whatever dir which needs to be shared by all nodes, create the repository and restore.

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.