Elastic Cloud Snapshot to AWS S3

Hi Elastic Members,

I Have a question of Elastic Cloud index snapshot, As I know if we want to send index snapshots to AWS s3 in terms of on-premise ELK solution,
We need to install repository-s3 plugin on each of ES nodes besides we need to create AWS s3 Bucket.

My question is that if the elastic cloud is natively supporting send index snapshots to AWS S3 without install repository-s3 plugin.

Could the Elastic Cloud team clarify the question I made?
It's better if you can provide the details of how to implement it.

Best Regards

Clusters running on Elastic Cloud do indeed use the repository-s3 plugin for sending snapshots to S3.

Hi @DavidTurner,

I mean that the repository-s3 plugin is installed on elastic cloud by default or we need to install it by manual on somewhere.

What's the step to install the repository-s3 plugin if it's not installed by default on elastic cloud?

Thanks in advance

If I remember correctly, there's a UI for configuring snapshots in Elastic Cloud clusters. I don't think you need to do anything else.

Hi @DavidTurner,

I didn't see there is any options can enable AWS S3 plugins on Console.

By the way, we need to take snapshots for indices by ourself. not the default snapshots on the elastic cloud you made as follow

The basically our Idea (retention) is that for last 1 to 3 months data we keep on hot nodes, for last 4 to 6 months data we keep on warm nodes, for data 6 months later, we will send to our AWS S3.

I just run an Elasticsearch GET snapshot API, I think this is your snapshot settings for AWS S3

GET /_snapshot

  "found-snapshots" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "xxxxxdfdfdfdfdfdfdfdfdf",
      "server_side_encryption" : "true",
      "compress" : "true",
      "aws_account" : "operations-5-eu-west-1",
      "buffer_size" : "20mb",
      "client" : "elastic-internal-dae10a",
      "base_path" : "snapshots/dfdfefefefsfefefefefef"

Could you tell me how I can send the snapshots to our AWS S3 Storage instead of yours?

Thanks in advance

Ah, I see. It looks like you can indeed snapshot to another repo.

After we setup AWS S3 Bucket and enable Elastic Cloud S3 Repository by follow your instruction . I did a snapshot for index metricbeat-2019 and send it to AWS S3. then disable logstash to send metrics to index metricbeat-2019. remove the current index metricbeat-2019 on elastic cloud. and did index restoration from our AWS S3. but the elastic cluster became unhealthy.

Some details may be could help for elastic cloud team

the snapshot we made on AWS S3:

GET /_snapshot/es-cloud-infra-s3/snapshot_1

  "snapshots" : [
      "snapshot" : "snapshot_1",
      "uuid" : "QjSwSoDXRvqhB02cwGPC0w",
      "version_id" : 6070099,
      "version" : "6.7.0",
      "indices" : [
      "include_global_state" : false,
      "state" : "SUCCESS",
      "start_time" : "2019-04-26T10:44:12.485Z",
      "start_time_in_millis" : 1556275452485,
      "end_time" : "2019-04-26T10:44:20.585Z",
      "end_time_in_millis" : 1556275460585,
      "duration_in_millis" : 8100,
      "failures" : [ ],
      "shards" : {
        "total" : 5,
        "failed" : 0,
        "successful" : 5

what I have checked after the cluster became unhealthy.

GET _cluster/health?level=shardes
  "cluster_name" : "dae10a741b4d4bb98065af29468fbb93",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 21,
  "active_shards" : 21,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 80.76923076923077

GET _cluster/health?level=indices
    "metricbeat-2019" : {
      "status" : "red",
      "number_of_shards" : 5,
      "number_of_replicas" : 0,
      "active_primary_shards" : 0,
      "active_shards" : 0,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5

Finally, I did a cluster restart want to bring it back, but with no luck.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.