Elasticsearch 8.5.2 SLM Exclude index and datastream does not work

Elastic Stack Elasticsearch
,
1

Dear community members, using SLM policy I would like to exclue some indices and datastream from y daily backup; using the policy below doesn't gives me any ERROR ; but it's NOT applied as expected; Because doing a 'GET policy' does not show me the exlusion parameters.

What did I do wrong here to exclude some data I don't need in my daily backup ?

PUT _slm/policy/daily_all.json
{
  "name": "<daily-snap-prod-{now/d}>",
  "schedule": "0 29 2 * * ?",
  "repository": "elk-prd-data",
  "config": {
    "include_global_state": true,
    "feature_states": [
      "none"
    ],
    "ignore_unavailable": true,
    "partial": true
  },
  "retention": {
    "expire_after": "365d",
    "min_count": 100,
    "max_count": 1000
  },
  "is_managed": true,
  "exclude": {
    "indices": [
      ".readonlyrest_kbn_sessions",
      ".reporting.kibana-*",
      ".monitoring-beats*",
      ".ds-.logs-deprecation.elasticsearch-default*",
      ".ds-.monitoring-beats*"
    ],
    "data_streams": [
      ".monitoring-beats-8-mb",
      ".monitoring-es-8-mb",
      ".monitoring-kibana-8-mb",
      ".monitoring-logstash-8-mb",
      "filebeat-8.5.2",
      "heartbeat-8.12.2",
      "heartbeat-8.13.0",
      "heartbeat-8.13.1",
      "heartbeat-8.5.2",
      "logs-admin-lgst",
      "logs-generic-default",
      "metricbeat-8.5.2",
      "packetbeat-8.5.2"
    ]
  }
}

{
  "acknowledged": true
}

BUT using GET _slm/policy/daily_all.json, doesn't show me the Indices and Datastream I would like to exclude for my daily backup:

{
  "daily_all.json": {
    "version": 6,
    "modified_date_millis": 1712643959851,
    "policy": {
      "name": "<daily-snap-prod-{now/d}>",
      "schedule": "0 29 2 * * ?",
      "repository": "elk-prd-data",
      "config": {
        "include_global_state": true,
        "feature_states": [
          "none"
        ],
        "ignore_unavailable": true,
        "partial": true
      },
      "retention": {
        "expire_after": "365d",
        "min_count": 100,
        "max_count": 1000
      }
    },
    "last_success": {
      "snapshot_name": "daily-snap-prod-2024.04.09-0lfwvyybq0qot9rzlshnfw",
      "start_time": 1712629739887,
      "time": 1712633704240
    },
    "last_failure": {
      "snapshot_name": "daily-snap-prod-2024.03.28-bs_j7vwmttapxjxcfwvmxa",
      "time": 1711634206016,
      "details": """{"type":"snapshot_exception","reason":"[elk-prd-data:daily-snap-prod-2024.03.28-bs_j7vwmttapxjxcfwvmxa] failed to create snapshot successfully, 17 out of 811 total shards failed"}"""
    },
    "next_execution_millis": 1712716140000,
    "stats": {
      "policy": "daily_all.json",
      "snapshots_taken": 15,
      "snapshots_failed": 1,
      "snapshots_deleted": 10,
      "snapshot_deletion_failures": 0
    }
  }
}

Thanks in advance for your time and help;

kr,

2

I see that we can include exclude indices in the graphical UI of kibana BUT I don't want to use this as I have a huge amount of DataStream and indices as well.

image
image1159×541 43.1 KB

3

re-writing the SLM this way (see below) let me see the exclusion when I do a GET _slm/policy/ BUT the snapshot still contains the index and datastream I would like to exclude :confused:

PUT _slm/policy/daily_all.json
{
  "name": "<daily-snap-prod-{now/d}>",
  "schedule": "0 29 2 * * ?",
  "repository": "elk-prd-data",
  "config": {
    "exclude": {
      "indices": [
        ".readonlyrest_kbn_sessions",
        ".reporting.kibana-*",
        ".monitoring-beats*",
        ".ds-.logs-deprecation.elasticsearch-default*",
        ".ds-.monitoring-beats*"
      ],
      "data_streams": [
        ".monitoring-beats-8-mb",
        ".monitoring-es-8-mb",
        ".monitoring-kibana-8-mb",
        ".monitoring-logstash-8-mb",
        "filebeat-*",
        "heartbeat-*",
        "logs-admin-lgst",
        "logs-generic-default",
        "metricbeat-*",
        "packetbeat-*"
      ]
    },
    "include_global_state": true,
    "feature_states": [
      "none"
    ],
    "ignore_unavailable": true,
    "partial": true
  },
  "retention": {
    "expire_after": "365d",
    "min_count": 100,
    "max_count": 1000
  },
  "is_managed": true
}
4

I manage to get the indices and datastream I wanted using this SLM policy definition:

Basically it gets all the .kibana, .security and the logs datastream ;

:bulb: But I think having a proper exclude statement is better for maintainability over time and make it easier to manage the policy as our environment evolves.

PUT _slm/policy/daily_all.json
{
  "name": "<daily-snap-prod-{now/d}>",
  "schedule": "0 29 2 * * ?",
  "repository": "elk-prd-data",
  "config": {
    "indices": [
      ".security*",
      ".kibana*",
      "logs-*"
    ],
    "include_global_state": true,
    "feature_states": [
      "none"
    ],
    "ignore_unavailable": true,
    "partial": true
  },
  "retention": {
    "expire_after": "365d",
    "min_count": 100,
    "max_count": 1000
  },
  "is_managed": true
}