Indices got deleted anonymously

Hi,
some of the indices are deleted. Following is the log we have.
Can you please let us know for the possibilities for same.

[2023-11-07T00:52:00,000][INFO ][o.e.x.m.MlDailyMaintenanceService] [ServerName1] triggering scheduled [ML] maintenance tasks
[2023-11-07T00:52:00,004][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [ServerName1] Deleting expired data
[2023-11-07T00:52:00,006][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [ServerName1] Successfully deleted [0] unused stats documents
[2023-11-07T00:52:00,006][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [ServerName1] Completed deletion of expired ML data
[2023-11-07T00:52:00,006][INFO ][o.e.x.m.MlDailyMaintenanceService] [ServerName1] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2023-11-07T01:30:00,000][INFO ][o.e.x.s.SnapshotRetentionTask] [ServerName1] starting SLM retention snapshot cleanup task
[2023-11-07T01:30:00,000][INFO ][o.e.x.s.SnapshotRetentionTask] [ServerName1] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2023-11-07T02:49:38,057][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [.kibana-observability-ai-assistant-conversations-000001/5XHP_QxAS-64BYoV2LzJ4Q] deleting index
[2023-11-07T02:49:38,750][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [.kibana-observability-ai-assistant-kb-000001/F6B2OO7mQOuWbMMQCUi6nQ] deleting index
[2023-11-07T02:49:39,566][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [testingindex1/6m1nPxl-Sua1XeK3LZDNnw] deleting index
[2023-11-07T02:49:41,326][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [testingindex2/LgaVHLSjTMSwIGFxVDADng] deleting index
[2023-11-07T02:49:42,202][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [testingindex3/AiCid0NGSbS6DGDGof8NfA] deleting index
.....
.....

Few questions:

  • Which elasticsearch version are you using (run GET /)
  • Is you elasticsearch cluster secured?
  • Is your elasticsearch cluster opened on internet?
  • Are you using ILM policies?

version info:

{
  "name": "******",
  "cluster_name": "******",
  "cluster_uuid": "******",
  "version": {
    "number": "8.10.0",
    "build_flavor": "default",
    "build_type": "tar",
    "build_hash": "******",
    "build_date": "2023-09-07T08:16:21.960703010Z",
    "build_snapshot": false,
    "lucene_version": "9.7.0",
    "minimum_wire_compatibility_version": "7.17.0",
    "minimum_index_compatibility_version": "7.0.0"
  },
  "tagline": "******"
}

Yes, our elasticsearch is not opened on internet

ILM Policy:

{
  ".alerts-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T09:04:42.083Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        }
      },
      "_meta": {
        "managed": true
      }
    },
    "in_use_by": {
      "indices": [
        ".internal.alerts-observability.metrics.alerts-default-000001",
        ".internal.alerts-observability.logs.alerts-default-000001",
        ".internal.alerts-observability.uptime.alerts-default-000001",
        ".internal.alerts-observability.slo.alerts-default-000001",
        ".internal.alerts-security.alerts-default-000001",
        ".internal.alerts-observability.apm.alerts-default-000001",
        ".internal.alerts-stack.alerts-default-000001"
      ],
      "data_streams": [],
      "composable_templates": [
        ".alerts-observability.slo.alerts-default-index-template",
        ".alerts-observability.uptime.alerts-default-index-template",
        ".alerts-observability.logs.alerts-default-index-template",
        ".alerts-stack.alerts-default-index-template",
        ".alerts-security.alerts-default-index-template",
        ".alerts-observability.apm.alerts-default-index-template",
        ".alerts-observability.metrics.alerts-default-index-template"
      ]
    }
  },
  ".deprecation-indexing-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.849Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "10gb"
            }
          }
        },
        "delete": {
          "min_age": "30d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "ILM policy used for deprecation logs"
      }
    },
    "in_use_by": {
      "indices": [
        ".ds-.logs-deprecation.elasticsearch-default-2023.10.11-000001"
      ],
      "data_streams": [
        ".logs-deprecation.elasticsearch-default"
      ],
      "composable_templates": [
        ".deprecation-indexing-template"
      ]
    }
  },
  ".fleet-actions-results-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.918Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_size": "300gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for fleet action results indices"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  ".fleet-file-fromhost-data-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:12.018Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "7d",
              "max_size": "10gb"
            }
          }
        },
        "delete": {
          "min_age": "7d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "policy for fleet uploaded files"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".fleet-fileds-fromhost-data"
      ]
    }
  },
  ".fleet-file-fromhost-meta-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.985Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_size": "10gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "policy for fleet uploaded file metadata"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".fleet-fileds-fromhost-meta"
      ]
    }
  },
  ".fleet-file-tohost-data-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.881Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "14d",
              "max_size": "10gb"
            }
          }
        },
        "delete": {
          "min_age": "24d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "policy for fleet deliverable files"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".fleet-fileds-tohost-data"
      ]
    }
  },
  ".fleet-file-tohost-meta-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.949Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_size": "10gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "policy for fleet deliverable file metadata"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".fleet-fileds-tohost-meta"
      ]
    }
  },
  ".kibana-observability-ai-assistant-ilm-policy-conversations": {
    "version": 1,
    "modified_date": "2023-10-11T09:04:41.440Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0s",
          "actions": {
            "rollover": {
              "max_age": "90d",
              "max_primary_shard_size": "50gb"
            }
          }
        }
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  ".monitoring-8-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.097Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "0ms",
          "actions": {
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "3d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "3d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "defaults": {
          "delete_min_age": "Using value of [3d] based on the monitoring plugin default"
        },
        "description": "Index lifecycle policy generated for [monitoring-*-8] data streams"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".monitoring-beats-mb",
        ".monitoring-ent-search-mb",
        ".monitoring-es-mb",
        ".monitoring-kibana-mb",
        ".monitoring-logstash-mb"
      ]
    }
  },
  ".preview.alerts-security.alerts-policy": {
    "version": 1,
    "modified_date": "2023-10-11T09:04:42.355Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "1d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "1d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "180-days-default": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.591Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "2d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {}
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "180d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "built-in ILM policy using the hot, warm, and cold phases with a retention of 180 days"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "30-days-default": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.467Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "2d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "30d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "built-in ILM policy using the hot and warm phases with a retention of 30 days"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "365-days-default": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.741Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "2d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {}
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "365d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "built-in ILM policy using the hot, warm, and cold phases with a retention of 365 days"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "7-days-default": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.506Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "2d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "7d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "7d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "built-in ILM policy using the hot and warm phases with a retention of 7 days"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "90-days-default": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.550Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "2d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {}
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "built-in ILM policy using the hot, warm, and cold phases with a retention of 90 days"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "behavioral_analytics-events-default_policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.200Z",
    "policy": {
      "phases": {
        "warm": {
          "min_age": "2d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {}
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "180d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "Built-in policy applied by default to behavioral analytics event data streams."
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        "behavioral_analytics-events-default"
      ]
    }
  },
  "ilm-history-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.812Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for the ILM history indices"
      }
    },
    "in_use_by": {
      "indices": [
        ".ds-ilm-history-5-2023.10.11-000001"
      ],
      "data_streams": [
        "ilm-history-5"
      ],
      "composable_templates": [
        "ilm-history"
      ]
    }
  },
  "kibana-event-log-policy": {
    "version": 1,
    "modified_date": "2023-10-11T09:04:42.285Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": false,
        "description": "ilm policy the Kibana event log, created initially by Kibana, but updated by the user, not Kibana"
      }
    },
    "in_use_by": {
      "indices": [
        ".ds-.kibana-event-log-8.10.0-2023.10.11-000001"
      ],
      "data_streams": [
        ".kibana-event-log-8.10.0"
      ],
      "composable_templates": [
        ".kibana-event-log-8.10.0-template"
      ]
    }
  },
  "kibana-reporting": {
    "version": 1,
    "modified_date": "2023-10-11T09:04:42.521Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {}
        }
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": []
    }
  },
  "logs": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.427Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for the logs index template installed by x-pack"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        "logs"
      ]
    }
  },
  "metrics": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.381Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for the metrics index template installed by x-pack"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        "metrics"
      ]
    }
  },
  "ml-size-based-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.152Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb"
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for machine learning state and stats indices"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".ml-state",
        ".ml-stats"
      ]
    }
  },
  "slm-history-ilm-policy": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.628Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for the SLM history indices"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".slm-history"
      ]
    }
  },
  "synthetics": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.663Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "50gb"
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for the synthetics index template installed by x-pack"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        "synthetics"
      ]
    }
  },
  "watch-history-ilm-policy-16": {
    "version": 1,
    "modified_date": "2023-10-11T08:40:11.703Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "3d",
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "delete": {
          "min_age": "4d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "managed": true,
        "description": "default policy for the watcher history indices"
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        ".watch-history-16"
      ]
    }
  }
}

Please read this about how to format.
I updated your posts but next time please format your code.

.kibana-observability-ai-assistant-ilm-policy-conversations ILM policy seems to be the policy which removed:

[2023-11-07T02:49:38,057][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [.kibana-observability-ai-assistant-conversations-000001/5XHP_QxAS-64BYoV2LzJ4Q] deleting index
[2023-11-07T02:49:38,750][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [.kibana-observability-ai-assistant-kb-000001/F6B2OO7mQOuWbMMQCUi6nQ] deleting index

For those 3 ones, I don't know:

[2023-11-07T02:49:39,566][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [testingindex1/6m1nPxl-Sua1XeK3LZDNnw] deleting index
[2023-11-07T02:49:41,326][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [testingindex2/LgaVHLSjTMSwIGFxVDADng] deleting index
[2023-11-07T02:49:42,202][INFO ][o.e.c.m.MetadataDeleteIndexService] [ServerName1] [testingindex3/AiCid0NGSbS6DGDGof8NfA] deleting index

May be it's "a bug" and you should anyway upgrade to 8.10.4 or 8.11.0.
Or may be someone connected to your cluster did remove those testingindex* indices?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.