Marvel duplicate cluster entries

Elastic Stack Elasticsearch
1

All of a sudden, Marvel started to monitor 2 cluster with the same name. How can I delete the data from one cluster? I think I could use the delete by query, but if the clusters' names are the same (say clusterA), how can I remove all data form the duplicate cluster and keep the original data?

And based on the data in the ES Marvel, there is only one cluster uuid.

[EDIT]
More information about the problem:

  • This didn't happen 'all of a sudden', but when the cluster which was being monitored was recreated;
  • The old cluster and new one DO have different cluster_uuid. But there seems to be not data of the old one in the Marvel indices (query "cluster_uuid:old_cluster_uudi" does not return any data) --> No use to delete by query then.

Now, I believe that there is some sort of metadata about the old cluster laying around, maybe on the .marvel-es-data indice. The question remains the same: how can I delete it?

[EDIT2]
When querying all data from the main Marvel index:

GET cluster:9200/.marvel-es-data-1/_search?q= *

hits: [
{ _source.cluster_uuid : "new_uuid",..},
{ _source.cluster_uuid : "new_uuid",..}
{ _source.cluster_uuid : "new_uuid",..}
..
]

There is no information about the "old_uuid" whatsover, so the main (metadata?) Marvel index does not have any pointer to it.

More info:

  • Elasticsearch 2.3
  • (dedicated) Marvel cluster
2

Hi Goncalo,

The .marvel-es-data-1 index is definitely our metadata index, but you can also glean some extra info by restricting it to the cluster_info type.

GET /.marvel-es-data-1/cluster_info/_search?pretty

This will give you only the cluster info documents, which is part of what gets loaded for the cluster listing screen. Beyond that, there's also the time-based index, where we more dynamically load / store the cluster state:

GET /.marvel-es-1-*/cluster_state/_search?pretty
{
  "size": 0,
  "aggs": {
    "group_by_cluster_uuid": {
      "terms": {
        "field": "source_node.cluster_uuid"
      }
    }
  }
}

If it sees more than one Cluster UUID, then it should appear that way. You can then use Delete By Query against the .marvel-es-1-* indices to delete whatever you need / want.

Hope that helps,
Chris