Cross cluster replication Data streams

Elastic Stack Elasticsearch
,
1

Hi, here is the environment i am working with.

Local cluster : running elastic stack. My local cluster is connected to the remote cluster.

Remote cluster: vm running elk stack in aws cloud. This a single node cluster with Fleet enabled and has agents connected sending metrics, windows logs and siem logs.

Goal: I would like to replicate the fleet data streams to my local cluster to then run ml jobs (and use the siem functions) on the replicated data.

Issue: I have created auto-follower patterns on the local cluster but nothing is being replicated on the local side. what does the auto-follow pattern need to match when trying to replicate data streams? How can i check if it is working? Does it create an index once it starts to follow on the local cluster that i can search?

Is there documentation highlighting this in detail?

There is mention of this here but it is very brief and there is no instruction or example specific to data streams. Manage auto-follow patterns | Elasticsearch Guide [7.15] | Elastic

Here is the auto-follow pattern i am using that is not working.

PUT /_ccr/auto_follow/test
{
  "remote_cluster": "elk-remote",
  "leader_index_patterns": [
    "logs-elastic_agent.endpoint_security-default"
  ],
  "follow_index_pattern": "REMOTE-{{leader_index}}"
}

some extra info:

GET /_ccr/stats
{
  "auto_follow_stats" : {
    "number_of_failed_follow_indices" : 0,
    "number_of_failed_remote_cluster_state_requests" : 0,
    "number_of_successful_follow_indices" : 0,
    "recent_auto_follow_errors" : [ ],
    "auto_followed_clusters" : [
      {
        "cluster_name" : "elk-remote",
        "time_since_last_check_millis" : 6003,
        "last_seen_metadata_version" : 1030
      }
    ]
  },
  "follow_stats" : {
    "indices" : [ ]
  }
}

Thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.