I have two elasticsearch cluster's that named as es-dev and another one is elastisearch,
In the es-dev cluster side we have 3master and 3data nodes and one kibana node. And we have another one cluster that was named as elasticsearch.
I want to collect the metrics from es-dev cluster and send them to standalone elasticsearch cluster. In the side the metrics was successfully collected log from the es-dev cluster 3master and 3data nodes. However, the metrics are not being sent to the standalone Elasticsearch cluster. Kindly do the needful and give me a solution of this issue. How to export metrics from es-dev cluster to standalone Elasticsearch cluster.
Note: Some Metricbeat logs are i mentioned here for your reference.
Jan 16 09:26:01 metricbeat[18438]: {"log.level":"error","@timestamp":"2025-01-16T09:26:01.460+0100","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch()): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 16 09:26:01 metricbeat[18438]: {"log.level":"warn","@timestamp":"2025-01-16T09:26:01.795+0100","log.logger":"elasticsearch.ccr","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/elasticsearch/ccr.(*MetricSet).Fetch","file.name":"ccr/ccr.go","file.line":78},"message":"the CCR feature is available with a platinum or enterprise Elasticsearch license. You currently have a basic license. Either upgrade your license or remove the ccr metricset from your Elasticsearch module configuration.","service.name":"metricbeat","ecs.version":"1.6.0"}
' ' '
Scraped Metrics from es-dev cluster (This is a Metricbeat logs)
Jan 16 09:55:51 metricbeat[18673]: {"log.level":"info","@timestamp":"2025-01-16T09:55:51.902+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":70529024}}}},"cpu":{"system":{"ticks":630,"time":{"ms":220}},"total":{"ticks":2160,"time":{"ms":640},"value":2160},"user":{"ticks":1530,"time":{"ms":420}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":27},"info":{"ephemeral_id":"9e4d9f44-7653-4672-bf16-b2e133fef6fb","uptime":{"ms":90064},"version":"8.14.3"},"memstats":{"gc_next":57482040,"memory_alloc":33828888,"memory_total":402882328,"rss":145113088},"runtime":{"goroutines":170}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":809,"active":0,"batches":3,"duplicates":772,"total":1581},"read":{"bytes":290650,"errors":3},"write":{"bytes":371002,"latency":{"histogram":{"count":8,"max":77,"mean":61.125,"median":59,"min":54,"p75":64.25,"p95":77,"p99":77,"p999":77,"stddev":6.881814804250402}}}},"pipeline":{"clients":28,"events":{"active":362,"filtered":3,"published":1469,"total":1472},"queue":{"acked":1581}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":6,"success":6},"enrich":{"events":6,"success":6},"index":{"events":444,"success":444},"index_recovery":{"events":252,"success":252},"index_summary":{"events":6,"success":6},"node_stats":{"events":6,"success":6},"shard":{"events":696,"success":696}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":7,"success":7},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":12,"success":12},"process":{"events":21,"success":21},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
' ' '