Metricbeat writing to .ds-metricbeat instead of .ds-.monitoring

Elastic Stack Beats
21

That is a problem

Yes, this error was resolved by elevating privileges of elastic.output user and specifying the kibana host.

Now the output of the setup command is not showing any errors. I ran it with the -e flag after specifying setup.ilm.overwrite: true in the metricbeat.yml file.

22

output of /usr/bin/metricbeat setup:

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

23

So, is it still writing to the wrong index?

24

Yes it is.

25

There is actually monitoring data showing up in Stack Monitoring of the new cluster. But it is not showing up in the Stack Monitoring of the monitoring cluster..

26

Delete the indices / data streams
Run Setup again
Try Again

27

which indices / data streams should I delete?

28

Your output section need to point to the monitoring cluster

output.elasticsearch:
  hosts: ["https://xxx:9200"]
  username: "metricbeat_monitoring_writer"
  password: "${metricbeat_monitoring_password}"
  ssl:
   verification_mode: "certificate"

That needs to be the monitoring cluster... sounds like it is pointing at the Source Cluster

29

Your output section need to point to the monitoring cluster

it is outputting to the monitoring cluster.

30

It can not be... there is no way the data could go back to the source cluster... unless it is pointed there

31

Here is my metricbeat.yml config

output.elasticsearch:
  hosts: ["https://esmonitor1.mynetwork.com:9200"]
  username: "metricbeat_monitoring_writer"
  password: "${metricbeat_monitoring_password}"
  ssl:
   verification_mode: "certificate"

My new elastic servers are elasticbase1.mynetwork.com and elasticbase2.mynetworkts.com

32

do you have the system module enabled? is it sending to the monitoring cluster.

There is something basic... are you sure you are accessing the correct metricbeat.yml

you can use the -c option from command line

Do you have CCS Setup?

Something basic... keep looking

run
metricbeat -c /full/path/metricbeat.yml -e -d "*"
and look for the connection information etc

33

I do not have the system module enabled.

Do you have CCS Setup?

I do not have cross cluster search enabled.

run metricbeat -c /full/path/metricbeat.yml -e -d "*"

My config is at /etc/metricbeat/metricbeat.yml

Here is a log related to connection


{"log.level":"info","@timestamp":"2024-02-21T13:25:43.402-0500","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://esmonitor1.mynetwork.com:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
34

Something Basic going on

Enable the system module see if the system data shows up in the monitoring cluster

35 
# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-elasticsearch.html

What version of metricbeat?

Why are you not using metricbeat 8.12.1... are you really using 7.10 that could be the problem... but your index names look like 8.12.1...

You have been upgrading?

36

Just download metricbeat 8.12.1 tar.gz un tar it... and configure and run in the foreground... that is what I would do

37

I have enabled the system module by renaming /etc/metricbeat/modules.d/system.yml.down to /etc/metricbeat/modules.d/system.yml

confirmed system module is loaded:

/usr/bin/metricbeat modules list
returned:

Enabled:
elasticsearch
kibana-xpack
system

Disabled:
activemq
aerospike
airflow
apache
appsearch
aws
awsfargate
azure
beat
beat-xpack
38

You should be using (But I think they are basically the same) just looking for reasons

$ ./metricbeat modules list
Enabled:
elasticsearch-xpack <---- HERE
system

Disabled:
activemq
39

Done!

/usr/bin/metricbeat modules list
Enabled:
elasticsearch-xpack
kibana-xpack
system

Disabled:
activemq
aerospike
airflow
apache
40

Also

/usr/bin/metricbeat test output
elasticsearch: https://esmonitor1.mynetwork.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: **monitoring cluster's ip**
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 8.12.1