HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat"

The metric beats service status showing that connection estabilished to elasticsearch but I get this errors
metricbeats.yml

 path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
  host: "https://192.168.21.111:5601"

output.elasticsearch:
  hosts: ["http://192.168.21.111:9200"]
  username: "elastic"
  password: "elastic"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"info","@timestamp":"2023-01-25T13:19:55.084Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":547759543}},"memory":{"mem":{"usage":{"bytes":62668800}}}},"cpu":{"system":{"ticks":4530,"time":{"ms":230}},"total":{"ticks":10210,"time":{"ms":550},"value":10210},"user":{"ticks":5680,"time":{"ms":320}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":9},"info":{"ephemeral_id":"140f7ccf-3e4c-4b81-9448-1dd8e6ac48e3","uptime":{"ms":513193},"version":"8.6.0"},"memstats":{"gc_next":26245352,"memory_alloc":21400904,"memory_total":862290848,"rss":147709952},"runtime":{"goroutines":93}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":74,"active":0,"batches":6,"total":74},"read":{"bytes":2654},"write":{"bytes":153924}},"pipeline":{"clients":19,"events":{"active":0,"filtered":6,"published":74,"total":80},"queue":{"acked":74}}},"metricbeat":{"elasticsearch":{"ccr":{"events":3,"failures":3},"cluster_stats":{"events":3,"failures":3},"enrich":{"events":3,"failures":3},"index":{"events":3,"failures":3},"index_recovery":{"events":3,"failures":3},"index_summary":{"events":3,"failures":3},"ml_job":{"events":3,"failures":3},"node_stats":{"events":3,"failures":3},"shard":{"events":3,"failures":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":10,"success":10},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":9,"success":9},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.07,"15":0.01,"5":0.04,"norm":{"1":0.0044,"15":0.0006,"5":0.0025}}}},"ecs.version":"1.6.0"}}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.ccr: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index_recovery: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.ml_job: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.enrich: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index_summary: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.091Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.node_stats: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.091Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}

Are username/pass OK? Test with curl

curl -u elastic:elastic  -X GET "192.168.21.111:9200/_cluster/health?pretty"
{
  "cluster_name" : "kds",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 19,
  "active_shards" : 19,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 86.36363636363636
}

Set user/pass for Kibana
Do you have enabled modules? If you do have, for instance the elasticsearch module, set also there in the module yml file.

 metricbeat modules list
Enabled:
elasticsearch-xpack
kibana
system

vim /etc/metricbeat/modules.d/kibana.yml

- module: kibana
  #metricsets:
  #  - status
  period: 10s
  hosts: ["https://192.168.21.111:5601"]
  #basepath: ""
  username: "elastic"
  password: "elastic"

elastic is super user, I can login via kibana interface

You have enabled: elasticsearch-xpack and kibana. For both you have to set params.
Note HTTP is for ES, and HTTPS for kibana. Check also for elasticsearch-xpack, use refence

thanks for reply,
when I set xpack.security.enabled: false in elasticsearch.yml then metricbeats connecting and fetching data well, but securitiy section in kibana gui returning with errors.
Actually I'm a little bit confused about certificates. elasticsearch-certutil generate *.p12 files but in metricbeat.yml claim #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] a ca.pem file
How can I generate ca.pem to metricbeats that compatible with elasticsearch ssl
In elasticsearch I follow the documentation and set ssl by

/usr/share/elasticsearch/bin/elasticsearch-certutil ca
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

I found this command and its work
openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elastic-ca.pem
topic can be closed

1 Like