HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat"

bagafoot · January 25, 2023, 1:37pm

The metric beats service status showing that connection estabilished to elasticsearch but I get this errors
metricbeats.yml

 path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
  host: "https://192.168.21.111:5601"

output.elasticsearch:
  hosts: ["http://192.168.21.111:9200"]
  username: "elastic"
  password: "elastic"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"info","@timestamp":"2023-01-25T13:19:55.084Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":547759543}},"memory":{"mem":{"usage":{"bytes":62668800}}}},"cpu":{"system":{"ticks":4530,"time":{"ms":230}},"total":{"ticks":10210,"time":{"ms":550},"value":10210},"user":{"ticks":5680,"time":{"ms":320}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":9},"info":{"ephemeral_id":"140f7ccf-3e4c-4b81-9448-1dd8e6ac48e3","uptime":{"ms":513193},"version":"8.6.0"},"memstats":{"gc_next":26245352,"memory_alloc":21400904,"memory_total":862290848,"rss":147709952},"runtime":{"goroutines":93}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":74,"active":0,"batches":6,"total":74},"read":{"bytes":2654},"write":{"bytes":153924}},"pipeline":{"clients":19,"events":{"active":0,"filtered":6,"published":74,"total":80},"queue":{"acked":74}}},"metricbeat":{"elasticsearch":{"ccr":{"events":3,"failures":3},"cluster_stats":{"events":3,"failures":3},"enrich":{"events":3,"failures":3},"index":{"events":3,"failures":3},"index_recovery":{"events":3,"failures":3},"index_summary":{"events":3,"failures":3},"ml_job":{"events":3,"failures":3},"node_stats":{"events":3,"failures":3},"shard":{"events":3,"failures":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":10,"success":10},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":9,"success":9},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.07,"15":0.01,"5":0.04,"norm":{"1":0.0044,"15":0.0006,"5":0.0025}}}},"ecs.version":"1.6.0"}}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.ccr: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index_recovery: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.ml_job: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.enrich: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.090Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index_summary: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.091Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.node_stats: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
Jan 25 13:19:55 elasticsearch06s01 metricbeat[77710]: {"log.level":"error","@timestamp":"2023-01-25T13:19:55.091Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}

Rios · January 25, 2023, 2:11pm

Are username/pass OK? Test with curl

bagafoot · January 25, 2023, 2:40pm

curl -u elastic:elastic  -X GET "192.168.21.111:9200/_cluster/health?pretty"
{
  "cluster_name" : "kds",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 19,
  "active_shards" : 19,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 86.36363636363636
}

Rios · January 25, 2023, 2:48pm

Set user/pass for Kibana
Do you have enabled modules? If you do have, for instance the elasticsearch module, set also there in the module yml file.

bagafoot · January 25, 2023, 2:57pm

 metricbeat modules list
Enabled:
elasticsearch-xpack
kibana
system

vim /etc/metricbeat/modules.d/kibana.yml

- module: kibana
  #metricsets:
  #  - status
  period: 10s
  hosts: ["https://192.168.21.111:5601"]
  #basepath: ""
  username: "elastic"
  password: "elastic"

elastic is super user, I can login via kibana interface

Rios · January 25, 2023, 3:41pm

You have enabled: elasticsearch-xpack and kibana. For both you have to set params.
Note HTTP is for ES, and HTTPS for kibana. Check also for elasticsearch-xpack, use refence

bagafoot · January 30, 2023, 8:52am

thanks for reply,
when I set xpack.security.enabled: false in elasticsearch.yml then metricbeats connecting and fetching data well, but securitiy section in kibana gui returning with errors.
Actually I'm a little bit confused about certificates. elasticsearch-certutil generate *.p12 files but in metricbeat.yml claim #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] a ca.pem file
How can I generate ca.pem to metricbeats that compatible with elasticsearch ssl
In elasticsearch I follow the documentation and set ssl by

/usr/share/elasticsearch/bin/elasticsearch-certutil ca
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

bagafoot · January 30, 2023, 11:02am

I found this command and its work
openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elastic-ca.pem
topic can be closed

rachel_gomez · February 2, 2023, 9:59am

Steps to Fix the 401 Unauthorized Error
Confirm the URL Is Correct. This might sound obvious, but the 401 error code might appear if the user entered the wrong URL in the browser's address bar.
Clear User End Issues.
Check Authentication Credentials.
Disable Password Protection.
Troubleshoot the Code.

Regards,
Rachel Gomez

system · March 2, 2023, 12:00pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.