Error fetching data for metricset kibana.node_rules: error making http request

Elastic Observability
1

Hi Folks,
I am almost getting mad. Please help me our here. I am trying to collect kibana monitoring data with metricbeat. I get strangely the following error:

{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.node_rules: error making http request: Get \"https://10.10.10.11:5601/api/status\.  x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"}

{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.node_actions: error making http request: Get \"https://10.10.10.11:5601/api/status.  x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"}

metricset kibana.stats: error making http request: Get \"https://10.10.10.11:5601/api/status\": x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"}.  x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"}

it is saying that unknow authority but I generated the certs with the self generated CA

here is the test output of metricbeat:

root@kibana:/etc/metricbeat# metricbeat test  output
elasticsearch: https://10.10.10.9:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.10.10.9
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK

here is metricbeat.yml config:
-kibana

 host: "https://10.10.10.11:5601"
  username: "user"
  password: "passw"
  • elastic.output:
#hosts: ["10.10.10.9:9200", "10.10.10.10:9200"]
  hosts: ["10.10.10.9:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "huseyin"
  #ssl.certificate: "/etc/kibana/certs/chamber1/chamber1.crt"
  #ssl.key: "/etc/kibana/certs/chamber1/chamber1.key"
  ssl.certificate_authorities: "/etc/kibana/certs/elasticsearch

here is kibana.yml:

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://10.10.10.9:9200", "https://10.10.10.10:9200"]


# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "user"
elasticsearch.password: "passw"
ssl.certificate: "/etc/kibana/certs/chamber1/chamber1.crt"
ssl.key: "/etc/kibana/certs/chamber1/chamber1.key"
ssl.certificate_authorities: "/etc/kibana/certs/elasticsearch-ca.pem"

here is the kibana authority cert:

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]

cert generate command:

#Generate crt and key file: 
./bin/elasticsearch-certutil cert --pem  --silent --in instances.yml --out certs.zip -ca /usr/share/elasticsearch/elastic-stack-ca.p12

here is the modules.d/kibana-xpack.yml file:

module: kibana
  # metricsets:
  #  - stats
  period: 10s
  xpack.enabled: true
  hosts: ["https://10.10.10.11:5601"]
  username: "kibana_huseyin"
  password: "huseyin"
  #basepath: ""
  server.ssl.enabled: true
  server.ssl.certificate: "/etc/kibana/certs/kibana-server.crt"
   server.ssl.key: "/etc/kibana/certs/kibana-server.key"
  #ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]

Can you please help me with this issue? how can I test or verify that I have the same CA .
I didnt see any section in the official side regarding, securing the kibana elasticsearch logstash with self generated CA and ssl certs.

2

@husoelasticbe

Please share your entire kibana.yml

Use 3 backticks to format code.

What URL do you use to bring up kibana in browser?

Pretty sure you have this incorrect backwards

3

here is the kibana.yml:

https://gist.github.com/husobaba/06939d8e5e57d6368e115b30d8633a94

here is the metricbeat:

https://gist.github.com/husobaba/34583472988f65917d04b5ed2463efe5

What URL do you use to bring up kibana in browser?
https://10.10.10.11:5601/

I hope you can check the yml files with gist.

4

In kibana.yml

These are not needed / meaningless since they are not prefixed with server or elasticsearch

ssl.certificate: "/etc/kibana/certs/chamber1/chamber1.crt"
ssl.key: "/etc/kibana/certs/chamber1/chamber1.key"
ssl.certificate_authorities: "/etc/kibana/certs/elasticsearch-ca.pem"

In the module

Not sure why you configured that way... You commented out lines.. changed things..

Start over on that and just enable, put in the metricsets and put in the certificate authority setting and user password etc

Not other stuff...

Remember in the module is about connecting to Kibana

Please share when you fixed

5

Ok I will try to take them out. and test that way. Nevertheless, Elasticsearch monitoring data is working fine. when this will be fixed I will share here as well.

6

It is really frustrating, I am still not able to collecting monitoring data using metricbeat. Here is the way that I followed. (I elaborated all the steps that I have taken)
Encrypt traffic between your browser and Kibana:

*** Kibana and the browser secure
cd /usr/share/elasticsearch
./bin/elasticsearch-certutil csr -name kibana-server 

mkdir CA_kibana
cp csr-bundle.zip CA_kibana/
cd CA_kibana/
unzip csr-bundle.zip 
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem -ca /usr/share/elasticsearch/elastic-stack-ca.p12 -name kibana-server
this command will generate the certificate-bundle.zip
unzip that file and take the crt file and copy this to the kibana server.

***** Change the following parameters:

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana-server.crt
server.ssl.key: /etc/kibana/certs/kibana-server.key

At this point my kibana and browser communications are in https.

I installed the metricbeat where my Elasticsearch is and I enabled the module elasticsearch-xpack to be able to collect monitoring data of elasticsearch.
Here are the modifications:

Metricbeat.yml:

output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  hosts: ["10.10.10.9:9200", "10.10.10.10:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "elastic"
  #ssl.certificate: "/usr/share/elasticsearch/huseyin/chamber1/chamber1.crt"
  #ssl.key: "/usr/share/elasticsearch/huseyin/chamber1/chamber1.key"
  ssl.certificate_authorities: "/usr/share/elasticsearch/old_certs/kibana/elasticsearch-ca.pem"

elasticsearch-xpack.yml file:

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.11/metricbeat-module-elasticsearch.html

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://10.10.10.9:9200"]
  username: "elastic"
  password: "elastic"
  ssl.enabled: true  
  ssl.certificate_authorities: "/usr/share/elasticsearch/old_certs/kibana/elasticsearch-ca.pem"

Until here everything is ok and smoothly configured. Additional info: The pem file: elasticsearch-ca.pem is coming from while Encrypting TLS on the HTTP client communications for Elasticsearch

command: ./bin/elasticsearch-certutil http

when I start to perform the same approach to collect kibana data I got the issue regarding :

error making http request: Get \"https://10.10.10.11:5601/api/status\": x509: cannot validate certificate for 10.10.10.11 because it doesn't contain any IP SANs","service.name

then when I try to generate new certificate wherein we can see the SAN (generating with my own CA)

command: ./bin/elasticsearch-certutil cert --pem  --silent --in instances.yml --out certs.zip -ca /usr/share/elasticsearch/elastic-stack-ca.p12

***example instances.yml file:

instances: 
  - name: "chamber1"
	- ip:
	- "10.10.10.9"
  - name: "chamber2"
	- ip:
	- "10.10.10.10"
	  - name: "logstash"
	- ip:
	- "10.10.10.13"

then I added the newly generated certificate in the configuration files.
metricbeat.yml:

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "https://10.10.10.11:5601"
  ssl.enabled: true
  ssl.certificate_authorities: "/etc/kibana/certs/kibana5/kibana.crt"
  username: "kibana"
  password: "kibana"
  
 output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.10.10.9:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "elastic"  
  ssl.certificate_authorities: "/etc/kibana/certs/kibana5/kibana.crt"

kibana-xpack.yml:


# Module: kibana
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.11/metricbeat-module-kibana.html

- module: kibana
  #metricsets:
  #  - stats
  period: 10s
  xpack.enabled: true
  hosts: ["https://10.10.10.11:5601"]
  username: "kibana"
  password: "kibana"
  ssl.enabled: true
  ssl.certificate_authorities: "/etc/kibana/certs/kibana5/kibana.crt"

Then I get the following error:
error making http request: Get "https://10.10.10.11:5601/api/status\": x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"}

While I'm generating the cert I used my own generated CA. Thus, do you have an idea where I am making a mistake? It will be very helpful to understand this procedure regarding secure collecting monitoring data of Kibana with metricbeat. Because this is not writing anywhere in the official website or at least I am missing it.

7

@husoelasticbe Sorry it is frustrating but this is really all cert stuff, not that different then any other web server etc...

BTW you have muted this topic.

You are generating your own certs via csr see here so the assumption is that you have a working knowledge of certs and you have an actual certificate authority that you are submitting your csrs to. Are you intentionally using csr ?
If you are NOT using an actual csr you should not be using csr mode.

The csr mode generates certificate signing requests (CSRs) that you can send to a trusted certificate authority to obtain signed certificates.

In general this is SSL Cert Stuff.... Generate a CA then Generate Certs.
Where the settings ask for CAs ... use CAs
Where the setting ask for certs ... use Certs
You seem to be confusing these two.
Also in sections where the code is communicating with Elastic ... use the elastic CAs' / Certs
in sections where the code is communicating with Kibana ... use the Kibana CAs' / Certs
And you can use the same CA for all if you want.

Overall I would get Elasticsearch and Kibana working... make sure it is working...
Then setup metricbeat

8

I didnt know it. I changed it to watching

9

I will redo everything and I will write everything step by step.

10

Let's back up a bit.
What are you trying to accomplish? And I mean that from a cert/SSL perspective

If you just install from scratch and the the stack Auto configure security it will set everything up for you with the exception of https on Kibana.

I highly recommend reading this section in detail. Walk through the sections....

And even for the Kibana setup

11

I look those pages and I made all the configuration again, now I can monitor them with metricbeat.
I made the exactly the same steps: (again).

When I setup the basic security, I used the commands

generate ca:

./bin/elasticsearch-certutil ca

generate the certificate and private key:

./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

encrytion of internode communications with TLS/SSL:

I used the elastic-certificates.p12 which is previous command output.

this way my elasticsearch nodes are secured TLS/SSL.

next step was "Set up basic security for the Elastic Stack plus secured HTTPS traffic"

Encrypt HTTP client communications for Elasticsearch. I used the command (like kibana to elasticsearch) ===>

./bin/elasticsearch-certutil http

like you mentioned:

Also in sections where the code is communicating with Elastic ... use the elastic CAs' / Certs
in sections where the code is communicating with Kibana ... use the Kibana CAs' / Certs
And you can use the same CA for all if you want.

And I used the same CA for all, Thank you for your guidance Stephen

While I am reading the sections, I also saw that we can also collect data with elastic agent after the version 8.5. Even with Fleet, we can centralize all those agents and their policies, we can check them they are healthy or not. It sounds great to me!
Currently I have 8.11 so I will try to set this up and see the differences. Eventually I can share my findings here.
Once again thank you for your help.

1 Like