Thanks for the detailed breakdown. It's just morning here, so when I dig out of the usual stuff, I'll take a look. Really appreciate it.
1 Like
I have similar changes to my kibana.yml.
I have a fleet server similar to yours that does not show any problems in kibana.
Differing from your setup, when I created the Elasticsearch integration, I followed these directions: Collecting Elasticsearch monitoring data with Elastic Agent | Elasticsearch Guide [8.11] | Elastic
which say to create a user with the remote_monitoring_collector role as a prerequisite and then in 5(c) to use that user info in the intergations Metrics->Settings->Advanced section. I don't see anything in that doc about setting certificate_authorities, and the entire SSL section is listed as optional on the integration config page.
I updated my integration to include both the user info and this certificate config, deployed it, and things started working.
My scope is also set to 'node', as it seemed to best match the instructions. As @ DaddyYusk commented, this is also confusing for those with dedicated masters and no load balancer.
I wasn't trying to install the Kibana integration, so I didn't review those instructions, but I imagine they will be equally valuable to people who try to follow the published instructions and end up looking for help.
Thanks for your reply!
1 Like
Here is my full .ymls... (sans comments) these are complete... not just snippets
# grep "^[^#;]" elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["stephenb-es-8-test"]
http.host: 0.0.0.0
# grep "^[^#;]" /etc/kibana/kibana.yml
server.host: "0.0.0.0"
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
elasticsearch.hosts: ['https://10.168.0.12:9200']
elasticsearch.serviceAccountToken: SDLKJFHSDLKJHDFa2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE3MDEyOTE5NTIzNDk6elV2NXlIS0RSbS1KUTNqODkyb1pCQQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1701291953176.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://10.168.0.12:9200'], ca_trusted_fingerprint: aslkdjfhasdlfkjhd2c971851afdf3b8cf593f5571e78f64f29545523a7}]
xpack.encryptedSavedObjects.encryptionKey: "askdjfh-2287346-laksdjfhaksfdjh-387246523984756"
1 Like
Yeah, we seem to leave that out a lot...
Self Signed Certs are painful...
I guess we just expect everyone to know you need to "carry around" your CA whenever you use self signed cert.
How many nodes / what is your config... follow my suggestions above... cluster is fine to assign to a data node if your cluster is not too big. If you use node but only point it to a single node you will not get all / complete monitoring data.
Yeah, the doc makes a lot of assumptions.
This is just a minimal cluster - three master nodes and two data nodes. The integration sends the info to 'localhost'.
Hi @stephenb,
And thanks again for your time!
Here is my setup :
- The "Kibana policy 1" has the "Kibana" integration
- The "Cluster policy 1" has the "Elasticsearch" integration
What exactly is not working?
Everything is working fine, only Stack Monitoring is only showing this :
Exactly What errors do you have?
No errors
What other changes have you made to your elasticsearch.yml and / or kibana.yml
# ELASTICSEARCH :
cluster.name: <REDACTED>
node.name: ovh-datah-1
node.roles: [ data_hot, data_content, ingest, transform ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.74.42
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
http.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.seed_hosts: ["192.168.74.30:9300", "192.168.74.31:9300", "192.168.74.32:9300"]
# KIBANA :
server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "http://ovh-kib-1:5601"
server.name: "ovh-kib-1"
elasticsearch.hosts: ["https://192.168.74.30:9200", "https://192.168.74.31:9200", "https://192.168.74.32:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "<REDACTED>"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ]
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
monitoring.ui.elasticsearch.username: "kibana_monitoring"
monitoring.ui.elasticsearch.password: "<REDACTED>"
xpack.encryptedSavedObjects.encryptionKey: <REDACTED>
xpack.reporting.encryptionKey: <REDACTED>
xpack.security.encryptionKey: <REDACTED>
If you have already pressed/selected self-monitoring, that makes changes that might need to be undone.
No didn't press it.
Are you sure that ca-cert is available and can be read by the agent?
Yes, readable by all.
Here is the setup for "Elasticsearch" integration :
Hope this helps!
Regards.
Quick update :
I've changed the "Elasticsearch" integration settings as below and now Stack Monitoring is showing data :
Should I be concerned with the fact that in Stack Monitoring :
- Elasticsearch is displaying "Self monitoring"
- Kibana "No monitoring"
Anyway thanks a lot for your help @stephenb
Take those out
Run this command
GET _cluster/settings
and provide the results
Sure!
Here it is after removing the 2 lines and restarting the Kibana service :
{
"persistent": {
"xpack": {
"monitoring": {
"migration": {
"decommission_alerts": "true"
}
}
}
},
"transient": {}
}
Hmmm try setting
Try
PUT _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"migration": {
"decommission_alerts": null
}
}
}
},
"transient": {}
}
Something still thinks self-monitoring but as long as you have data that is the important part
this may clear that flag... you can set it back if needed.
Well, sadly nothing changed.
But anyway now data is flowing in Stack Monitoring, so I can live with this state.
Thanks!
You have a buch of nodes are you sure one of the elasticsearch.yml does not have a monitoring setting in it?
you could also try to set
PUT _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"collection": {
"enabled": "true"
}
}
}
},
"transient": {}
}
then
PUT _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"collection": {
"enabled": null
}
}
}
},
"transient": {}
}
I really don't have an answer...
1 Like
Well, I go through all my elasticsearch.yml file on each node and there is no monitoring settings in them.
But after applying your suggestion below :
PUT _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"collection": {
"enabled": "true"
}
}
}
},
"transient": {}
}
Now Stack Monitoring > Kibana is showing this :
So I'm fine now, many thanks!
Have a great week-end.
Regards.
And now clear it !!
PUT _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"collection": {
"enabled": null
}
}
}
},
"transient": {}
}
If I clear it, I go back to the previous state where it says "No monitoring"...
You definitely want to leave that setting cleared / null ...
There is a bit stuck somewhere... if you really want it cleared you could stop all the monitoring ... delete / clear all the monitoring data and try again... that may / may not fix it... but if you are getting
Or dive into the Chrome Dev Tools and Find out which query that flag is coming back in ....
But if it is working... perhaps not...
1 Like
Hi @stephenb, and thanks a lot for the advice!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.