Hi,
I have 3 node cluster elasticsearch (every on different VM (RHEL8)- node 1, 2 ,3), kibana is installed at one of this nodes (node 2).
There is config for one node (other's the same, but with diff names):
# ---------------------------------- Cluster --------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: es-cluster
# ---------------------------------- END Cluster ----------------------------
# ------------------------------------ Node ---------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
node.roles: [ data, master,data_content,data_hot, data_warm, data_cold, data_frozen ]
# ------------------------------------ END Node -----------------------------
bootstrap.memory_lock: true
# ----------------------------------- Paths ---------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
# --------------------------------- END Paths -------------------------------
# ---------------------------------- Network --------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: nameOfTheMachine
http.port: 9200
# -------------------------------- END Network ------------------------------
# --------------------------------- Discovery -------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["IP_Node_1", "IP_Node_2", "IP_Node_3"]
discovery.type: multi-node
# ------------------------------- END Discovery -----------------------------
#----------------------- BEGIN SECURITY CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 02-11-2022 10:14:20
#
# ---------------------------------------------------------------------------
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.monitoring.collection.enabled: true
#----------------------- END SECURITY CONFIGURATION -------------------------
Certs was generated as one for all nodes (at one of the nodes created CA (default elasticserach) then create http.p12 and elastic-certificates.p12) then copy to all others nodes and sign it with commands:
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
After this, I can curl all nodes (check cluster health etc.) - everything return "correct" response.
Next I install kibana at node 2, with config:
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "IP_node2"
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
server.publicBaseUrl: "https://IP_node2:5601"
# The Kibana server's name. This is used for display purposes.
server.name: "kibana"
# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.keystore.path: /etc/kibana/certs/http.p12
server.ssl.keystore.password: "PassForCerthttp.p12"
# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://IP_node1:9200", "https:IP_node2:9200", "https://IP_node3:9200"]
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"
elasticsearch.password: "PassForKibanaSystem"
# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
elasticsearch.ssl.verificationMode: "certificate"
# Enables you to specify a file where Kibana stores log output.
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid
monitoring.ui.ccs.enabled: false
xpack.encryptedSavedObjects.encryptionKey: keyGenerted
xpack.reporting.encryptionKey: keyGenerted
xpack.security.encryptionKey: keyGenerted
there index settings:
PUT _ilm/policy/name_api_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1h"
}
}
},
"delete": {
"min_age": "29d",
"actions": {
"delete": {}
}
}
}
}
}
PUT _template/name_api_template
{
"index_patterns": ["name-api-*"],
"settings": {
"number_of_shards": 2,
"number_of_replicas": 2,
"index.lifecycle.name": "name_api_policy",
"index.lifecycle.rollover_alias": "name-api"
}
}
PUT name-api-000001
{
"mappings": {
....
},
"settings": {
"index": {
"number_of_shards": "2",
"number_of_replicas": "2"
}
}
}
POST _aliases
{
"actions" : [
{ "add" : { "index" : "name-api-000001", "alias" : "name-api", "is_write_index": true} }
]
}
I can login to Kibana, check ILM, indexes, use devtools (send request), stack monitoring etc. Everything is correct (at kibana logs and cluster logs i dont see any warrnings or errors)
but when ILM reach the conditions then at all nodes elasticsearch logs i can see:
"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", what's strange, becouse i can curl for anything elasticsearch nodes with permission kibana_system like:
curl -X GET "https://node1/2/3:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty" --key certificates/elasticsearch-ca.pem -k -u kibana_system
This INFO says that credentials are incorrect, but i can request with this credential (like above).
That communicate only shows when index with ILM reach rollover state, when i request for
GET /name-api-000001/_ilm/explain
i get following response :
{
"indices": {
"name-api-000001": {
"index": "name-api-000001",
"managed": true,
"policy": "name_api_policy",
"index_creation_date_millis": 1672753491558,
"time_since_index_creation": "17.13h",
"lifecycle_date_millis": 1672753491558,
"age": "17.13h",
"phase": "hot",
"phase_time_millis": 1672753492170,
"action": "rollover",
"action_time_millis": 1672753492374,
"step": "check-rollover-ready",
"step_time_millis": 1672753492374,
"phase_execution": {
"policy": "name_api_policy",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1h"
}
}
},
"version": 1,
"modified_date_in_millis": 1672753363104
}
}
}
}
Can someone help - why does this INFO shows, when credentials are ok, and I can do everything by browser (on my desktop - not exactly at nodes)?
Why index dont create next one (000002) ?
Index has flag: "is_write_index": true
