ELK security setup

Diler_Mohammed · December 10, 2024, 9:38pm

Hello all,

I'm currently running a manual security setup for my elasticsearch, logstash, and kibana docker containers. I've setup dockerfiles, and docker-entrypoints for each of these containers.

Also I generated self signed certs in a certs directory using elastic cert util on my host machine that is being mounted to each of the docker containers certs directories.

I am able to verify that ca.crt and ca.key is present as well as instance.crt and instance.key on both the host machine as well as each individual containers file directory. However when running these containers I'm getting these errors:

Kibana logs:

[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate

On the elasticsearch logs I'm getting this:

"log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[faf897bfe949][transport_worker][T#16]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"LNS7ImfmT-KAEJNp6S-Dmg","elasticsearch.node.id":"tmUYbAvyRTyjvwgIS6PULA","elasticsearch.node.name":"faf897bfe949","elasticsearch.cluster.name":"elasticsearch"}
2024-12-10 16:06:21 {"@timestamp":"2024-12-10T21:06:21.017Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.18.0.2:9200, remoteAddress=/172.18.0.1:54176}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[faf897bfe949][transport_worker][T#14]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"LNS7ImfmT-KAEJNp6S-Dmg","elasticsearch.node.id":"tmUYbAvyRTyjvwgIS6PULA","elasticsearch.node.name":"faf897bfe949","elasticsearch.cluster.name":"elasticsearch"}

Logstash container logs:

[logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.18.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.18.0.2] failed: Connection refused>}

Setting up the whole security for the elk stack has been frustrating. Any guidance on where to go from here, and what could be some things to double check to ensure the whole setup is correct will be greatly appreciated!

strawgate · December 11, 2024, 12:53am

I would strongly recommend looking at elastic cloud for kubernetes if you're open to deploying on k8s as it really simplifies cluster deployment with the ECK operator and Cards: Elastic Cloud on Kubernetes [2.15] | Elastic

For dev and test I run KIND which allows you to deploy Kubernetes nodes on Docker and then I deploy ECK onto my KIND cluster.

If you'd prefer a docker solution, there is a great blog series here with some samples Getting started with the Elastic Stack and Docker-Compose | Elastic Blog

For plain docker you could also check out: Install Elasticsearch with Docker | Elasticsearch Guide [8.16] | Elastic

Now for your current setup, in order to help you, you'd probably need to share the elasticsearch.yml, kibana.yml, Logstash.yml, the entry point scripts you created and the various deploy commands you used to understand where the issue may be coming from.

But my guess is there's an issue with the ca certs provided to kibana and Logstash as well as the current Elasticsearch hosts configuration for them as well

Diler_Mohammed · December 11, 2024, 9:59pm

Hi thank you for the reply,

So a quick background: The main reason for my current approach is that this setup is being deployed in AWS, so I’m avoiding the docker-compose method. Instead, I’m using Dockerfiles and custom entry points to automate certificate generation, security configuration, and deployment directly within the container lifecycle.

Elasticsearch.yml

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/node/instance.key
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/node/instance.crt
xpack.security.http.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certs/ca/ca.crt"]

Logstash.yml

xpack.monitoring.enabled: false
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate_authorities: ["/usr/share/logstash/config/certs/ca/ca.crt"]

Kibana.yml

server.host: "0.0.0.0"
elasticsearch.hosts: ["https://elasticsearch:9200"]
# SSL/TLS Settings
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/certs/ca/ca.crt"]
elasticsearch.username: "elastic"
elasticsearch.password: "${ELASTIC_PASSWORD}"
# Optional: Turn on TLS for Kibana itself (if required)
server.ssl.enabled: true
server.ssl.certificate: "/usr/share/kibana/config/certs/node/instance.crt"
server.ssl.key: "/usr/share/kibana/config/certs/node/instance.key"

Logstash.conf

input {
  file {
    path => "/usr/share/logstash/data/jsondocs/*.json"
    start_position => "beginning"
    sincedb_path => "nul"
    codec => json
    type => "json"
    add_field => { "filename" => "%{[@metadata][path]}" }
  }
}
filter {
  if [type] == "json" {
    mutate {
      gsub => [ "filename", ".*/", "" ]
      gsub => [ "filename", "\.json", "" ]
    }
    mutate {
      remove_field => ["@version", "host", "path"]
    }
  }
}
output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    user => "elastic"
    password => "${elasticsearch.password}"
    cacert => "/usr/share/logstash/config/certs/ca/ca.crt"
    index => "%{filename}"
  }
  stdout { codec => rubydebug }
}

Elasticsearch dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:8.15.3
COPY elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
USER root
RUN chmod +x /usr/local/bin/docker-entrypoint.sh && \
    mkdir -p /usr/share/elasticsearch/config/certs && \
    chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config/certs
USER elasticsearch
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

Elasticsearch docker entrypoint

#!/bin/bash
set -e
CERTS_DIR="/usr/share/elasticsearch/config/certs"
if [ ! -d "$CERTS_DIR" ]; then
  mkdir -p "$CERTS_DIR"
  chown elasticsearch:elasticsearch "$CERTS_DIR"
fi
if [ ! -f "$CERTS_DIR/elastic-stack-ca.p12" ]; then
  /usr/share/elasticsearch/bin/elasticsearch-certutil ca --silent --pem --out "$CERTS_DIR/elastic-stack-ca.p12"
fi
if [ ! -f "$CERTS_DIR/node-certificates.zip" ]; then
  /usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --pem \
    --ca "$CERTS_DIR/elastic-stack-ca.p12" \
    --out "$CERTS_DIR/node-certificates.zip"
  unzip "$CERTS_DIR/node-certificates.zip" -d "$CERTS_DIR"
fi
exec /usr/share/elasticsearch/bin/elasticsearch

Logstash dockerfile

FROM docker.elastic.co/logstash/logstash:8.15.3
COPY logstash.yml /usr/share/logstash/config/logstash.yml
COPY pipeline/logstash.conf /usr/share/logstash/pipeline/logstash.conf
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
USER root
RUN chmod +x /usr/local/bin/docker-entrypoint.sh && \
    mkdir -p /usr/share/logstash/config/certs
USER logstash
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

Logstash docker entrypoint

#!/bin/bash
set -e
CERTS_DIR="/usr/share/logstash/config/certs"
if [ ! -f "$CERTS_DIR/elastic-stack-ca.p12" ]; then
  echo "CA certificate missing at $CERTS_DIR. Please ensure the certs are mounted properly."
  exit 1
fi
if [ ! -f /usr/share/logstash/config/logstash.keystore ]; then
  /usr/share/logstash/bin/logstash-keystore create --silent
  echo "$ELASTIC_PASSWORD" | /usr/share/logstash/bin/logstash-keystore add elasticsearch.password --stdin --force
fi
exec /usr/share/logstash/bin/logstash

Kibana dockerfile

FROM docker.elastic.co/kibana/kibana:8.15.3
COPY kibana.yml /usr/share/kibana/config/kibana.yml
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
USER root
RUN chmod +x /usr/local/bin/docker-entrypoint.sh && \
    mkdir -p /usr/share/kibana/config/certs
USER kibana
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

Kibana docker entrypoint

#!/bin/bash
set -e
CERTS_DIR="/usr/share/kibana/config/certs"
if [ ! -d "$CERTS_DIR" ]; then
  mkdir -p "$CERTS_DIR"
  chown kibana:kibana "$CERTS_DIR"
fi
if [ ! -f "$CERTS_DIR/elastic-stack-ca.p12" ]; then
  echo "CA certificate missing at $CERTS_DIR. Ensure the certs are mounted correctly."
  exit 1
fi
exec /usr/local/bin/kibana-docker

These are the files I used in my attempt to setup the security for my elk stack.

Thank you!

strawgate · December 11, 2024, 10:09pm

It might make sense to stop the Logstash container and restart the Kibana and Elasticsearch containers so that we can make sure that any errors we're seeing in the Elasticsearch log are specific to Kibana. Can you share the full container logs from the Elasticsearch and Kibana containers after restarting them? Once we have Kibana and Elasticsearch working then we can add back in Logstash.

Once you've gotten the Kibana container running, can you exec into it and curl https://elasticsearch:9200 with creds and make sure that you can connect? While you're execed in it might be a good opportunity to make sure the kibana.yml in the container is the one you expect it to be as well.

I also don't see you passing hostnames to the cert generation and I thought that using the --silent param required providing a yml file with information like the hostname. Can you confirm that the ES node has elasticsearch as the name on the certificate?

Diler_Mohammed · December 12, 2024, 8:27pm

I ran just the Elasticsearch and Kibana containers and these were the logs:

Elasticsearch logs (full log is very long so I included where it fails and some more lines above it)

2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.492Z", "log.level": "INFO", "message":"adding index template [metrics-apm.service_transaction.60m@template] for index patterns [metrics-apm.service_transaction.60m-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.499Z", "log.level": "INFO", "message":"adding index template [metrics-apm.app@template] for index patterns [metrics-apm.app.*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.504Z", "log.level": "INFO", "message":"adding index template [metrics-apm.service_summary.1m@template] for index patterns [metrics-apm.service_summary.1m-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.512Z", "log.level": "INFO", "message":"adding index template [traces-apm@template] for index patterns [traces-apm-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.558Z", "log.level": "INFO", "message":"adding index lifecycle policy [.monitoring-8-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.665Z", "log.level": "INFO", "message":"adding ingest pipeline behavioral_analytics-events-final_pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.666Z", "log.level": "INFO", "message":"adding ingest pipeline logs-default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.666Z", "log.level": "INFO", "message":"adding ingest pipeline logs@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.666Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.transaction@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline logs-apm.error@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline apm@pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.service_summary@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.app@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.service_transaction@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline traces-apm.rum@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline traces-apm@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline logs-apm.app@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.service_destination@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.667Z", "log.level": "INFO", "message":"adding ingest pipeline ent-search-generic-ingestion", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.668Z", "log.level": "INFO", "message":"adding ingest pipeline search-default-ingestion", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.668Z", "log.level": "INFO", "message":"adding ingest pipeline logs@json-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.668Z", "log.level": "INFO", "message":"adding ingest pipeline logs@json-message", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.675Z", "log.level": "INFO", "message":"adding component template [behavioral_analytics-events-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.679Z", "log.level": "INFO", "message":"adding component template [logs@settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.682Z", "log.level": "INFO", "message":"adding component template [logs-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.741Z", "log.level": "INFO", "message":"adding index template [behavioral_analytics-events-default] for index patterns [behavioral_analytics-events-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.752Z", "log.level": "INFO", "message":"adding index template [logs] for index patterns [logs-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.800Z", "log.level": "INFO", "message":"adding index lifecycle policy [ml-size-based-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.847Z", "log.level": "INFO", "message":"adding index lifecycle policy [logs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.889Z", "log.level": "INFO", "message":"adding index lifecycle policy [synthetics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.932Z", "log.level": "INFO", "message":"adding index lifecycle policy [7-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:15:59 {"@timestamp":"2024-12-12T19:15:59.985Z", "log.level": "INFO", "message":"adding index lifecycle policy [30-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.052Z", "log.level": "INFO", "message":"adding index lifecycle policy [90-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.117Z", "log.level": "INFO", "message":"adding index lifecycle policy [metrics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.186Z", "log.level": "INFO", "message":"adding index lifecycle policy [180-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.264Z", "log.level": "INFO", "message":"adding index lifecycle policy [365-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.348Z", "log.level": "INFO", "message":"adding index lifecycle policy [metrics@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.438Z", "log.level": "INFO", "message":"adding index lifecycle policy [7-days@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.496Z", "log.level": "INFO", "message":"adding index lifecycle policy [180-days@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.551Z", "log.level": "INFO", "message":"adding index lifecycle policy [90-days@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.629Z", "log.level": "INFO", "message":"adding index lifecycle policy [synthetics@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.702Z", "log.level": "INFO", "message":"adding index lifecycle policy [logs@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.789Z", "log.level": "INFO", "message":"adding index lifecycle policy [30-days@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.856Z", "log.level": "INFO", "message":"adding index lifecycle policy [365-days@lifecycle]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:00 {"@timestamp":"2024-12-12T19:16:00.922Z", "log.level": "INFO", "message":"adding index lifecycle policy [ilm-history-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.011Z", "log.level": "INFO", "message":"adding index lifecycle policy [slm-history-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.073Z", "log.level": "INFO", "message":"adding index lifecycle policy [watch-history-ilm-policy-16]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.128Z", "log.level": "INFO", "message":"adding index lifecycle policy [.deprecation-indexing-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.234Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-file-tohost-data-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.287Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-actions-results-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.332Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-file-fromhost-data-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.383Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-file-fromhost-meta-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.429Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-file-tohost-meta-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.526Z", "log.level": "INFO", "message":"Node [{75f586ee2499}{s1tFo3XmS_OOxej65utjEQ}] is selected as the current health node.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][management][T#5]","log.logger":"org.elasticsearch.health.node.selection.HealthNodeTaskExecutor","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.627Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.631Z", "log.level": "INFO", "message":"license [808f8224-bfa8-420a-b4b9-f1eace5115a9] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.684Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.internal@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.684Z", "log.level": "INFO", "message":"adding ingest pipeline traces-apm@pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:16:01 {"@timestamp":"2024-12-12T19:16:01.684Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm@pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:22:26 {"@timestamp":"2024-12-12T19:22:26.541Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][transport_worker][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:33:20 {"@timestamp":"2024-12-12T19:33:20.991Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][transport_worker][T#4]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 14:37:53 {"@timestamp":"2024-12-12T19:37:53.478Z", "log.level": "WARN", "message":"http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/172.18.0.2:9200, remoteAddress=/172.18.0.2:45316}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][transport_worker][T#9]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}
2024-12-12 15:01:43 {"@timestamp":"2024-12-12T20:01:43.424Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[75f586ee2499][transport_worker][T#11]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"MK2P_6ErRFCknjnnYwdwlg","elasticsearch.node.id":"s1tFo3XmS_OOxej65utjEQ","elasticsearch.node.name":"75f586ee2499","elasticsearch.cluster.name":"elasticsearch"}

Diler_Mohammed · December 12, 2024, 8:29pm

Kibana

2024-12-12 14:15:40 Running import script for Kibana objects...
2024-12-12 14:15:40 Waiting for Kibana to be ready...
2024-12-12 14:15:40 Kibana is not ready yet... waiting...
2024-12-12 14:15:41 Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.15/production.html#openssl-legacy-provider
2024-12-12 14:15:42 {"log.level":"info","@timestamp":"2024-12-12T19:15:42.429Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.7.0","env":{"pid":1,"proctitle":"/usr/share/kibana/bin/../node/glibc-217/bin/node","os":"linux 5.15.167.4-microsoft-standard-WSL2","arch":"x64","host":"a7e61ac50600","timezone":"UTC+00","runtime":"Node.js v20.15.1"},"config":{"active":{"source":"start","value":true},"breakdownMetrics":{"source":"start","value":false},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false},"centralConfig":{"source":"start","value":false},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["git_rev","3933429968aafb1ba31319fc38649d0f974044bf"]],"sourceValue":{"git_rev":"3933429968aafb1ba31319fc38649d0f974044bf"}},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureSpanStackTraces":{"source":"start","sourceValue":false},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.15.3","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.7.0"}
2024-12-12 14:15:42 Native global console methods have been overridden in production environment.
2024-12-12 14:15:44 [2024-12-12T19:15:44.446+00:00][INFO ][root] Kibana is starting
2024-12-12 14:15:44 [2024-12-12T19:15:44.532+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
2024-12-12 14:15:45 Kibana is not ready yet... waiting...
2024-12-12 14:15:50 Kibana is not ready yet... waiting...
2024-12-12 14:15:54 [2024-12-12T19:15:54.906+00:00][INFO ][plugins-service] The following plugins are disabled: "cloudChat,cloudExperiments,cloudFullStory,profilingDataAccess,profiling,securitySolutionServerless,serverless,serverlessObservability,serverlessSearch".
2024-12-12 14:15:54 [2024-12-12T19:15:54.969+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
2024-12-12 14:15:55 [2024-12-12T19:15:55.062+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
2024-12-12 14:15:55 [2024-12-12T19:15:55.096+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
2024-12-12 14:15:55 [2024-12-12T19:15:55.303+00:00][INFO ][plugins-system.standard] Setting up [166] plugins: [devTools,translations,share,searchConnectors,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldsMetadata,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,entityManager,eventLog,actions,observabilityAIAssistant,investigate,notifications,cloudDataMigration,aiAssistantManagementSelection,advancedSettings,grokdebugger,console,searchNotebooks,searchHomepage,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,navigation,graph,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,alerting,logsDataAccess,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,lens,maps,dataVisualizer,dashboard,triggersActionsUi,transform,stackConnectors,searchPlayground,integrationAssistant,stackAlerts,ruleRegistry,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,metricsDataAccess,logsShared,aiops,discover,reporting,canvas,ml,searchInferenceEndpoints,elasticAssistant,logsExplorer,fleet,osquery,indexManagement,textBasedLanguages,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,observabilityAiAssistantManagement,datasetQuality,dataQuality,cloudSecurityPosture,cloudDefend,securitySolution,securitySolutionEss,observability,uptime,synthetics,slo,observabilityLogsExplorer,observabilityOnboarding,observabilityAIAssistantApp,discoverEnhanced,links,dashboardEnhanced,apmDataAccess,infra,upgradeAssistant,monitoring,logstash,apm,ux,assetsDataAccess]
2024-12-12 14:15:55 [2024-12-12T19:15:55.527+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 61ba1d06-8eac-4129-8786-2ce1945f1d9d
2024-12-12 14:15:56 [2024-12-12T19:15:56.141+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
2024-12-12 14:15:56 [2024-12-12T19:15:56.836+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Ubuntu 20.04 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
2024-12-12 14:15:57 [2024-12-12T19:15:57.129+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2024-12-12 14:15:57 [2024-12-12T19:15:57.129+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
2024-12-12 14:15:57 [2024-12-12T19:15:57.144+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2024-12-12 14:15:57 [2024-12-12T19:15:57.144+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
2024-12-12 14:15:57 [2024-12-12T19:15:57.342+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2024-12-12 14:15:57 [2024-12-12T19:15:57.514+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2024-12-12 14:15:57 [2024-12-12T19:15:57.743+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
2024-12-12 14:15:58 [2024-12-12T19:15:58.258+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2024-12-12 14:15:58 [2024-12-12T19:15:58.260+00:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
2024-12-12 14:16:01 [2024-12-12T19:16:01.462+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2024-12-12 14:16:03 [2024-12-12T19:16:03.051+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
2024-12-12 14:16:04 [2024-12-12T19:16:04.331+00:00][INFO ][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] and policy update batch size of [25]
2024-12-12 14:16:04 [2024-12-12T19:16:04.332+00:00][INFO ][plugins.securitySolution.endpoint:complete-external-response-actions] Registering task [endpoint:complete-external-response-actions] with timeout of [5m] and run interval of [60s]
2024-12-12 14:16:05 Kibana is not ready yet... waiting...
2024-12-12 14:16:06 [2024-12-12T19:16:06.204+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
2024-12-12 14:16:10 Kibana is not ready yet... waiting...
2024-12-12 14:16:11 [2024-12-12T19:16:11.696+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate

After exec into kibana and running the curl command this is the error message I recieve:

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401

Hmm okay regarding passing hostnames to the cert generation I have not created a yml file with hostname information included. How would I go about this in the cert generation?

I ran this openssl command to view the info about instance.crt:

openssl x509 -in /usr/share/elasticsearch/config/certs/node/instance/instance.crt -text -noout

And the output is showing Subject: CN = instance

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            25:22:38:e0:ef:d7:87:da:c3:e7:6c:87:f3:17:e6:ec:8b:c9:92:ce
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Elastic Certificate Tool Autogenerated CA
        Validity
            Not Before: Dec  5 16:54:27 2024 GMT
            Not After : Dec  5 16:54:27 2027 GMT
        Subject: CN = instance
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b2:aa:00:4f:b2:72:cb:af:1d:34:68:00:56:ae:
                    d9:47:2e:f4:8e:60:5c:5e:3b:59:c0:ce:72:95:f9:
                    85:70:42:87:15:44:db:21:58:81:4d:65:07:ae:26:
                    5b:2d:b0:83:79:4f:c8:9e:c4:d8:db:c1:2c:ea:c8:
                    b1:5c:f8:18:0c:8b:09:de:25:3b:90:dd:d7:56:d4:
                    da:2a:a4:1a:24:5d:60:f1:b4:77:b4:d7:ef:40:a4:
                    a0:83:f0:1f:65:f9:b4:bb:dc:3a:b2:2e:90:b2:93:
                    bc:06:27:2f:fd:6c:1a:0f:bb:f3:38:17:22:10:1c:
                    f0:d9:f2:f5:54:b5:ae:d9:7e:a1:87:19:92:10:f8:
                    6b:58:86:41:d6:7e:ee:50:4e:88:7c:25:b8:7a:40:
                    ff:72:07:5a:e9:81:39:05:ff:c7:f6:3f:39:33:f5:
                    49:85:2a:65:52:79:9e:ef:80:3b:fd:90:7b:ab:ce:
                    17:b6:46:ee:e8:53:56:0f:c3:4e:16:40:ca:ef:33:
                    d1:60:79:59:2c:fe:3d:b1:9c:33:38:25:61:0e:27:
                    e7:99:10:1b:82:83:c0:88:e0:dc:dd:64:dd:04:63:
                    69:de:f8:c7:30:43:33:70:2d:7b:eb:02:3c:51:92:
                    b7:e8:d2:6f:79:fc:ea:0f:87:cf:b2:f3:f1:7a:66:
                    28:eb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                4D:98:AC:93:C1:7A:4A:28:33:04:62:8B:A0:36:82:8E:D9:6E:FD:DA
            X509v3 Authority Key Identifier:
                keyid:47:31:5A:2A:03:CB:99:CF:A9:2E:DC:F7:D5:EA:19:AB:A8:23:D5:6A

            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         75:7f:7b:58:22:9b:d3:95:ff:78:9a:84:b6:35:db:39:7f:7f:
         65:14:86:c5:21:6a:7b:e3:49:ab:e2:c3:a3:2c:72:27:ab:ee:
         2d:af:15:72:47:de:39:52:dc:5d:9f:11:8d:f5:e3:b9:72:cf:
         73:6e:db:ed:d3:1a:51:c9:59:a0:9f:48:c4:38:f6:ad:50:ca:
         cb:4a:8e:6b:99:7b:ac:2a:5b:52:18:8f:26:0f:9a:47:8e:c4:
         ed:c0:b4:3a:9f:64:dc:5e:d4:88:6e:27:a8:f7:f2:36:ae:82:
         27:13:3a:3c:41:75:4d:42:b9:c5:7c:5d:51:78:ae:4f:7a:25:
         29:ae:ed:93:0b:e9:64:0b:e7:46:22:7f:38:18:df:72:08:74:
         9d:54:89:6a:ca:08:06:56:2e:c1:9e:62:90:6b:a2:ad:01:c3:
         bd:f8:75:d7:30:45:27:7d:81:d9:3c:28:64:85:96:a2:73:33:
         b8:4f:15:81:97:a2:74:1f:19:2c:4d:16:7a:a6:70:21:62:8f:
         5c:c9:6d:7d:fc:1b:6c:cb:a9:3f:ce:0e:c1:92:d2:17:bf:b4:
         80:55:19:e3:51:0b:aa:84:1e:19:a7:f4:90:87:39:70:0b:bf:
         c2:3f:ac:be:48:22:79:8c:3e:96:fc:dc:44:0a:2f:8a:13:98:
         71:da:07:0b

Also when I openssl into the ca.crt this is my output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            97:6f:a6:36:39:82:df:7f:b0:1c:4c:6a:bd:07:36:9b:9e:e4:b5:31
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Elastic Certificate Tool Autogenerated CA
        Validity
            Not Before: Dec  5 16:35:42 2024 GMT
            Not After : Dec  5 16:35:42 2027 GMT
        Subject: CN = Elastic Certificate Tool Autogenerated CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:9d:c3:c9:e4:02:38:27:8c:00:0c:76:22:52:6b:
                    75:a9:a6:1c:93:83:e9:51:2e:81:e9:7a:d1:5c:8c:
                    05:8b:03:4e:42:80:a8:2f:60:5f:de:87:d3:50:10:
                    87:11:4f:5f:bd:2d:e3:a2:1d:a1:c6:2e:06:a8:6a:
                    08:dd:85:19:ce:6b:9c:aa:1f:49:ba:3a:91:19:55:
                    37:cf:48:07:7d:37:a1:e9:7c:81:bc:92:fe:b0:28:
                    cf:2e:1a:f3:60:76:a1:67:7d:26:a9:6e:62:bd:fc:
                    3e:cf:0f:f8:76:bb:3c:0e:57:e1:1d:4e:98:d1:42:
                    b9:62:72:4a:eb:b8:94:d1:3e:aa:47:af:44:df:10:
                    18:29:fd:a5:bf:ad:31:af:b3:d4:2a:9c:24:d1:04:
                    de:0e:76:d7:31:42:05:a2:4c:0b:42:09:34:37:25:
                    01:09:fa:b1:33:83:33:a7:ab:c6:4a:1c:0e:c2:cf:
                    ff:a5:8c:37:61:58:95:0e:5d:d1:30:c3:49:98:79:
                    66:e2:37:34:b8:03:31:9c:6d:c6:d8:00:b3:27:74:
                    da:07:4b:e7:81:a1:56:8d:10:f9:b0:8a:42:2c:a1:
                    eb:4a:80:6c:4e:11:d5:0f:43:7c:d8:7e:a9:29:ec:
                    c9:b9:8a:5d:71:0e:d1:d6:dd:15:ab:8a:b4:e7:1c:
                    6b:05
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                C3:DB:97:21:55:F5:4B:E1:E6:39:EE:7C:11:84:FF:31:8C:75:AC:A7
            X509v3 Authority Key Identifier:
                keyid:C3:DB:97:21:55:F5:4B:E1:E6:39:EE:7C:11:84:FF:31:8C:75:AC:A7

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         3b:ec:6c:c2:59:c1:6f:f3:18:8a:ed:ea:92:5d:18:cc:0b:68:
         34:e8:d4:1b:52:3f:0f:f9:9f:4c:39:2e:2e:7c:b9:5f:1c:fc:
         05:0e:92:0b:35:24:36:9a:cd:a0:a4:c3:c5:67:98:db:40:f6:
         54:f8:b6:46:fe:58:cc:b3:65:74:fc:2b:34:63:63:0e:41:71:
         31:1c:18:98:8e:ab:ba:c6:49:ef:42:aa:bc:5e:46:ad:3c:1f:
         50:69:0c:1e:c8:35:e2:83:f8:6d:bd:8e:3f:f8:64:15:ea:fa:
         97:e9:38:6d:4d:8c:f7:80:0e:78:e7:f5:87:a0:97:d1:0d:cf:
         b6:b2:c1:fd:4f:2e:b3:04:a2:75:88:6b:65:46:fb:5c:da:f2:
         53:f5:79:fd:78:33:99:e6:0e:53:4b:fe:b4:2b:d3:9a:32:d0:
         2e:5a:af:87:46:dd:fb:97:55:a2:36:ef:4e:e3:06:d2:f7:38:
         fd:31:f1:e2:10:ae:6c:1e:85:cf:29:17:89:02:18:63:6f:a3:
         4d:10:26:83:91:5f:b1:e7:7b:d4:2e:4c:9a:6a:e5:68:ef:58:
         ad:93:a7:d6:e6:cb:13:1e:15:c9:68:b3:c2:e8:20:ee:7d:8f:
         90:f8:dc:3b:2d:a5:fa:0e:4b:34:8f:e3:8e:0f:3d:a1:3a:57:
         1b:86:97:6c

strawgate · December 12, 2024, 10:09pm

Can you provide the elastic username and password via CURL and make sure it can authenticate? Adding -u elastic:elasticpassword should be sufficient.

Looks like it's just set to instance. You could try disabling SSL verification on Kibana (or just disabling hostname verification) and see if it can startup, that would tell you that the issue is solely in the certificates.

Disabling SSL verification would be elasticsearch.ssl.verificationMode: none and disabling just the hostname verification (which would ignore that instance on the cert != elasticsearch in the hostname) would be elasticsearch.ssl.verificationMode: certificate.

For re-issuing the certs with the proper CN, you'd setup a yaml like described here elasticsearch-certutil | Elasticsearch Guide [8.17] | Elastic

strawgate · December 17, 2024, 11:58pm

Hey @Diler_Mohammed just wanted to check in -- were you able to get up and running?

Topic		Replies	Views
Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate Kibana elastic-stack-security , docker	4	100	September 18, 2024
Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.19.0.3:9200 Elasticsearch docker	7	10000	February 10, 2022
Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200 Kibana elastic-stack-security , docker	1	7420	June 22, 2022
Deploying ELK cluster on production server - generated and existing server certificates not working Kibana elastic-stack-security , docker	2	345	April 2, 2023
Unable to retrieve version information from Elasticsearch nodes. unable to get local issuer certificate Elasticsearch elastic-stack-security , docker	1	739	September 8, 2021

ELK security setup

Related topics