I am setting up docker container for elasticsearch and kibana : (customised from docker compose) im doing setup container steps manually
image used : docker.elastic.co/elasticsearch/elasticsearch:8.11.1
========================================================================================================================
step 1:
i am starting an init container and generating ssl certificates and ca certs .manually removing the container provide correct privilages to mounted folder certs. and starting a new es01 container .
source ./.env
mkdir -p ~/certs
docker pull docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
docker run -it --rm \
-m 3GB \
--group-add 0 \
-u 0 \
-v /home/elastic/certs:/usr/share/elasticsearch/config/certs \
-e ELASTIC_PASSWORD=${ELASTIC_PASSWORD} \
-e KIBANA_PASSWORD=${KIBANA_PASSWORD} \
docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} \
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable";
exit 1;
fi;
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es01\n"\
" dns:\n"\
" - es01\n"\
" - localhost\n"\
" - elasticsearch.tagitmobile.com\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - 172.17.0.198\n"\
" - 172.19.0.2\n"\
" - name: kibana\n"\
" dns:\n"\
" - kibana\n"\
" - localhost\n"\
" - 172.17.0.000\n"\
" - 172.19.0.2\n"\
" - kibana.myown.com\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: fleet\n"\
" dns:\n"\
" - fleet-server\n"\
" - localhost\n"\
" - 172.17.0.000\n"\
" - 172.19.0.2\n"\
" - fleet-server.myown.com\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: apm\n"\
" dns:\n"\
" - 172.17.0.000\n"\
" - 172.19.0.2\n"\
" - apm-server\n"\
" - localhost\n"\
" - apm-server.myown.com\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R 1003:1003 config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://elasticsearch.myown.com:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
=======================================================================================================
step 2:
below is my docker command for Elasticsearch .
source ./.env
docker run -d --name es01 \
-m 3GB \
--network=elastic \
-p ${ES_PORT}:9200 \
-v /home/elastic/certs:/usr/share/elasticsearch/config/certs \
-v /home/elastic/esdata01:/usr/share/elasticsearch/data \
-e node.name=es01 \
-e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
-e cluster.name=${CLUSTER_NAME} \
-e discovery.type=single-node \
-e ELASTIC_PASSWORD=${ELASTIC_PASSWORD} \
-e bootstrap.memory_lock=true \
-e xpack.security.enabled=true \
-e xpack.security.http.ssl.enabled=true \
-e xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es01/es01.key \
-e xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es01/es01.crt \
-e xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt \
-e xpack.security.transport.ssl.enabled=true \
-e xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es01/es01.key \
-e xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es01/es01.crt \
-e xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt \
-e xpack.security.transport.ssl.verification_mode=certificate \
-e xpack.license.self_generated.type=${LICENSE} \
--ulimit memlock=-1:-1 \
--health-cmd="curl -s --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'" \
--health-interval=10s \
--health-timeout=10s \
--health-retries=120 \
docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
==========================================================================================
STEP 3 error log:
"
{"@timestamp":"2023-11-20T14:26:58.173Z", "log.level": "WARN", "message":"caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.19.0.2:9200, remoteAddress=/172.17.0.000:39194}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#4]","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"yuyhFsNfQoyK1djrxH6jHw","elasticsearch.node.id":"mI1W-pz0QDWofp2U0SxtUg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","error.type":"io.netty.handler.codec.DecoderException","error.message":"javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate","error.stack_trace":"io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate\n\tat java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)\n\tat java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)\n\tat java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365)\n\tat java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287)\n\tat java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)\n\tat java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)\n\tat java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)\n\tat java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)\n\tat java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)\n\tat java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)\n\tat java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)\n\tat io.netty.handler@4.1.94.Final/io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297)\n\tat io.netty.handler@4.1.94.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353)\n\tat io.netty.handler@4.1.94.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)\n\tat io.netty.handler@4.1.94.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)\n\t... 16 more\n"}
"
STEP 5 :
i have verified the certificate authority and can see my hostnames in ca.crt file
under --> X509v3 Subject Alternative Name: .
STEP 6:
On host and contianer i have added the certificate authority ca.crt to etc/pki/ca-trust/source/anchors repsective location in container also .
i have provided pub/private key with proper privilages i have also provided xpack.security.transport.ssl.verification_mode=certificate
why am i gettting this error in logs i can see --->Netty4HttpChannel{localAddress=/172.19.0.2:9200, remoteAddress=/172.17.0.000:39194}
i have added these IP also in ca under SAN stilll error is not getting resolved can u pls advice .
i am able to proceed and setup elastic/kibana / and fleet . but some elatic agents are not streaming data while some hosts stream data . the attached error log is in es01 container