I am trying to reduce the TTL period to 10 mins and then after 8 mins, the certificate should reload but it does not happens, then it fails and restarts the Logstash, this process restart also fails and the complete pod restarts then runs. Why it is not able to reload, and then not able to restart logstash process?
Thanks in advance.
Hello,
Which certiicate?
Please share your logs, it is not possible to know what is happening without the logs.
It is the internal certificate.
2023-10-02T03:23:54.411913167+02:00 io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
2023-10-02T03:23:54.414756675+02:00 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.417504825+02:00 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.420340431+02:00 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.423383790+02:00 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.426008991+02:00 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.429181907+02:00 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.432767742+02:00 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.436461652+02:00 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.440534061+02:00 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.444140699+02:00 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.447798668+02:00 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.451207607+02:00 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.455150572+02:00 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.458290019+02:00 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.461293026+02:00 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.464633120+02:00 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.467576752+02:00 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.470472000+02:00 at java.lang.Thread.run(Thread.java:829) [?:?]
2023-10-02T03:23:54.473210104+02:00 Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
2023-10-02T03:23:54.475945917+02:00 at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
2023-10-02T03:23:54.479015120+02:00 at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
2023-10-02T03:23:54.482143176+02:00 at sun.security.ssl.TransportContext.fatal(TransportContext.java:347) ~[?:?]
2023-10-02T03:23:54.485006887+02:00 at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
2023-10-02T03:23:54.487733459+02:00 at sun.security.ssl.TransportContext.dispatch(TransportContext.java:186) ~[?:?]
2023-10-02T03:23:54.490434446+02:00 at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
2023-10-02T03:23:54.493404712+02:00 at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
2023-10-02T03:23:54.495967547+02:00 at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
2023-10-02T03:23:54.498900917+02:00 at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
2023-10-02T03:23:54.501594004+02:00 at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
2023-10-02T03:23:54.504536298+02:00 at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
2023-10-02T03:23:54.507572022+02:00 at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:298) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.510362065+02:00 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1338) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.513165468+02:00 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.516214898+02:00 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.518965928+02:00 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.521813272+02:00 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
2023-10-02T03:23:54.529303415+02:00 ... 17 more
It shows bad certificate error when tries to reload the new certificate which is being found.
Which internal certificate? The certitificate used in Elasticsearch output? The certificate used on an input? Is this the CA or the client certificate?
This error is pretty clear, there is something wrong with the certificate.
Whar certificate is this? It is still not clear as you can use a certificate in many places.
Can you share your logstash configuration?
Input client cert.
As mentioned before, there is something wrong in the certificate you need to double check it.
Also, you didn't share any configuration nor any log showing that Logstash is restarting, can you share the logs showing that logstash is restarting?
sorry, cant share the complete logs, but sharing some of the parts.
2023-10-02T03:24:59.574215917+02:00 {"version": "1.0.0", "timestamp": "2023-10-02T01:24:59.570+00:00", "severity": "info", "service_id": "eric-log-transformer", "metadata": {"proc_id": "supervisor.config_reload.sh", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "pod_uid": "624a9dbc-5581-4c56-9351-6e9d526f7f72", "container_name": "logtransformer", "node_name": "node-10-63-134-150", "namespace": "footprint"}, "message": "New certificate found /run/secrets/input-cert/srvcert.pem with validity: notBefore=Oct 2 01:23:09 2023 GMT; notAfter=Oct 2 01:33:39 2023 GMT"}
2023-10-02T03:25:18.307072430+02:00 {"version": "1.0.0", "timestamp": "2023-10-02T01:25:18.304+00:00", "severity": "error", "service_id": "eric-log-transformer", "metadata": {"proc_id": "supervisor.monitor_input_certs.sh", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "pod_uid": "624a9dbc-5581-4c56-9351-6e9d526f7f72", "container_name": "logtransformer", "node_name": "node-10-63-134-150", "namespace": "footprint"}, "message": "Certificate reload failure detected, Logstash input pipeline is using an expired certificate since at least 301 seconds ago. Requesting supervisor to restart Logstash."}
2023-10-02T03:25:18.722045532+02:00 {"version": "1.0.0", "timestamp": "2023-10-02T01:25:18.717+00:00", "severity": "error", "service_id": "eric-log-transformer", "metadata": {"proc_id": "supervisor.init.sh", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "pod_uid": "624a9dbc-5581-4c56-9351-6e9d526f7f72", "container_name": "logtransformer", "node_name": "node-10-63-134-150", "namespace": "footprint"}, "message": "Restarting Logstash as instructed by certificate monitor."}
2023-10-02T03:25:18.733909392+02:00 {"version": "1.1.0", "timestamp": "2023-10-02T01:25:18.731Z", "severity": "warning", "service_id": "eric-log-transformer", "metadata" : {"namespace": "footprint", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "container_name": "logtransformer"}, "message": "[logstash.runner] SIGTERM received. Shutting down."}
2023-10-02T03:25:37.626742018+02:00 {"version": "1.1.0", "timestamp": "2023-10-02T01:25:37.618Z", "severity": "error", "service_id": "eric-log-transformer", "metadata" : {"namespace": "footprint", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "container_name": "logtransformer"}, "message": "[logstash.inputs.tcp] null: closing due:"}
2023-10-02T03:25:37.638001518+02:00 io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
2023-10-02T03:24:11.778609895+02:00 {"version": "1.1.0", "timestamp": "2023-10-02T01:24:11.769Z", "severity": "info", "service_id": "eric-log-transformer", "metadata" : {"namespace": "footprint", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "container_name": "logtransformer"}, "message": "[org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 192.168.43.30:37894] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate (caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate)"}
2023-10-02T03:27:19.825490172+02:00 {"version": "1.0.0", "timestamp": "2023-10-02T01:27:19.821+00:00", "severity": "critical", "service_id": "eric-log-transformer", "metadata": {"proc_id": "supervisor.init.sh", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "pod_uid": "624a9dbc-5581-4c56-9351-6e9d526f7f72", "container_name": "logtransformer", "node_name": "node-10-63-134-150", "namespace": "footprint"}, "message": "Logstash has not shut down since termination was requested 121 seconds ago. Terminating Logstash process with SIGKILL and exiting."}
2023-10-02T03:27:19.839602445+02:00 {"version": "1.0.0", "timestamp": "2023-10-02T01:27:19.831+00:00", "severity": "error", "service_id": "eric-log-transformer", "metadata": {"proc_id": "supervisor.init.sh", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "pod_uid": "624a9dbc-5581-4c56-9351-6e9d526f7f72", "container_name": "logtransformer", "node_name": "node-10-63-134-150", "namespace": "footprint"}, "message": "Terminating children of logstash wrapper process (pid 66) first using SIGKILL."}
2023-10-02T03:27:19.854817406+02:00 {"version": "1.0.0", "timestamp": "2023-10-02T01:27:19.850+00:00", "severity": "info", "service_id": "eric-log-transformer", "metadata": {"proc_id": "supervisor.init.sh", "pod_name": "eric-log-transformer-7d4744b89-9h9sd", "pod_uid": "624a9dbc-5581-4c56-9351-6e9d526f7f72", "container_name": "logtransformer", "node_name": "node-10-63-134-150", "namespace": "footprint"}, "message": "Logstash was terminated and returned exit code 137."}
Apart from this, when I increase the TTL period, it works fine at that time.
These logs says that your certificate is expired, it seems that there is some issue with the certificate, as mentione before you need to check and fix it.
Not sure what is this TTL configuration you mention, can you provide the configuration and where you are changing it?
Thanks buddy, there was an issue with the certificate providing service, it was sending tampered certificate.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.