X-pack installation to ELK stack

Elastic Stack Elasticsearch
1

Hi All ,

i have a working ELK stack with Filebeat and kafka .

I want to secure it with X-pack .
I have gone through the documents but i am facing issue while implementing it .

Can someone help on this ?

i can share details if required .

2

If you have a commercial subscription with us, then your support engineer can help.

Otherwise, we're able to support you here, but you will need to provide much more information.

  • What version are you running?
  • What steps have you taken?
  • What issue did you run into?
  • What have you tried so far?
1 Like
3

I dont have commercial subsciption , i am testing with trial version .

Below are the details ,

  • i am testing with 6.1.1 versions of Elastic search and X-pack .

  • I have installed x-pack on all 3 nodes of elastic search cluster . also generated node certificate for all 3 nodes .

  • Update elasticsearch.yml with below ,

###################################################### X-Pack ###############################################
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml,*
xpack.ssl.key: /etc/elasticsearch/certs/node01/node01.key
xpack.ssl.certificate: /etc/elasticsearch/certs/node01/node01.crt
xpack.ssl.certificate_authorities: /etc/elasticsearch/certs/node01/node01.crt
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

on all 3 nodes .

I have generated certificates on 1 node for all 3 nodes with certgen cmd .

  • Now after generating certificates and copying them to each node and updating yml . when i am restarting elastic cluster , it is throwing below exceptions ,

exceptions are related to certificates ,

[2018-01-04T06:22:13,397][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [node02] send message failed [channel: org.
elasticsearch.transport.netty4.NettyTcpChannel@de4289b]
javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_151]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:255) ~[netty-handler-4.1.13.Final.jar:4.1.13.Fin
al]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1162) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-
4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.13.Final.jar:4.1
.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.
1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transpo
rt-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transpo
rt-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport
-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.
13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transpo
rt-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transpo
rt-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.
jar:4.1.13.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13
.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Fi
nal]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1
.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.F
inal]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final
.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2018-01-04T06:22:13,399][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [node02] exception caught on transport laye
r [org.elasticsearch.transport.netty4.NettyTcpChannel@de4289b], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate

4

exception continued ,

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.13.Final.jar:4.
1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.13.Final.jar:4
.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transpo
rt-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transpo
rt-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport
-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_151]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:255) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1162) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

	[2018-01-04T06:22:13,630][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [node02] client did not trust this server's certificate, closing connection org.elasticsearch.transport.netty4.NettyTcpChannel@5f423fb7

[2018-01-04T06:22:13,829][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node02] caught exception while handling client http traffic, closing connection [id: 0xc7f5d548, L:0.0.0.0/0.0.0.0:9200 ! R:/10.81.101.226:51106]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a486f73743a207669656c6b30326473792e64736f6e652e3364732e636f6d3a393230300d0a557365722d4167656e743a204d616e7469636f726520302e362e310d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]

*Looks like i havent generated certs properly . Please help in resolving the same .

Let me know if any other information is required .

5

Recently i have updated shard allocation settings as well ,
below are the details ,

########################################### Shard allocation #############################################

cluster.routing.allocation.enable: all
#cluster.routing.allocation.node_concurrent_incoming_recoveries: 2
#cluster.routing.allocation.node_concurrent_outgoing_recoveries: 2
cluster.routing.allocation.node_concurrent_recoveries: 2
cluster.routing.allocation.node_initial_primaries_recoveries: 4
cluster.routing.allocation.same_shard.host: false

################################ Shard rebalancing #######################

cluster.routing.rebalance.enable: all
cluster.routing.allocation.allow_rebalance: always
cluster.routing.allocation.cluster_concurrent_rebalance:2

############################# shard balancing Heuristcs #################

cluster.routing.allocation.balance.shard: 0.45f
cluster.routing.allocation.balance.index: 0.55f
cluster.routing.allocation.balance.threshold: 1.0f

######################## Disk Based Shard Allocation ##########################

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
cluster.routing.allocation.disk.watermark.flood_stage: 95%
cluster.info.update.interval: 50s
cluster.routing.allocation.disk.include_relocations: true

but still not working .

elasticsearch[21489]: [2018-01-04T13:10:02,414][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [Node01] client did not trust this server's certificate, closing connection org.elasticsearch.transport.netty4.NettyTcpChannel@121c68c4

please help on the same

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.