Xpack.audit.index.client.host over ssl fails with a handshake error

Hello,

I have two elasticsearch clusters: a production one and a monitoring cluster which ideally should also host the audit indices. Both clusters are 5.4.0 (x-pack as well)

I have enabled transport ssl in the monitoring cluster:
xpack:
security:
transport:
ssl:
enabled: true
verification_mode: none
key: /etc/elasticsearch/x-pack/ssl/.key
certificate: /etc/elasticsearch/x-pack/ssl/.crt
certificate_authorities: /etc/elasticsearch/x-pack/ssl/ca.crt
client_authentication: none
I have then verified that the cluster is running as intended.
Then in my production cluster I enabled audit.index.client like this:
xpack:
security:
audit:
enabled: true
outputs: [ index, logfile ]
logfile:
events:
include: access_granted, anonymous_access_denied, authentication_failed, realm_authentication_failed, access_denied, tampered_request, connection_denied, run_as_denied
exclude: connection_granted, run_as_granted
prefix:
emit_node_name: true
emit_node_host_address: true
index:
client: # xpack.security.audit.index.client settings used to store audit index to seperate cluster
hosts: 'hostname:9300'
cluster:
name: "elastic_monitoring_cluster" # The name of the remote cluster
xpack:
security:
user: :
rollover: weekly
flush_interval: 2s
bulk_size: 4096
events:
emit_request_body: true
settings:
index:
number_of_shards: 2
number_of_replicas: 1

This the above configuration of the clusters, my production cluster fails to send audit indices to remote cluster here is the trace log:

[23][internal:tcp/handshake] sent to [{#transport#-1}{KRIiEIluSPGibVxwH9salA}{}{:] (timeout: [null])
[2017-05-09T20:58:47,041][DEBUG][o.e.c.t.TransportClientNodesService] [security-audit-client-_node] failed to connect to node [{#transport#-1}{KRIiEIluSPGibVxwH9salA}{}{:], ignoring...
org.elasticsearch.transport.ConnectTransportException: [:] general node connection failure
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:550) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:115) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:334) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:408) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:358) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:391) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.IllegalStateException: handshake failed
at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1599) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:533) ~[elasticsearch-5.4.0.jar:5.4.0]
... 9 more
Caused by: org.elasticsearch.transport.TransportException: connection reset
at org.elasticsearch.transport.TcpTransport.onChannelClosed(TcpTransport.java:1641) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.netty4.Netty4Transport.access$100(Netty4Transport.java:94) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:401) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:391) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) ~[?:?]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:686) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:664) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:85) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:142) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:524) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:478) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more

Other than that both clusters are functioning properly. If i will run an openssl s_client -connect to the monitoring fqdn:port then it connects properly without warnings.

If I will disable transport ssl in the monitoring cluster (by setting xpack.security.transport.ssl.enabled: false), then the prod cluster is able to send the audit indices to the remote monitoring cluster without issues.

It seams that there is a handshake issues between the 2 clusters or a bug maybe ? (certs and ca are proper http protocol is working fine via ssl (by using monitoring.exporters)

Any help will be appreciated

Regrads,

Lefteris Agrianitis

Hi @lagrianitis,

Sorry for there being a such a delay in getting a response to you. The client for auditing needs to be configured to use SSL. So you will need to put the xpack.security.transport.ssl settings under the audit client settings. You're basically configuring an independent transport client when you do remote auditing.

Jay

I can confirm that this was the issue. Thanks for the reply

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.