Hello,
I have two elasticsearch clusters: a production one and a monitoring cluster which ideally should also host the audit indices. Both clusters are 5.4.0 (x-pack as well)
I have enabled transport ssl in the monitoring cluster:
xpack:
security:
transport:
ssl:
enabled: true
verification_mode: none
key: /etc/elasticsearch/x-pack/ssl/.key
certificate: /etc/elasticsearch/x-pack/ssl/.crt
certificate_authorities: /etc/elasticsearch/x-pack/ssl/ca.crt
client_authentication: none
I have then verified that the cluster is running as intended.
Then in my production cluster I enabled audit.index.client like this:
xpack:
security:
audit:
enabled: true
outputs: [ index, logfile ]
logfile:
events:
include: access_granted, anonymous_access_denied, authentication_failed, realm_authentication_failed, access_denied, tampered_request, connection_denied, run_as_denied
exclude: connection_granted, run_as_granted
prefix:
emit_node_name: true
emit_node_host_address: true
index:
client: # xpack.security.audit.index.client settings used to store audit index to seperate cluster
hosts: 'hostname:9300'
cluster:
name: "elastic_monitoring_cluster" # The name of the remote cluster
xpack:
security:
user: :
rollover: weekly
flush_interval: 2s
bulk_size: 4096
events:
emit_request_body: true
settings:
index:
number_of_shards: 2
number_of_replicas: 1
This the above configuration of the clusters, my production cluster fails to send audit indices to remote cluster here is the trace log:
[23][internal:tcp/handshake] sent to [{#transport#-1}{KRIiEIluSPGibVxwH9salA}{}{:] (timeout: [null])
[2017-05-09T20:58:47,041][DEBUG][o.e.c.t.TransportClientNodesService] [security-audit-client-_node] failed to connect to node [{#transport#-1}{KRIiEIluSPGibVxwH9salA}{}{:], ignoring...
org.elasticsearch.transport.ConnectTransportException: [:] general node connection failure
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:550) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:115) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:334) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:408) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:358) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:391) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.IllegalStateException: handshake failed
at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1599) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:533) ~[elasticsearch-5.4.0.jar:5.4.0]
... 9 more
Caused by: org.elasticsearch.transport.TransportException: connection reset
at org.elasticsearch.transport.TcpTransport.onChannelClosed(TcpTransport.java:1641) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.netty4.Netty4Transport.access$100(Netty4Transport.java:94) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:401) ~[?:?]
at org.elasticsearch.transport.netty4.Netty4Transport$ChannelCloseListener.operationComplete(Netty4Transport.java:391) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) ~[?:?]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:686) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:664) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:85) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:142) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:524) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:478) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Other than that both clusters are functioning properly. If i will run an openssl s_client -connect to the monitoring fqdn:port then it connects properly without warnings.
If I will disable transport ssl in the monitoring cluster (by setting xpack.security.transport.ssl.enabled: false), then the prod cluster is able to send the audit indices to the remote monitoring cluster without issues.
It seams that there is a handshake issues between the 2 clusters or a bug maybe ? (certs and ca are proper http protocol is working fine via ssl (by using monitoring.exporters)
Any help will be appreciated
Regrads,
Lefteris Agrianitis