Turning Monitor with Metricbeat on (Metricbeat -> ElasticSearch)

Elastic Stack Beats
1

Hi all,

I'm trying to turning my "Monitor with Metricbeat" on but i'm getting the following errors:

- From Metricbeat log: 

Apr 24 12:06:40 l-qa-bes48 metricbeat[47280]: 2020-04-24T12:06:40.332+0100        INFO        module/wrapper.go:252        Error fetching data for metricset elasticsearch.node_stats: error making http request: Get https://192.168.28.48:9200/_nodes/_local/stats: x509: certificate signed by unknown authority
Apr 24 12:06:43 l-qa-bes48 metricbeat[47280]: 2020-04-24T12:06:43.235+0100        INFO        module/wrapper.go:252        Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get https://192.168.28.48:9200/_nodes/_local/nodes: x509: certificate signed by unknown authority

-From ElasticSearch log:

[2020-04-24T12:06:41,217][WARN ][o.e.h.AbstractHttpServerTransport] [elasticsearch-Node1-PP] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.28.48:9200, remoteAddress=/192.168.28.48:56848}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:473) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:291) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:184) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:729) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:684) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:499) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:475) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        ... 16 more

I have a 3 node cluster of elasticsearch having kibana installed on the same server of Node 1.

My metricbeat.yml file:

	output.elasticsearch:
	  # Array of hosts to connect to.
	  hosts: ["https://192.168.28.48:9200"]
	  # Protocol - either `http` (default) or `https`.
	  protocol: "https"
	  # Authentication credentials - either API key or username/password.
	  username: "XXXXX"
	  password: "XXXXXXXXXXXXXXXXXXX"
	  ssl.certificate_authorities: ["/etc/metricbeat/elastic-stack-ca.pem"]

My elasticsearch-xpack.yml:

	- module: elasticsearch
	  metricsets:
		- ccr
		- cluster_stats
		- enrich
		- index
		- index_recovery
		- index_summary
		- ml_job
		- node_stats
		- shard
	  period: 10s
	  hosts: ["https://192.168.28.48:9200"]
	  username: "XXXXXXXXXXXXXX"
	  password: "XXXXXXXXXXXXXXXXXXX"
	  xpack.enabled: true

My elastic elasticsearch.yml (NODE 1 of 3):

	cluster.name: elasticsearch76-pp
	node.name: elasticsearch-Node1-PP
	http.port: 9200
	discovery.seed_hosts: ["elasticsearch-Node1-PP", "elasticsearch-Node2-PP", "elasticsearch-Node3-PP"]
	cluster.initial_master_nodes: ["elasticsearch-Node1-PP", "elasticsearch-Node2-PP", "elasticsearch-Node3-PP"]
	xpack.security.enabled: true
	xpack.security.transport.ssl.enabled: true
	xpack.security.transport.ssl.verification_mode: certificate
	xpack.security.transport.ssl.keystore.path: elastic-certificates-Node1.p12
	xpack.security.transport.ssl.truststore.path: elastic-certificates-Node1.p12
	xpack.security.http.ssl.enabled: true
	xpack.security.http.ssl.keystore.path: "http.p12"

Can someone help me with this?
Thanks

1 Like
2

Hi,

What worked for me was to add the ssl options to the module file of the metricbeat. It seems that the /etc/metricbeat/metricbeat.yml is overriden by the options defined under /etc/metricbeat/modules.d/

So, try to make sure that the file:
/etc/metricbeat/modules.d/elasticsearch-xpack.yml

has the following content (as per your confs):

username: "xxxx"
password: "xxxx"
ssl.enabled: true
ssl.certificate_authorities: ["/etc/metricbeat/elastic-stack-ca.pem"]

Good luck!

4 Likes
3

Thanks a lot @pup_seba , it's working now!!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.