Hello,
I'm new to ELK and I recently upgraded the 6.0.0 ELK stack to 6.6.1. Everything went pretty okay but for some reason all the certificates stopped working. I'm using docker containers and this all worked before. I'm trying to figure out where to begin and start to unravel the issues.
Filebeat 6.6.1 is running on web app servers
This sends logs to Logstash 6.6.1 container on a separate VM
Then I have elasticsearch 6.6.1 container on another vm
with Kibana 6.6.1 container on yet another vm.
I have tried following the documentation but since I never set this up I'm sort of lost on where to begin.
The logs are not getting consumed by logstash and logstash isn’t talking to elasticsearch.
Logstash complains about the certificate and the certificate hasn’t changed.
Errors with filebeat
2019-02-20T22:49:15.211117667Z 2019-02-20T22:49:15.210Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":220,"time":{"ms":11}},"total":{"ticks":410,"time":{"ms":14},"value":410},"user":{"ticks":190,"time":{"ms":3}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"7f1cc96a-6d9f-4662-a955-4b81486b95a2","uptime":{"ms":60029}},"memstats":{"gc_next":17835520,"memory_alloc":11226752,"memory_total":25692880}},"filebeat":{"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"errors":1},"write":{"bytes":159}},"pipeline":{"clients":4,"events":{"active":4119,"retry":2048}}},"registrar":{"states":{"current":4}},"system":{"load":{"1":0.34,"15":1.27,"5":0.6,"norm":{"1":0.0213,"15":0.0794,"5":0.0375}}}}}}
2019-02-20T22:49:45.209583732Z 2019-02-20T22:49:45.209Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":230,"time":{"ms":5}},"total":{"ticks":430,"time":{"ms":8},"value":430},"user":{"ticks":200,"time":{"ms":3}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"7f1cc96a-6d9f-4662-a955-4b81486b95a2","uptime":{"ms":90028}},"memstats":{"gc_next":17835520,"memory_alloc":11527936,"memory_total":25994064}},"filebeat":{"events":{"active":1,"added":1},"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":4,"events":{"active":4120,"total":1}}},"registrar":{"states":{"current":4}},"system":{"load":{"1":0.34,"15":1.24,"5":0.57,"norm":{"1":0.0213,"15":0.0775,"5":0.0356}}}}}}
2019-02-20T22:49:56.904999862Z 2019-02-20T22:49:56.904Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash:443)): read tcp 172.18.0.2:43368->10.10.0.22:443: read: connection reset by peer
2019-02-20T22:49:56.905041162Z 2019-02-20T22:49:56.904Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://logstash:443)) with 6 reconnect attempt(s)
2019-02-20T22:50:15.210925904Z 2019-02-20T22:50:15.210Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":240,"time":{"ms":9}},"total":{"ticks":440,"time":{"ms":12},"value":440},"user":{"ticks":200,"time":{"ms":3}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"7f1cc96a-6d9f-4662-a955-4b81486b95a2","uptime":{"ms":120025}},"memstats":{"gc_next":17835520,"memory_alloc":11847104,"memory_total":26313232}},"filebeat":{"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"errors":1},"write":{"bytes":159}},"pipeline":{"clients":4,"events":{"active":4120,"retry":2048}}},"registrar":{"states":{"current":4}},"system":{"load":{"1":0.29,"15":1.2,"5":0.54,"norm":{"1":0.0181,"15":0.075,"5":0.0338}}}}}}
Errors with logstash
2019-02-20T22:49:56.913114932Z [2019-02-20T22:49:56,912][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
2019-02-20T22:49:56.913162932Z java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/logstash/config/certs/service.key
…
2019-02-20T22:49:56.913260931Z Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
2019-02-20T22:49:56.913264631Z at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1046) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
2019-02-20T22:49:56.913268531Z at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
2019-02-20T22:49:56.913272330Z at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
2019-02-20T22:49:56.913276130Z ... 20 more
2019-02-20T22:49:56.913279730Z Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
Any help would be greatly appreciated.