Hi,
*I am using ELK 7.9
I am trying use Winlogbeat to ship data to an Elasticsearch node, which has secure connection settings enabled:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
... and also API authentication
xpack.security.authc.api_key.enabled: true
Monitoring is also enabled in the node.
Since the beat will be running externally to the node, I am trying to expose the less sensitive data possible. That is why I followed the steps from official documentation to create the required roles, users and API keys, so I can use API key authentication in the beat config, rather than expose username and password.
If I got it right from the official documentation, you can use either one of the following authentication methods. So I'm trying the API key:
- Basic authentication credentials (username and password).
- Token-based (API key) authentication.
- Public Key Infrastructure (PKI) certificates.
After few tests I keep getting errors related to the certificate verification.
Currently Kibana and Logstash can connect to Elasticsearch without issues, with the security connection settings enabled.
Perhaps I am missing or mixing something, but can't find what it is. I tried my bets with the official documentation.
Please help me.
Thank you
winlogbeat.yml (only Elasticsearch output settings)
output.elasticsearch:
hosts: ["https://ip_address:9200"]
api_key: "dftyiuyiYliYFLIYLt:lkyrxYRertckLKYCRRdhtjku"
setup.ilm.check_exists: false
*Also tried adding the SSL settings,
ssl.enabled: true
ssl.certificate_authorities: ["C:\Program Files\Winlogbeat\certs\elasticsearch-ca.pem"]
ssl.verification_mode: certificate
Winlogbeat log output
2020-08-28T13:18:05.722-0400 INFO instance/beat.go:299 Setup Beat: winlogbeat; Version: 7.9.0
2020-08-28T13:18:05.722-0400 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'winlogbeat-7.9.0' as ILM is enabled.
2020-08-28T13:18:05.722-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://ELK_node_IP:9200
2020-08-28T13:18:05.722-0400 INFO [publisher] pipeline/module.go:113 Beat name: WIN-FOLUGKDG0C3
2020-08-28T13:18:05.722-0400 INFO beater/winlogbeat.go:69 State will be read from and persisted to C:\ProgramData\winlogbeat\.winlogbeat.yml
2020-08-28T13:18:05.754-0400 WARN [cfgwarn] registered_domain/registered_domain.go:60 BETA: The registered_domain processor is beta.
2020-08-28T13:18:05.798-0400 WARN [cfgwarn] registered_domain/registered_domain.go:60 BETA: The registered_domain processor is beta.
2020-08-28T13:18:05.818-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://ELK_node_IP:9200
2020-08-28T13:18:05.818-0400 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-08-28T13:18:05.818-0400 INFO instance/beat.go:450 winlogbeat start running.
2020-08-28T13:18:05.839-0400 WARN beater/eventlogger.go:124 EventLog[Microsoft-Windows-Sysmon/Operational] Open() error. No events will be read from this source. The specified channel could not be found. Check channel configuration.
2020-08-28T13:18:05.865-0400 INFO [monitoring] elasticsearch/elasticsearch.go:245 Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying. Error: cannot connect underlying Elasticsearch client: Get "https://ELK_node_IP:9200": x509: certificate signed by unknown authority
2020-08-28T13:18:08.719-0400 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-08-28T13:18:08.728-0400 INFO beater/eventlogger.go:88 EventLog[Security] successfully published 1 events
2020-08-28T13:18:08.728-0400 INFO beater/eventlogger.go:88 EventLog[Security] successfully published 1 events
2020-08-28T13:18:08.728-0400 INFO beater/eventlogger.go:88 EventLog[Security] successfully published 1 events
2020-08-28T13:18:09.721-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-08-28T13:18:09.721-0400 INFO [publisher] pipeline/retry.go:223 done
2020-08-28T13:18:09.721-0400 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://ELK_node_IP:9200))
2020-08-28T13:18:11.599-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://ELK_node_IP:9200)): Get "https://ELK_node_IP:9200": x509: certificate signed by unknown authority
2020-08-28T13:18:11.599-0400 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://ELK_node_IP:9200)) with 1 reconnect attempt(s)
2020-08-28T13:18:11.599-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-08-28T13:18:11.599-0400 INFO [publisher] pipeline/retry.go:223 done
2020-08-28T13:18:13.796-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://ELK_node_IP:9200)): Get "https://ELK_node_IP:9200": x509: certificate signed by unknown authority
2020-08-28T13:18:13.796-0400 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://ELK_node_IP:9200)) with 2 reconnect attempt(s)
2020-08-28T13:18:13.796-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-08-28T13:18:13.796-0400 INFO [publisher] pipeline/retry.go:223 done
2020-08-28T13:18:20.197-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://ELK_node_IP:9200)): Get "https://ELK_node_IP:9200": x509: certificate signed by unknown authority
2020-08-28T13:18:20.197-0400 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://ELK_node_IP:9200)) with 3 reconnect attempt(s)
2020-08-28T13:18:20.197-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-08-28T13:18:20.197-0400 INFO [publisher] pipeline/retry.go:223 done
2020-08-28T13:18:35.822-0400 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":359,"time":{"ms":359}},"total":{"ticks":4671,"time":{"ms":4671},"value":4671},"user":{"ticks":4312,"time":{"ms":4312}}},"handles":{"open":337},"info":{"ephemeral_id":"90212e25-0337-4407-8258-45c16b71f038","uptime":{"ms":30226}},"memstats":{"gc_next":73309840,"memory_alloc":59462024,"memory_total":278371456,"rss":99377152},"runtime":{"goroutines":40}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":6,"events":{"active":4117,"filtered":2212,"published":4116,"retry":100,"total":6329}}},"msg_file_cache":{"ApplicationHits":59,"ApplicationMisses":5,"ApplicationSize":5,"SecurityHits":6099,"SecurityMisses":1,"SecuritySize":1,"SystemHits":240,"SystemMisses":8,"SystemSize":8},"published_events":{"Security":3,"total":3},"system":{"cpu":{"cores":4}}}}}
2020-08-28T13:18:35.852-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://ELK_node_IP:9200)): Get "https://ELK_node_IP:9200": x509: certificate signed by unknown authority
2020-08-28T13:18:35.852-0400 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://ELK_node_IP:9200)) with 4 reconnect attempt(s)
2020-08-28T13:18:35.852-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-08-28T13:18:35.852-0400 INFO [publisher] pipeline/retry.go:223 done