Winlogbeat getting x509: certificate signed by unknown authority when sending to elasticsearch

[ciphee (Daniel)]

Hello,

I have recently sucessfully connected my winlogbeat to logstash, however now I am trying to connect it to elastic in order to do the --setup dashboards and --setup index. I was trying to disable logstash and connect it to elasticsearch directly but I am getting the error ERROR x509: certificate signed by unknown authority when I try and test my config. I am testing this on a development server that has had certificates and xpack settings enabled.

C:\winlogbeats> .\winlogbeat.exe -c .\winlogbeat.yml test output
elasticsearch: https://x.251.10.68:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: x.251.10.68
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... ERROR x509: certificate signed by unknown authority

My winlogbeat.yml file contains:
winlogbeat.yml

winlogbeat.event_logs:

• name: Application
  ignore_older: 72h
• name: Security
• name: System

setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

setup.kibana:

host: "https://128.251.10.84:5601"
ssl.verification_mode: none

output.elasticsearch:
hosts: ["https://x.251.10.68:9200", "https://x.251.10.59:9200", "https://x.251.10.84:9200"]
protocol: "https"
protocol: "https"
username: "elastic"
password: "mypassword"

processors:

• add_host_metadata: ~
• add_cloud_metadata: ~

logging.level: info

Original Logstash Pipeline:

input {
beats {
port => 5959
}
}

output {
elasticsearch {
hosts => ["https://x.251.10.68:9200", "https://x.251.10.84:9200", "https://x.251.10.59:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
ssl => true
user => 'elastic'
password => 'mypassword'
document_type => "%{[@metadata][type]}"
}
}

Do I need to import a certificate from kibana in order to make a connection to elasticsearch? ANy thoughts to get winlogbeat to coperate with elasticsearch so I can setup dashboards and indexes?

[TimV (Tim Vernum)]

This is a result of how your Elasticsearch http certificate has been generated/signed.
You didn't provide your elasticsearch.yml, so we can't tell what certificate you're using there, and how it was configured.

In elasticsearch, you will have settings for one or more of:

xpack.security.http.ssl.keystore.path
xpack.security.http.ssl.certificate
xpack.security.http.ssl.key
xpack.ssl.keystore.path
xpack.ssl.certificate
xpack.ssl.key

Those control which certificate your ES node presents on the http port (9200). They will have been issued by a certificate authority. If you created them using the elasticsearch-certutil tool, then you will probably have your own certificate authority, and you will need to export it into a PEM format that winlogbeat can read, and configure it in output.elasticsearch.ssl.certificate_authorities

Alternatively, if you got the certificate from an existing Certificate Authority (a commercial CA, or your own corporate CA) then you will need to get a copy of the CA in PEM format and use it for output.elasticsearch.ssl.certificate_authorities.

As a temporary workaround, you could set output.elasticsearch.ssl.verification_mode: none but this is a dangerous setting, and will disable many of the intended benefits of using SSL. We strongly advise against configuring that on a production server.

As a separate issue:

username: "elastic"

We don't recommend using the elastic user for data ingest. That user has full superuser privileges and can do everything on your cluster (delete data, change cluster settings, change user passwords, etc). If someone gets access to the password (which is stored in plaintext in your beats config) then they have full control of your cluster.

You should create a new user specifically for beat ingest, and give it a role that only has access to the indices you want winlogbeat to write to

[ciphee (Daniel)]

Thanks for the reply, that was very informative. Right now this system is just a dev cluster and is only used for testing purposes and no real logs are being sent to it, but I definitely understand those security risks of using the elastic account.

I copied the settings from my elasticsearch.yml and it is below. Would I need to just copy these certs to the winlogbeat host and add the path to the winlogbeat.yml? Would one of these need to be recreated as a pem file?

cluster.name: mv-dev
node.name: ${HOSTNAME}
path.data: /data
path.logs: /var/log/elasticsearch
network.host: x.251.10.84
node.data: true
node.ingest: true
discovery.zen.ping.unicast.hosts: ["x.251.10.68", "x.251.10.84", "x.251.10.59"]
path.repo: ["/data/backup"]

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/nuqs-esdev02/nuqs-esdev02.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/nuqs-esdev02.nuqs-esdev02.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/nuqs-esdev02.com/nuqs-esdev02.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/nuqs-esdev02.com/nuqs-esdev02.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]

xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true