I've been digging into this for a few days now and I feel I've reached a wall.
-
elastic-agentconnects to both elastic and fleet and I can see the logs coming in. -
elastic-endpointconnects to fleet but refuses to connect to elastic.
Versions:
- Security endpoint integration - 8.2.0
- Everything else - 8.1.3
- Ubuntu 20.04 for all machines
Docker Setup:
- 3 es nodes, kibana & fleet all on same host
- TLS enabled with self-signed certs
Agent Setup:
- Has endpoint integration
- Enrolled VIA fleet successfully with --insecure flag
-
elastic-agent.serviceandElasticEndpoint.serviceboth running - Self-signed ca.crt copied to trusted root directory on host ran
update-ca-certificates
Here's some log outputs where you can see the errors:
sudo tail -f /opt/Elastic/Endpoint/state/log/endpoint-000000.log | grep "Elasticsearch connection"
{"@timestamp":"2022-04-27T20:52:02.636461758Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:07.666434545Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:12.697770186Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:17.734238229Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:22.766463795Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
from /opt/Elastic/Endpoint I ran ./elastic-endpoint test output
Testing output connections using config file: [/opt/Elastic/Endpoint/elastic-endpoint.yaml]
Using proxy:
Elasticsearch server: https://10.118.0.3:9200
Status: SSL peer certificate or SSH remote key was not OK [SSL: no alternative certificate subject name matches target host name '10.118.0.3'] ()
Help: Host needs to trust server cert or server cert needs to be added to Elasticsearch/Fleet config
Global artifact server: https://artifacts.security.elastic.co
Status: Success
Fleet server: https://10.118.0.3:8220
Status: Success
& finally elastic-agent diagnostics
elastic-agent version: 8.1.3
build_commit: 271435c21bfd4e2e621d87c04f4b815980626978 build_time: 2022-04-19 11:42:39 +0000 UTC snapshot_build: false
Applications:
* name: endpoint-security route_key: default
error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory
* name: filebeat route_key: default
process: filebeat id: 6fe5d29f-dfa8-408e-a812-20fdea1108a6 ephemeral_id: 05630af1-426c-4fe4-a8d4-464d92ed0cde elastic_license: true
version: 8.1.3 commit: 271435c21bfd4e2e621d87c04f4b815980626978 build_time: 2022-04-19 09:29:51 +0000 UTC binary_arch: amd64
hostname: lemmy-tryp username: root user_id: 0 user_gid: 0
* name: metricbeat route_key: default
process: metricbeat id: 91610db5-3aab-4499-9830-ec6d32b69a56 ephemeral_id: 4db08166-5ee8-4220-8eb1-06dd45e75e80 elastic_license: true
version: 8.1.3 commit: 271435c21bfd4e2e621d87c04f4b815980626978 build_time: 2022-04-19 09:42:04 +0000 UTC binary_arch: amd64
hostname: lemmy-tryp username: root user_id: 0 user_gid: 0
* name: filebeat_monitoring route_key: default
process: filebeat id: 6fe5d29f-dfa8-408e-a812-20fdea1108a6 ephemeral_id: 05630af1-426c-4fe4-a8d4-464d92ed0cde elastic_license: true
version: 8.1.3 commit: 271435c21bfd4e2e621d87c04f4b815980626978 build_time: 2022-04-19 09:29:51 +0000 UTC binary_arch: amd64
hostname: lemmy-tryp username: root user_id: 0 user_gid: 0
* name: metricbeat_monitoring route_key: default
process: metricbeat id: 91610db5-3aab-4499-9830-ec6d32b69a56 ephemeral_id: 4db08166-5ee8-4220-8eb1-06dd45e75e80 elastic_license: true
version: 8.1.3 commit: 271435c21bfd4e2e621d87c04f4b815980626978 build_time: 2022-04-19 09:42:04 +0000 UTC binary_arch: amd64
hostname: lemmy-tryp username: root user_id: 0 user_gid: 0
Obviously these two are the culprit but I can't seem to find out why
Status: SSL peer certificate or SSH remote key was not OK [SSL: no alternative certificate subject name matches target host name '10.118.0.3'] ()
Help: Host needs to trust server cert or server cert needs to be added to Elasticsearch/Fleet config
name: endpoint-security route_key: default
error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory
The issue doesn't seem to be related to the 1st error as I've already copied over the ca.crt to the host trusted root directory and I can see it in my elastic-endpoint.yaml anyways as shown below.
fleet:
access_api_key: <my secret I don't want the world to see :P>
agent:
id: 8e9baa82-f40f-496a-85f1-7294c12982fb
logging:
level: info
monitoring:
http:
enabled: false
host: ""
port: 6791
pprof: null
enabled: true
host:
id: 39f47717d43c723ca8b1bae56246ea55
hosts:
- https://10.118.0.3:8220
protocol: http
reporting:
check_frequency_sec: 30
threshold: 10000
ssl:
renegotiation: never
verification_mode: none
timeout: 10m0s
inputs:
- artifact_manifest:
artifacts:
endpoint-eventfilterlist-linux-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-eventfilterlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-eventfilterlist-macos-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-eventfilterlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-eventfilterlist-windows-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-eventfilterlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-exceptionlist-linux-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-exceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-exceptionlist-macos-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-exceptionlist-windows-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-hostisolationexceptionlist-linux-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-hostisolationexceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-hostisolationexceptionlist-macos-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-hostisolationexceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-hostisolationexceptionlist-windows-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-hostisolationexceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-trustlist-linux-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-trustlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-trustlist-macos-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-trustlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
endpoint-trustlist-windows-v1:
compression_algorithm: zlib
decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
decoded_size: 14
encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
encoded_size: 22
encryption_algorithm: none
relative_url: /api/fleet/artifacts/endpoint-trustlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
manifest_version: 1.0.0
schema_version: v1
data_stream:
namespace: default
id: 63151788-edc6-4bf1-a360-77bddbe8df84
meta:
package:
name: endpoint
version: 8.2.0
name: lemmy-sec
policy:
linux:
behavior_protection:
mode: prevent
supported: true
events:
file: true
network: true
process: true
logging:
file: info
malware:
mode: prevent
memory_protection:
mode: prevent
supported: true
popup:
behavior_protection:
enabled: true
message: ""
malware:
enabled: true
message: ""
memory_protection:
enabled: true
message: ""
mac:
behavior_protection:
mode: prevent
supported: true
events:
file: true
network: true
process: true
logging:
file: info
malware:
mode: prevent
memory_protection:
mode: prevent
supported: true
popup:
behavior_protection:
enabled: true
message: ""
malware:
enabled: true
message: ""
memory_protection:
enabled: true
message: ""
windows:
antivirus_registration:
enabled: false
behavior_protection:
mode: prevent
supported: true
events:
dll_and_driver_load: true
dns: true
file: true
network: true
process: true
registry: true
security: true
logging:
file: info
malware:
mode: prevent
memory_protection:
mode: prevent
supported: true
popup:
behavior_protection:
enabled: true
message: ""
malware:
enabled: true
message: ""
memory_protection:
enabled: true
message: ""
ransomware:
enabled: true
message: ""
ransomware:
mode: prevent
supported: true
revision: 1
type: endpoint
use_output: default
output:
elasticsearch:
api_key: <my secret I don't want the world to see :P>
hosts:
- https://10.118.0.3:9200
ssl:
ca_trusted_fingerprint: <my secret I don't want the world to see :P>
certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
<my secret I don't want the world to see :P>
-----END CERTIFICATE-----
revision: 5
The 2nd error is interesting because it seems to be pointing towards the wrong .sock, when I go into the directories I can see this:
root@lemmy-tryp:/opt/Elastic/Endpoint/cache# ls
ElasticEndpointServiceCommsSocket artifacts eBPF resources
root@lemmy-tryp:/opt/Elastic/Agent/data/tmp# ls
default elastic-agent.sock
root@lemmy-tryp:/opt/Elastic/Agent/data/tmp# cd default
root@lemmy-tryp:/opt/Elastic/Agent/data/tmp/default# ls
filebeat metricbeat
Seems like I need to point my elastic-endpoint to the correct socket location but after searching I'm unable to find any information related to configuring the socket in in the docs or github.
As a possible workaround both my ELK stack and host I want to monitor are on the same VPC network so for this exercise I don't actually care if the communication is secured, just need that elastic-endpoint to connect!
Thanks for any help 