Error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory

I've been digging into this for a few days now and I feel I've reached a wall.

  • elastic-agent connects to both elastic and fleet and I can see the logs coming in.
  • elastic-endpoint connects to fleet but refuses to connect to elastic.

Versions:

  • Security endpoint integration - 8.2.0
  • Everything else - 8.1.3
  • Ubuntu 20.04 for all machines

Docker Setup:

  • 3 es nodes, kibana & fleet all on same host
  • TLS enabled with self-signed certs

Agent Setup:

  • Has endpoint integration
  • Enrolled VIA fleet successfully with --insecure flag
  • elastic-agent.service and ElasticEndpoint.service both running
  • Self-signed ca.crt copied to trusted root directory on host ran update-ca-certificates

Here's some log outputs where you can see the errors:

sudo tail -f /opt/Elastic/Endpoint/state/log/endpoint-000000.log | grep "Elasticsearch connection"

{"@timestamp":"2022-04-27T20:52:02.636461758Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:07.666434545Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:12.697770186Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:17.734238229Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}
{"@timestamp":"2022-04-27T20:52:22.766463795Z","agent":{"id":"8e9baa82-f40f-496a-85f1-7294c12982fb","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":27099,"thread":{"id":27122}}}

from /opt/Elastic/Endpoint I ran ./elastic-endpoint test output

Testing output connections using config file: [/opt/Elastic/Endpoint/elastic-endpoint.yaml]

Using proxy: 

Elasticsearch server: https://10.118.0.3:9200
        Status: SSL peer certificate or SSH remote key was not OK [SSL: no alternative certificate subject name matches target host name '10.118.0.3'] ()
        Help: Host needs to trust server cert or server cert needs to be added to Elasticsearch/Fleet config

Global artifact server: https://artifacts.security.elastic.co
        Status: Success

Fleet server: https://10.118.0.3:8220
        Status: Success

& finally elastic-agent diagnostics

elastic-agent  version: 8.1.3
               build_commit: 271435c21bfd4e2e621d87c04f4b815980626978  build_time: 2022-04-19 11:42:39 +0000 UTC  snapshot_build: false
Applications:
  *  name: endpoint-security  route_key: default
     error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory
  *  name: filebeat               route_key: default
     process: filebeat            id: 6fe5d29f-dfa8-408e-a812-20fdea1108a6          ephemeral_id: 05630af1-426c-4fe4-a8d4-464d92ed0cde  elastic_license: true
     version: 8.1.3               commit: 271435c21bfd4e2e621d87c04f4b815980626978  build_time: 2022-04-19 09:29:51 +0000 UTC           binary_arch: amd64
     hostname: lemmy-tryp         username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat             route_key: default
     process: metricbeat          id: 91610db5-3aab-4499-9830-ec6d32b69a56          ephemeral_id: 4db08166-5ee8-4220-8eb1-06dd45e75e80  elastic_license: true
     version: 8.1.3               commit: 271435c21bfd4e2e621d87c04f4b815980626978  build_time: 2022-04-19 09:42:04 +0000 UTC           binary_arch: amd64
     hostname: lemmy-tryp         username: root                                    user_id: 0                                          user_gid: 0
  *  name: filebeat_monitoring    route_key: default
     process: filebeat            id: 6fe5d29f-dfa8-408e-a812-20fdea1108a6          ephemeral_id: 05630af1-426c-4fe4-a8d4-464d92ed0cde  elastic_license: true
     version: 8.1.3               commit: 271435c21bfd4e2e621d87c04f4b815980626978  build_time: 2022-04-19 09:29:51 +0000 UTC           binary_arch: amd64
     hostname: lemmy-tryp         username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat_monitoring  route_key: default
     process: metricbeat          id: 91610db5-3aab-4499-9830-ec6d32b69a56          ephemeral_id: 4db08166-5ee8-4220-8eb1-06dd45e75e80  elastic_license: true
     version: 8.1.3               commit: 271435c21bfd4e2e621d87c04f4b815980626978  build_time: 2022-04-19 09:42:04 +0000 UTC           binary_arch: amd64
     hostname: lemmy-tryp         username: root                                    user_id: 0                                          user_gid: 0

Obviously these two are the culprit but I can't seem to find out why

Status: SSL peer certificate or SSH remote key was not OK [SSL: no alternative certificate subject name matches target host name '10.118.0.3'] ()
        Help: Host needs to trust server cert or server cert needs to be added to Elasticsearch/Fleet config
 name: endpoint-security  route_key: default
     error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory

The issue doesn't seem to be related to the 1st error as I've already copied over the ca.crt to the host trusted root directory and I can see it in my elastic-endpoint.yaml anyways as shown below.

fleet:
  access_api_key: <my secret I don't want the world to see  :P>
  agent:
    id: 8e9baa82-f40f-496a-85f1-7294c12982fb
    logging:
      level: info
    monitoring:
      http:
        enabled: false
        host: ""
        port: 6791
      pprof: null
  enabled: true
  host:
    id: 39f47717d43c723ca8b1bae56246ea55
  hosts:
  - https://10.118.0.3:8220
  protocol: http
  reporting:
    check_frequency_sec: 30
    threshold: 10000
  ssl:
    renegotiation: never
    verification_mode: none
  timeout: 10m0s
inputs:
- artifact_manifest:
    artifacts:
      endpoint-eventfilterlist-linux-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-eventfilterlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-eventfilterlist-macos-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-eventfilterlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-eventfilterlist-windows-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-eventfilterlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-exceptionlist-linux-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-exceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-exceptionlist-macos-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-exceptionlist-windows-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-hostisolationexceptionlist-linux-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-hostisolationexceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-hostisolationexceptionlist-macos-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-hostisolationexceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-hostisolationexceptionlist-windows-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-hostisolationexceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-trustlist-linux-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-trustlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-trustlist-macos-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-trustlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
      endpoint-trustlist-windows-v1:
        compression_algorithm: zlib
        decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
        decoded_size: 14
        encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
        encoded_size: 22
        encryption_algorithm: none
        relative_url: /api/fleet/artifacts/endpoint-trustlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
    manifest_version: 1.0.0
    schema_version: v1
  data_stream:
    namespace: default
  id: 63151788-edc6-4bf1-a360-77bddbe8df84
  meta:
    package:
      name: endpoint
      version: 8.2.0
  name: lemmy-sec
  policy:
    linux:
      behavior_protection:
        mode: prevent
        supported: true
      events:
        file: true
        network: true
        process: true
      logging:
        file: info
      malware:
        mode: prevent
      memory_protection:
        mode: prevent
        supported: true
      popup:
        behavior_protection:
          enabled: true
          message: ""
        malware:
          enabled: true
          message: ""
        memory_protection:
          enabled: true
          message: ""
    mac:
      behavior_protection:
        mode: prevent
        supported: true
      events:
        file: true
        network: true
        process: true
      logging:
        file: info
      malware:
        mode: prevent
      memory_protection:
        mode: prevent
        supported: true
      popup:
        behavior_protection:
          enabled: true
          message: ""
        malware:
          enabled: true
          message: ""
        memory_protection:
          enabled: true
          message: ""
    windows:
      antivirus_registration:
        enabled: false
      behavior_protection:
        mode: prevent
        supported: true
      events:
        dll_and_driver_load: true
        dns: true
        file: true
        network: true
        process: true
        registry: true
        security: true
      logging:
        file: info
      malware:
        mode: prevent
      memory_protection:
        mode: prevent
        supported: true
      popup:
        behavior_protection:
          enabled: true
          message: ""
        malware:
          enabled: true
          message: ""
        memory_protection:
          enabled: true
          message: ""
        ransomware:
          enabled: true
          message: ""
      ransomware:
        mode: prevent
        supported: true
  revision: 1
  type: endpoint
  use_output: default
output:
  elasticsearch:
    api_key: <my secret I don't want the world to see  :P>
    hosts:
    - https://10.118.0.3:9200
    ssl:
      ca_trusted_fingerprint: <my secret I don't want the world to see  :P>
      certificate_authorities:
      - |
        -----BEGIN CERTIFICATE-----
 <my secret I don't want the world to see  :P>
        -----END CERTIFICATE-----
revision: 5

The 2nd error is interesting because it seems to be pointing towards the wrong .sock, when I go into the directories I can see this:

root@lemmy-tryp:/opt/Elastic/Endpoint/cache# ls
ElasticEndpointServiceCommsSocket  artifacts  eBPF  resources
root@lemmy-tryp:/opt/Elastic/Agent/data/tmp# ls
default  elastic-agent.sock
root@lemmy-tryp:/opt/Elastic/Agent/data/tmp# cd default
root@lemmy-tryp:/opt/Elastic/Agent/data/tmp/default# ls
filebeat  metricbeat

Seems like I need to point my elastic-endpoint to the correct socket location but after searching I'm unable to find any information related to configuring the socket in in the docs or github.

As a possible workaround both my ELK stack and host I want to monitor are on the same VPC network so for this exercise I don't actually care if the communication is secured, just need that elastic-endpoint to connect!

Thanks for any help :slight_smile:

Doing some more ts. Here's what's listening on the ports. Note tcp6 where elastic-endpoint is cutoff at the end?

root@lemmy-tryp:~# sudo netstat -anp | grep 6788
tcp        0      0 127.0.0.1:6788          0.0.0.0:*               LISTEN      674/elastic-agent   
unix  3      [ ]         STREAM     CONNECTED     1886788  668/dbus-daemon      /run/dbus/system_bus_socket
root@lemmy-tryp:~# sudo netstat -anp | grep 6789
tcp        0      0 127.0.0.1:6789          0.0.0.0:*               LISTEN      674/elastic-agent   
tcp        0      0 127.0.0.1:38472         127.0.0.1:6789          ESTABLISHED 2228/filebeat       
tcp        0      0 127.0.0.1:38452         127.0.0.1:6789          ESTABLISHED 2215/metricbeat     
tcp        0      0 127.0.0.1:6789          127.0.0.1:38506         ESTABLISHED 674/elastic-agent   
tcp        0      0 127.0.0.1:6789          127.0.0.1:38452         ESTABLISHED 674/elastic-agent   
tcp        0      0 127.0.0.1:6789          127.0.0.1:38472         ESTABLISHED 674/elastic-agent   
tcp        0      0 127.0.0.1:38506         127.0.0.1:6789          ESTABLISHED 2257/metricbeat     
tcp        0      0 127.0.0.1:38426         127.0.0.1:6789          ESTABLISHED 2183/filebeat       
tcp        0      0 127.0.0.1:6789          127.0.0.1:51918         ESTABLISHED 674/elastic-agent   
tcp        0      0 127.0.0.1:6789          127.0.0.1:38426         ESTABLISHED 674/elastic-agent   
tcp6       0      0 127.0.0.1:51918         127.0.0.1:6789          ESTABLISHED 38508/elastic-endpo

and additionally

root@lemmy-tryp:~# lsof -iTCP -P -n | grep elastic-e
elastic-e 38508            root   33u  IPv6  768045      0t0  TCP 127.0.0.1:51918->127.0.0.1:6789 (ESTABLISHED)
root@lemmy-tryp:~# ps aux| grep elastic-endpoint
root       38508  2.8  5.1 877116 104652 ?       Ssl  Apr27  22:21 /opt/Elastic/Endpoint/elastic-endpoint run
root       83894  0.0  0.0   8164   720 pts/0    S+   10:41   0:00 grep --color=auto elastic-endpoint

Now some more logs

root@lemmy-tryp:~# tcpdump -nn -i lo port 51918 -vv
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
10:37:37.311084 IP (tos 0x0, ttl 64, id 20804, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0xf0d7), seq 3710496807, ack 4292078324, win 3090, options [nop,nop,TS val 1135831261 ecr 1135816223], length 0
10:37:37.311145 IP (tos 0x0, ttl 64, id 61320, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [.], cksum 0xfe28 (incorrect -> 0xfae8), seq 1, ack 1, win 512, options [nop,nop,TS val 1135831261 ecr 1135816223], length 0
10:37:42.275586 IP (tos 0x0, ttl 64, id 61321, offset 0, flags [DF], proto TCP (6), length 193)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [P.], cksum 0xfeb5 (incorrect -> 0x8273), seq 1:142, ack 1, win 512, options [nop,nop,TS val 1135836226 ecr 1135816223], length 141
10:37:42.275611 IP (tos 0x0, ttl 64, id 20805, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0x8ec1), seq 1, ack 142, win 3090, options [nop,nop,TS val 1135836226 ecr 1135836226], length 0
10:37:42.275965 IP (tos 0x0, ttl 64, id 20806, offset 0, flags [DF], proto TCP (6), length 104)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [P.], cksum 0xfe5c (incorrect -> 0xf3e9), seq 1:53, ack 142, win 3090, options [nop,nop,TS val 1135836226 ecr 1135836226], length 52
10:37:42.276193 IP (tos 0x0, ttl 64, id 61322, offset 0, flags [DF], proto TCP (6), length 91)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [P.], cksum 0xfe4f (incorrect -> 0xed33), seq 142:181, ack 53, win 512, options [nop,nop,TS val 1135836226 ecr 1135836226], length 39
10:37:42.276201 IP (tos 0x0, ttl 64, id 20807, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0x8e66), seq 53, ack 181, win 3090, options [nop,nop,TS val 1135836226 ecr 1135836226], length 0
10:37:57.279105 IP (tos 0x0, ttl 64, id 20808, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0x53cc), seq 52, ack 181, win 3090, options [nop,nop,TS val 1135851229 ecr 1135836226], length 0
10:37:57.279171 IP (tos 0x0, ttl 64, id 61323, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [.], cksum 0xfe28 (incorrect -> 0x5ddd), seq 181, ack 53, win 512, options [nop,nop,TS val 1135851229 ecr 1135836226], length 0
10:38:02.278664 IP (tos 0x0, ttl 64, id 61324, offset 0, flags [DF], proto TCP (6), length 193)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [P.], cksum 0xfeb5 (incorrect -> 0x37cb), seq 181:322, ack 53, win 512, options [nop,nop,TS val 1135856229 ecr 1135836226], length 141
10:38:02.278692 IP (tos 0x0, ttl 64, id 20809, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0xf192), seq 53, ack 322, win 3090, options [nop,nop,TS val 1135856229 ecr 1135856229], length 0
10:38:02.279142 IP (tos 0x0, ttl 64, id 20810, offset 0, flags [DF], proto TCP (6), length 104)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [P.], cksum 0xfe5c (incorrect -> 0xaec7), seq 53:105, ack 322, win 3090, options [nop,nop,TS val 1135856229 ecr 1135856229], length 52
10:38:02.279425 IP (tos 0x0, ttl 64, id 61325, offset 0, flags [DF], proto TCP (6), length 91)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [P.], cksum 0xfe4f (incorrect -> 0xa176), seq 322:361, ack 105, win 512, options [nop,nop,TS val 1135856230 ecr 1135856229], length 39
10:38:02.323108 IP (tos 0x0, ttl 64, id 20811, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0xf10a), seq 105, ack 361, win 3090, options [nop,nop,TS val 1135856273 ecr 1135856230], length 0
10:38:17.503080 IP (tos 0x0, ttl 64, id 20812, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0xb5bf), seq 104, ack 361, win 3090, options [nop,nop,TS val 1135871453 ecr 1135856230], length 0
10:38:17.503131 IP (tos 0x0, ttl 64, id 61326, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [.], cksum 0xfe28 (incorrect -> 0xbfa5), seq 361, ack 105, win 512, options [nop,nop,TS val 1135871453 ecr 1135856273], length 0
10:38:22.283554 IP (tos 0x0, ttl 64, id 61327, offset 0, flags [DF], proto TCP (6), length 193)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [P.], cksum 0xfeb5 (incorrect -> 0x485c), seq 361:502, ack 105, win 512, options [nop,nop,TS val 1135876234 ecr 1135856273], length 141
10:38:22.283579 IP (tos 0x0, ttl 64, id 20813, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0x5460), seq 105, ack 502, win 3090, options [nop,nop,TS val 1135876234 ecr 1135876234], length 0
10:38:22.283866 IP (tos 0x0, ttl 64, id 20814, offset 0, flags [DF], proto TCP (6), length 104)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [P.], cksum 0xfe5c (incorrect -> 0x515d), seq 105:157, ack 502, win 3090, options [nop,nop,TS val 1135876234 ecr 1135876234], length 52
10:38:22.284291 IP (tos 0x0, ttl 64, id 61328, offset 0, flags [DF], proto TCP (6), length 91)
    127.0.0.1.51918 > 127.0.0.1.6789: Flags [P.], cksum 0xfe4f (incorrect -> 0x309f), seq 502:541, ack 157, win 512, options [nop,nop,TS val 1135876234 ecr 1135876234], length 39
10:38:22.284303 IP (tos 0x0, ttl 64, id 20815, offset 0, flags [DF], proto TCP (6), length 52)
    127.0.0.1.6789 > 127.0.0.1.51918: Flags [.], cksum 0xfe28 (incorrect -> 0x5405), seq 157, ack 541, win 3090, options [nop,nop,TS val 1135876234 ecr 1135876234], length 0
^C
21 packets captured
42 packets received by filter
0 packets dropped by kernel

More info to ts with

root@lemmy-tryp:~# dig localhost

; <<>> DiG 9.16.1-Ubuntu <<>> localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8291
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;localhost.                     IN      A

;; ANSWER SECTION:
localhost.              0       IN      A       127.0.0.1

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Apr 28 10:53:12 UTC 2022
;; MSG SIZE  rcvd: 54

Thanks for all the details. The output from elastic-endpoint test output and Endpoint's config are very useful here.

I'm surprised that connecting to the same address over https works for Fleet Server and fails for Elasticsearch. The only two differences between those connections are the port connected to, and that for Elasticsearch there is an additional certificate authority available for validation and a CA fingerprint.

The additional certificate authority for Elasticsearch shouldn't cause an issue (its additive), and it doesn't look like you're getting far enough along for the CA fingerprint to be the culprit.

If you want to run down getting a secure connection, three things you might try:

  1. Remove the Elasticsearch CA in the config just to make sure that isn't causing the issue
  2. Changing the Elasticsearch port in the config the Fleet Server's port just to see if the SSL connection establishes (of course this isn't a workable solution long term since it won't result in data being actually written to Elasticsearch).
  3. Check if there is a proxy sitting in front of Elasticsearch that is serving up a different SSL certificate to Endpoint than is expected. I think openssl s_client -showcerts -connect 10.118.0.3:9200 compared to the same command against port 8220 (Fleet Server) will help diagnose if that's the case.

However, you say you aren't very worried about HTTPS. In that case, based on the error you're seeing I think if you add into the ssl settings for Elasticsearch verification_mode: certificate (or none) the connection will establish. (link)

1 Like

So changing it to verification_mode: certificate worked! I had to do this in Kabana under the advanced YAML section. All the filebeat stuff in the security overview swapped to Endpoint so I would say the pipeline is complete and I have line-data on the network map finally. Thank you!

Since I was curious I also compared openssl s_client -showcerts -connect 10.118.0.3:9200 against openssl s_client -showcerts -connect 10.118.0.3:8220 and found out that it's serving up different certs, eso1 and fleet respectively the latter of which was my ca.crt. So maybe I am behind a reverse proxy.

Haven't got a chance to try it but if I were to add es01.crt to my advanced YAML config should that solve it? The script I was using dropped fleet/ca.crt in there for me.

Great, I'm glad data is flowing for you now.

It's worth a try, but I suspect that won't work. The hostname in the certificate Endpoint received from connecting to 10.118.0.3:9200 doesn't seem like it matched the address/hostname the connection was to (10.118.0.3), causing the full verification to fail.