Endpoint security configuration

[ttyser (Tomas)]

Hello all, I have an agent deployed via fleet on red hat server and I want to use the endpoint security there. I did add the endpoint security integration under the policy the red hat agent is bound to. Now it seems like the endpoint security integration is deployed on the red har server, but its not sending any data. When looking into /opt/Elastic/Endpoint/state/log/endpoint-000000.log I see the following:

{"@timestamp":"2022-09-29T17:04:51.505793025Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:04:56.584886045Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:04:56.585562335Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:01.650654765Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:01.651124587Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:05.7893431Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":71,"name":"Logging.cpp"}}},"message":"Logging.cpp:71 Logging directory cleaned up, current size: 4276961","process":{"pid":13348,"thread":{"id":13352}}}
{"@timestamp":"2022-09-29T17:05:06.716216337Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:06.716744821Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:11.783569857Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:11.784023597Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:16.868162401Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:16.868579418Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:21.93507682Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:21.935224018Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:26.998394775Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":13348,"thread":{"id":13355}}}
{"@timestamp":"2022-09-29T17:05:26.998512665Z","agent":{"id":"9c3f597a-6b98-41a1-a68f-b80a605506b8","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":13348,"thread":{"id":13355}}}

This seems to me like all I need to do is to add the Elasticsearch http_ca.crt to the red hat server and customize the elastic-endpoint.yaml file with something like:

output:
  elasticsearch:
    api_key: _8zfiIMBLpY_2jDpd3tG:_r-kPp04Ql6KKA91yhJniA
    hosts:
    - https://10.212.25.197:9200
    protocol: http
  reporting:
    check_frequency_sec: 30
    threshold: 10000
  ssl:
    certificate_authorities:
    - /opt/Elastic/certs/http_ca.crt
    renegotiation: never
    verification_mode: full
  timeout: 10m0s

But when I restarted the endpoint service, the .yaml file was rewriten to the default config. Can someone explain how should this work? Thank you!

[ferullo (Daniel Ferullo)]

Hi @ttyser

The Endpoint configuration file you're editing is not designed to be changed on the host. It'll be overwritten as needed. To make the change you're trying to make go to Fleet -> Settings then find the relevant output in the Output table (by default it'll be "default") and click on the edit icon on the right side of the entry. In the right hand fly out that appears there is an Advanced YAML configuration section. If you make your changes there you should see them be sent down to all of the Agent's and Endpoint's associated with that output.

I hope that helps. If not, comment back and let me know and we can dig into this further.

[ttyser (Tomas)]

Ok, this is place where I was looking first. But here everything is grayed now. I guess beacause Kibana was configured via enrollment token? Not sure.

When looking into kibana.yml I see this:

# This section was automatically generated during setup.
elasticsearch.hosts: ['https://10.212.25.197:9200', 'https://10.212.25.198:9200', 'https://10.212.25.199:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2NjM5MzMxMTA0OTE6REE4dG11SVJTVGk0ZmJZbWxPMFRhZw
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1663933111431.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://10.212.25.197:9200'], ca_trusted_fingerprint: f638104b2c49a6dc75381ce5ec4fcf6989f9cbacc7fdaefeb7301ed91ec11547}]    

[ttyser (Tomas)]

Hey Daniel, a fix was suggested to me by James Spiteri, so I will look into it and update here later.

[ttyser (Tomas)]

I can confirm the issue was fixed by adding the /opt/Elastic/certs/http_ca.crt to the "linux.advanced.elasticsearch.tls.ca_cert" section under "Show advanced settings" option in Kibana GUI under " Edit Endpoint and Cloud Security integration" policy.

[ferullo (Daniel Ferullo)]

Great! I'm glad you got it working.

For posterity's sake, I'll say that the reason the Fleet output settings I were greyed out is because you've edited kibana.yml. So if you'd wanted to solve this issue via Fleet settings rather than via Endpoint's policy you'd have to have made changes to kibana.yml. But what you did works, I'm not suggesting you make any changes.

[ttyser (Tomas)]

Hi Daniel, thank you for the info. Yes, I did edit kibana.yml with the Kibana encryption keys:

# Kibana Encryption Keys.
xpack.encryptedSavedObjects.encryptionKey: 80286b6d9158138153496b871653d436
xpack.reporting.encryptionKey: e814db6bbe953b45381fdba4a48d0371
xpack.security.encryptionKey: 368eb4d926929a09f9beafe46d0107f5

Also I have cluster of 3 nodes and when I created the first node I did use the Kibana enrollment token to connect the Kibana with the Elasticsearch node. Then later I did add two more nodes via node enrollment tokens. After that I noticed that when the node-1 went down, the Kibana was showing message like cant read the licence or something similar and other errors. So I went to kibana.yml and put the 2 aditional nodes there. Since that is seems to be ok. Based on what you said I could probably add them via setting>Outputs in Kibana GUI. But how would I add the Encryption keys?

Let me ask you one more thing which can be little bit related to what we are speaking about now. I noticed that the problem I had with the Endpoint security is still affecting other integrations, like for example "System" deployed via fleet to endpoint. So where do I set this up? Thank you!

UPDATE:
Based on other post where you did reply I undrstand that all the confugration should be primarily done from Kibana gui > Fleet>Settings>Outputs>Advanced YAML configuration. Is there a way how to recover the possibility to do this? What I am trying to say is, if I try to remove the configuration I did manualy directlly to kibana.yml will I get the functionality to add config via kibana gui back?

UPDATE2:
Daniel, I desided to create all from the scratch so I have much "clearer" installation. My problem is, when trying to connect Kibana to ES with enrollement token, the Kibana will by default be on "(http://localhost:5601)" which means I will not be able to start the webgui. If I will go to kibana.yml and add the info there, I will again end up in with Fleet output settings greyed. How do I go around this?

[ttyser (Tomas)]

Ok, I solved that by not using the enrollment token and just configure the Kibana manually via the kibana.yml and adding the ES http_ca.crt into Kibana node /etc/kibana/certs. I can now edit the Fleet config via webgui.