Elastic agent unhealthy because of elastic defend integration

Hello community I'm having an issue in my elastic agent, I installed the agent in windows and Linux machines is working well and I can get the event logs and syslog, however when I try to add elastic defend integration the agent becomes unhealthy. here are the screen shots.

as you can see here it says policy degraded and Elasticsearch connection error.

Agent error logs are
[elastic_agent][info] Unit state changed endpoint-default (STARTING->HEALTHY): Applied policy {a1254802-bdd4-4009-a54c-87c7d787c6a7}
09:20:40.729
elastic_agent
[elastic_agent][warn] Unit state changed endpoint-default-a1254802-bdd4-4009-a54c-87c7d787c6a7 (HEALTHY->DEGRADED): Applied policy {a1254802-bdd4-4009-a54c-87c7d787c6a7}
09:20:40.729
elastic_agent
[elastic_agent][warn] Unit state changed endpoint-default (HEALTHY->DEGRADED): Applied policy {a1254802-bdd4-4009-a54c-87c7d787c6a7}

anyone who can help me please.

Thanks.

Hi @laale1

This error means that Endpoint is not able to connect to Elasticsearch. The two most likely causes are a networking issue (like a firewall) blocking the connection or an SSL configuration error.

To diagnose it, I recommend turning on debug logging (click on Logs in the screenshot you shared, then change the "Agent logging level" at the bottom on screen to "Debug" and click "Apply changes"). In a few moments you should see Endpoint start emitting debug logs (which contain {... "log": {"level":"debug"...} ...} in them).

To check Endpoint logs open an Administrator cmd.exe window and open the highest numbered file in c:\Program Files\Elastic\Endpoint\state\log\ (e.g. endpoint-000000.log). In the logs you should see messages like what's below every time Endpoint tries to connect to Elasticsearch just before the log Elasticsearch connection is down.

In my example below the problem was that Elasticseach was not running on localhost:1443. You'll see different logs depending on what the problem is. If from reading them you still aren't sure what the problem is, just share sanitized logs here.

{"@timestamp":"2023-08-17T20:28:19.58268557Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"debug","origin":{"file":{"line":672,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:672 Making HTTP request without a proxy","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.585224682Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"debug","origin":{"file":{"line":859,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:859 Adding 274 CA certificates","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.585261548Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"debug","origin":{"file":{"line":897,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:897 Removing SNI from no connection? https://localhost:1443/","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.585274454Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"debug","origin":{"file":{"line":1026,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1026 Establishing GET connection to [https://localhost:1443/]","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.932462326Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"debug","origin":{"file":{"line":322,"name":"Http.cpp"}}},"message":"Http.cpp:322 CURL error 0: No error","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.932483463Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"debug","origin":{"file":{"line":334,"name":"Http.cpp"}}},"message":"Http.cpp:334 HTTP code 404: Not Found","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.932579839Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":244,"name":"ElasticsearchClient.cpp"}}},"message":"ElasticsearchClient.cpp:244 HTTP Status Code (404): {\"message\":\"Unknown resource.\",\"ok\":false}","process":{"pid":39871,"thread":{"id":39884}}}
{"@timestamp":"2023-08-17T20:28:19.932604527Z","agent":{"id":"0268482d-8fec-411b-b97e-9d6556b40238","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":93,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:93 Elasticsearch connection is down","process":{"pid":39871,"thread":{"id":39884}}}

I should add, if you are using Logstash rather than Elasticsearch, then look for the logs "Logstash connection is down". However, if you're using Logstash another common issue could be that Logstash is receiving documents from Endpoint but not forwarding them into Elasticsearch, in which case Endpoint won't report any errors in it's logs.

Hello @ferullo thanks for your replay, debug error is saying :-

elastic_agent

[elastic_agent][debug] update marker not present at 'C:\Program Files\Elastic\Agent\data'

elastic_agent
[elastic_agent][debug] update marker not present at 'C:\Program Files\Elastic\Agent\data'

Hey @laale1. Those are Agent logs, we need to look at Endpoint logs (while Endpoint is fully managed by Agent, they're different services and log independently of each other). If you can't look at them on the host you can see them in the Stack on the same page where you change the log level to Debug. Make sure to select just the elastic_agent.endpoint_security Dataset and select all 4 Log Levels.

hey @ferullo thanks for the replies I finally solved the issue.
it the certificate I was using salve signed certificate and I wasn't adding the --insecure when installing the agents.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.