I tried everything from very beginning again , but elastic defend still not working , complaining about SSL problem .
Could you tell me after I install the elastic I can see there is http_ca.crt , what should I do with this crt ? How to solve the " self-signed cert in cert chain" problem ?
You need to convert it into PEM format for use in the policy.
A handy shortcut would be to use openssl to just caputre it from your ES response
openssl s_client -connect 192.168.71.130:9200 2>/dev/null | openssl x509 -outform PEM
Note, you can also use certificate fingerprint in ES output configuration instead of providing the PEM. You can grab it for example from a browser
BTW, I haven't watched that video which you posted above. There's another way to deal with self-signed CAs, you can import them to certificate store on your target machine, this way they are trusted as if they were provided by external certificate authority. It's possible that this method was used in the video example for simplicity.
I think I may find where the issue is ? I found there are no lock in my localhost:9200 , and it is complaining cert is not safe . but I have installed the http_ca.crt as root trust cert . and also , when I go in kibana , browser also says , there are no cert in the website .
I found the finger print there
鉁?Elasticsearch security features have been automatically configured!
鉁?Authentication is enabled and cluster connections are encrypted.
鈩癸笍 Password for the elastic user (reset with bin/elasticsearch-reset-password -u elastic):
P3Am2jHluSthq15fntfx
鈩癸笍 HTTP CA certificate SHA-256 fingerprint:
fc6782c835f01e720218b0613c219132ca6d9b5c2788c90182d645b51bc4dcd9
鈩癸笍 Configure Kibana to use this cluster:
鈥?Run Kibana and click the configuration link in the terminal when Kibana starts.
鈥?Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
eyJ2ZXIiOiI4LjEzLjIiLCJhZHIiOlsiMTkyLjE2OC43MS4xMzA
Yet after I pasted in output setting , it is still complaining :
16:36:25.076
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] MessageHelpers.cpp:313 CURL error: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self-signed certificate in certificate chain]
16:36:25.094
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] Certificates.cpp:196 Skipping store location 327680
16:36:25.094
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] Certificates.cpp:196 Skipping store location 393216
16:36:25.094
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] Certificates.cpp:271 Number of certificates: 24
16:36:25.099
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] Certificates.cpp:196 Skipping store location 327680
16:36:25.099
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] Certificates.cpp:196 Skipping store location 393216
16:36:25.099
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] Certificates.cpp:271 Number of certificates: 25
16:36:25.100
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][notice] BulkQueueConsumer.cpp:19
AFTER I installed the http_ca.crt to ROOT TRUST CERT , it begins to say
16:37:01.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] ElasticsearchClient.cpp:206 HTTP Status Code (401): {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
16:37:01.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] BulkQueueConsumer.cpp:122 Will not attempt to connect to Elasticsearch for 1 more seconds due to 401 unauthorized connection response
16:37:01.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][notice] BulkQueueConsumer.cpp:193 Elasticsearch connection is down
16:37:03.826
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] AgentContext.cpp:486 Endpoint is setting status to DEGRADED, reason: Unable to connect to output server
16:37:06.638
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] ElasticsearchClient.cpp:206 HTTP Status Code (401): {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
16:37:06.638
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] BulkQueueConsumer.cpp:122 Will not attempt to connect to Elasticsearch for 3 more seconds due to 401 unauthorized connection response
16:37:06.638
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][notice] BulkQueueConsumer.cpp:193 Elasticsearch connection is down
16:37:11.756
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] ElasticsearchClient.cpp:206 HTTP Status Code (401): {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
16:37:11.756
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] BulkQueueConsumer.cpp:122 Will not attempt to connect to Elasticsearch for 7 more seconds due to 401 unauthorized connection response
16:37:11.756
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][notice] BulkQueueConsumer.cpp:193 Elasticsearch connection is down
16:37:16.770
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][notice] BulkQueueConsumer.cpp:193 Elasticsearch connection is down
16:37:20.375
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-04-28T095040.log
16:37:20.375
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-04-28T104420.log
16:37:20.391
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-04-29T113952.log
16:37:20.437
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-04-30T120438.log
16:37:20.454
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-04-30T164551.log
16:37:20.469
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-04T203323.log
16:37:20.500
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T091437.log
16:37:20.500
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T093518.log
16:37:20.516
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T102039.log
16:37:20.532
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T102216.log
16:37:20.578
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T102631.log
16:37:20.578
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T105204.log
16:37:20.594
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T105547.log
16:37:20.594
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] LogStorage.cpp:572 Unsynced document log store: C:\Program Files\Elastic\Endpoint\state\documents\documents-2024-05-06T111326.log
16:37:21.807
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] ElasticsearchClient.cpp:206 HTTP Status Code (401): {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id crWYEI8B1dqHVaRseWrV","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
Here is the status
Moreover , I can not see the policy in the output of inspect cmd , it there the problem located ?
Is there any idea about it ?
Since it's your local stack setup I'd suggest to query the API keys on Elasticsearch directly using Basic HTTP authentication (user:password).
Well, I've just realized you are running Endpoint 8.13.2 on Chinese language Windows. Sorry to catch it so late, but we are aware of a rare (hopefully rare) bug with handling Chinese file names which causes unexpected Endpoint service restarts. It has been fixed in 8.13.3. This could explain why the Endpoint is not getting the policy from Agent.
Could you launch Task Manager and check if elastic-endpoint.exe pid is changing? Alternatively you can repeatedly run this in console
TASKLIST /FI "IMAGENAME eq elastic-endpoint.exe"
If it's changing, that's not good, you need to upgrade to 8.13.3.
Also, if you are affected by this bug, but 8.13.3 doesn't solve it, please let us know.
Seems not happening to me ,
Is there any else potential cause that endpoint is not getting policy ?
The endpoint is healthy now , I think I am getting close ... only , the output seems still in trouble
The log says
[elastic_agent.endpoint_security][warning] AgentContext.cpp:486 Endpoint is setting status to DEGRADED, reason: Unable to connect to output server
Then I follow your advice , I added the fingher print to output
The result is
Sorry again for these so many stuff
"The result" is yet another set of issues.
- policy signature
The error "Not applying config from agent because signature verification failed" means that Agent forwarded policy from the stack to Endpoint but it was not signed properly. It's signed up the stack, maybe in fleet but I'm not sure where exactly, I'm Endpoint developer.
You're doing a lot of changes, it's possible that you've changed the signing key in the stack after having Endpoint installed. It such case it won't accept any new policy because of the signature verification.
Just uninstall Endpoint, then stop and start Agent service to make it reinstall Endpoint.
"C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" uninstall
sc stop "Elastic Agent"
sc start "Elastic Agent"
- IPv6
The localhost resolves to IPv6 [::1] on your system, I'm not sure if that's supported.
I hope you've already found this page
Installing in an air-gapped environment | Elastic Installation and Upgrade Guide [8.13] | Elastic
Damn! does that mean it is working ?
I did you told me to uninstall the endpoint and restart it , moreover , I uninstalled the agent and reinstalled it .
before that , I always use
.\elastic-agent.exe install --insecure -f to create another fleet server ( cuz it asked me to create a healthy fleet server) is this cmd not good ?
Seems working
, Could you tell me what was the reason?all my steps are recorded in this post. cuz I still have 2 more machine to do same configuration.
I don't recommend using force switch `-f``
Elastic Agent command reference | Fleet and Elastic Agent Guide [8.13] | Elastic
The note here is perhaps misleading
If the Elastic Agent is already installed on the host, using
--forcemay result in unpredictable behavior with duplicate Elastic Agents appearing in Fleet.
The stress is on unpredictable behavior, where duplicate Agents appearing in Fleet is just a minor side effect.
Yes, if you had already Agent with Elastic Defend installed using --force certainly could lead to broken/unmanageable Elastic Defend.
Thank you for all this time , it is now working ! but allow me to ask one more problem , why it can not detect mimikatz , while I have actived 1088 rules?
And then I have set the elastic security from prevent to detect , still no use .. how should I test it ?
You're confusing SIEM rules with Elastic Defend policy. Click the Manage button on bottom-left side, then configure the policy.
Elastic Defend policy defines how Endpoint service is behaving on the target machine. You want everything in prevention mode to have it block malicious things.
SIEM rules run in the stack analyzing incoming data.
I indeed want to anaysis data in the elastic , for that I followed two guides
And I can see they are also adding rules just like mine
Certainly, SIEM has also great potential
