All Rules are showing Failed

As of last night this also happened on a fresh install of 7.11.1 on a new install of CentOS 8. All packages pulled from the repo.

Hi Devin,

Thanks for clearing the air a bit on the issue. In regards to the elastic agent integration

When I had 7.11 set up with Elastic Agent through Fleet I was getting errors across the board with out-of-box rules. In this case is the recommended path to add integrations to the Fleet Policy to satisfy these checks?

For example, for the following error:

The following index patterns did not match any indices: ["auditbeat-*","*"]

I would need to add integrations that had indexes for auditbeat-*,* to policy to ensure that logs for that rule would be captured correct?

I would need to add integrations that had indexes for auditbeat-, to policy to ensure that logs for that rule would be captured correct?

Essentially yes but there is a condition that is special specifically to the* index pattern where that index is not set up until the endpoint integration ships data (specifically, the endpoint detects something) but as long as auditbeat-* or* match some index in Elasticsearch then you won't see this specific message for that rule.

For those that are have the same issue on existing clusters that have existing integrations and have gone over the long thread. Here's a method to force success as per @Devin_Hurley recommendations. Not sure how to trigger the Endgame-* to be created without getting into higher risk malware situation.

  1. In DEV tools in Kibana " GET logs-endpoint.alerts-*/_mapping " If it's empty try below"
  2. Create sandbox VM. "Windows 10" No need to patch.
  3. Load new policy with Endpoint/System/Windows. In systems integration disable systems load "not supported on windows". Set the Endpoint to detect only! Make sure to register Endpoint as the AV for windows or defender will stop you.
  4. Load Endpoint on the device.
  5. Download Mimikatz from GitHub.
  6. Execute Minikatz. You do not need to do anything past run it. This will force load the index and at least allow Elastic Security to stop showing as failed.

One thing to note is this is not a fix another thing that broken. Analyze event no longer works "Error loading data". Any event's you had like email or webhook no longer function.

Recreate index it get's deleted on upgrades or wait for malware. Someone will do it sooner or later.
Avoid moving to 7.11 or 7.11.1 until it's fixed upstream if you use Endpoint. The devs are awesome it will get fixed soon.

1 Like

I set up as you sait, with endpoint, system and windows integration only to detect.
Unfortunately Elastic Security in version 7.11.1 (it was working and notifying a malware on host and not letting eicar execute) is somehow not woking anymore.

Here's a recorded gif. I can execute mimikatz and eicar and nothing's happening : /


The analyzer event no longer works is known and there is a fix coming incredibly soon for it:

Workaround for it in the meantime is:

  1. In the Detections page, find the alert of interest, click on "Investigate in Timeline".
  2. Inside of Timeline, find the alert event and click on "Analyze Event".
  3. User should be able to investigate

Sometimes the alerts can take a while before they show up from the endpoints and we are working to reduce the time on that but you should see the alerts from the endpoints.

1 Like

@Frank_Hassanabad Glad to hear it! Kind of like that analyze feature while not the quickest to see what the file name was the chain of events is wonderful.

@Edvin22 Disable smart screen and app and browser control settings. May have to do it by GPO on that build. That's Windows Defender kicking in preventing Endpoint from ever seeing the event on the device. Which would prevent Elastic/Kibana from ever seeing the event.

I'm using Win 10 Enterprise LTSC builds and don't have any of the standard pro's running in my environment. It's actually best to have Defender running along side Endpoint IMHO just due to the fact that it's a file level scan tool where Endpoint is a process based. Just make sure to setup the limits for Defender and you get the best of both worlds until testing like this.

1 Like

Thank you for your reply I will take in mind the connection between file and process level scan.
Although I tried disabling Defender and SmartScreen and still no reaction from Elastic EDR to Eicar file. I'm following the same steps I did in previous 7.10 release but in 7.11 it's just not working anymore. I'm using default settings.

Just from what I've seen Eicar does not get detected by Endpoint. It's a dummy "file" with no payload so it's not surprising to have it skipped. Try minikatz and see if it triggers at least the Endpoint Security Event. It will not trigger anything to recreate the endgame-* index unless you actually run it. Minikatz will not cause harm just by running it directly at least it's current version. It's a tool and the payload is only used after you have selected a target.

For anyone else that run's into this part. Here's an example that shows the index is still present days after the event has been cleared and is the expected results. They should not drop to Failed when nothing is triggered.

Endpoint Security I trigged with Minikatz and now the index was recreated and the monitor shows it good. It does NOT mean that an event was triggered in the last few minutes.
Malware -Prevented- Endpoint Security is endgame-* index which is not created until something actually would trigger it. I for one will live with Failed until we can figure out how to create the index again without actually running malware in my network. So far I haven't figured out how to get the endpoint agent to push the index and it's mostly from lack of trying honestly.

1 Like