@PublicName thank you for the detailed info.
I'll focus on the Event Filters in this post and pull in others who are more knowledgable on the Agent upgrade failures.
First, a few responses:
I have not updated the agents as of yet. As I'll keep saying please do not make agents depended on subversion :-).
Apologies on this, I should have been more clear. It's certainly not required to upgrade Agents every time, I was asking for debugging purposes.
Is it possible to have Elastic drop the events as well not just the agent.
As far as I know there is no functionality like this currently. Are you imagining some type of background process that deletes documents already written to ES based on a set of filters?
Regarding Event Filters - judging from the successful Policy Responses on your first post, it looks like they are successfully downloading your Event Filter list. Note that these Event Filters only apply to Endpoint events. Extending Event Filter functionality to the other subprocesses shipped by the Agent is something that we've discussed, but it's not currently implemented.
Can you confirm that the new Events coming in that should be filtered out come from Endpoints? In addition, they must be classified as event.kind: event to be picked up by filters. You can do this by confirming that the documents you see in logs-* contain agent.type: endpoint and event.kind: event.
There should be two sections in the doc:
"agent": {
"id": XXXXX,
"type": "endpoint",
"version": "7.14.1"
},
...
"event": {
...
"kind": "event",
"module": "endpoint",
...
},
Alternatively, you can do this through the UI by going to "Security > Hosts > Events" page and adding a filter to the filter bar like this: agent.type : "endpoint" and event.kind : "event" and event.code : "4690".
Similar to this:
If you confirm that these documents are coming from the Endpoint and are Events, then it's possible the artifacts are not being created correctly. To check, you could go check the Policy yml of the Agent Policy you are using. If you open it up in the UI, you should see a section for the eventfilters list.
Similar to this:
Let me know what you find and I can check on a workaround.
For the security event log side here are my most common dropped either with event filter or from the legacy beats. 5156, 5157, 5158, 4658, 4656, 4690, 5152, 5447, 5152, 5154, 4663, 4703 the bulk of which are windows firewall events. Why? Because they are absolutely useless in my environment. 90% of it is broadcast traffic and we already network monitoring in place that watches the rest.
