Elastic Siem external alerts

[abr4xc (Omar Avilez)]

I am trying to set up a new integration for an EDR that is not listed on the Kibana integrations yet, in order to set up the external alerts for that EDR i am adding the event.kind to alert but its not showing up in the external alerts session, i just set the field on the json, do i need to do something else to set the ECS, here is an example of the JSON document.

{
    "info":"None",
    "customer_id":"xxx",
    "created_at":"2022-07-28T13: 02: 50.574Z",
    "severity":"medium",
    "event_service_event_id":"xxx",
    "threat":"PsExec",
    "threat_cleanable":false,
    "when":"2022-07-28T13: 01: 18.000Z",
    "location":"xxx-xxx",
    "id":"xxx",
    "type":"Event: :Endpoint: :Threat: :PuaDetected",
    "description":"PUA detectada: 'PsExec' at 'C:\\ProgramData\\xxx\\xxx\\xxx\\apps\\xxx\\xxx\\xxx'",
    "source":"xxx-xxx\\xxx-xxx",
    "data":{
       "created_at":1659013370574,
       "endpoint_id":"xxx-xxx",
       "endpoint_java_id":"xxx-xxx",
       "endpoint_platform":"windows",
       "endpoint_type":"computer",
       "event_service_id":"xxx",
       "inserted_at":1659013370574,
       "source_info":{
          "ip":"x.x.x.x"
       },
       "threat_id":"62b61e2e88e3b414831e22c5",
       "threat_status":"NOT_CLEANUPABLE",
       "user_match_id":"xxx",
       "user_match_uuid":"xxx"
    },
   "event":{
      "kind":"alert",
      "category":[
         "intrusion_detection",
         "network"
      ]
   },
   "host":{
      "ip":"x.x.x.x",
      "name":"PCxxx"
   }
}

[Mike_Paquette]

Hi @abr4xc, Welcome to our community! Thanks for trying out Elastic Security with your EDR data!

(BTW, we're working on new Elastic Agent integrations for other EDR solutions, so maybe we'll have an Elastic-supplied version for you in the future). If you can share which EDR this event is from, perhaps we can provide some info.

But for now, as you suggest, you need to get your data into an ECS-compliant, or at least ECS-compatible format.

As background, the Elastic SIEM/Security app, including its external alerts views, detection rules, signals, and detection alerts, requires your data to be indexed in an ECS-compliant format. As I think you know, ECS is an open source, community-developed schema that specifies field names and Elasticsearch data types for each field, and provides descriptions and example usage.

It appears that you've already added event.kind:alert and some appropriate values for event.category. So far so good! You've also added host.ip and host.name fields, which is great.

However, I do see a few things that are problematic:

• your event does not seem to have an @timestamp field. I see multiple time fields in your event - you will need to copy one of them to @timestamp. ECS defines three time fields @timestamp, event.created, and event.ingested. Only @timestamp is mandatory. (ECS definition)
• your event's threat field will conflict with ECS threat.* fields.
• your event's source field will conflict with ECS source.* fields.
• There may be other issues, such as conflicts with the Elasticsearch field data types you're using in your index mapping template.

Please check out these general guidelines for creating ECS-compliant data:

1. Each indexed document (e.g., your log, event, etc.) MUST have the @timestamp field.
2. Your index mapping template must specify the Elasticsearch field data type for each field as defined by ECS. For example, your @timestamp field must be of the date field data type, etc.. This ensures that there will not be any mapping conflicts in your indices.
3. The original fields from your log/event SHOULD be copied/renamed/converted to the corresponding ECS-defined field name and data type.
4. Additional ECS fields, such as the ECS Categorization fields SHOULD be populated for each log/event, to allow proper inclusion of your data into dashboards and detection rules.

Here's a simple graphic that I created to help get this point across. It appears your data is still in the red zone, and you need to move it at least to the yellow zone to see your EDR alerts appear in the Elastic Security/SIEM app, and then to green to enjoy all the benefits Elastic Security has to offer. It's a bit of work up front, but we think it's worth it.

Can you share where your log data is coming from? (e.g., some security device? or some host computer?)

[abr4xc (Omar Avilez)]

Thanks for the explanation @Mike_Paquette, i will take a look at the ECS guidelines and make the data compliant. these logs are from Sophos EDR/XDR , the getting all the logs from sophos central and then depending on the log (endpoint/firewall/etc) injesting then into elastic, are you working now with sophos EDR for an integration?

[jamie.hynds]

Hey @abr4xc, I'm the Product Manager responsible for our SIEM integrations. Sophos is certainly on our roadmap, and we're currently working with Sophos on getting access to an instance to use for development. Can you confirm if it's Sophos Central you are collecting logs from (I believe their Intercept X XDR offering also sends logs via Sophos Central)? If so, the pipeline in this issue may provide you with some ECS mappings to get started.

[abr4xc (Omar Avilez)]

Hi @jamie.hynds some field in the ingest template must be updated in order for the external alerts to work and the filebeat json input is not working properly for the sophos json output, i already fix it myself and created a bot for sophos, if you want to i could write it in got and that way contribute to the community.

regards,