Cisco Umbrella Ingest

[jasonmull]

Hello,

I'm trying to integrate multi-tenant Cisco Umbrella DNS logs into Elastic SIEM and I'm running into a few issues. These logs are located in an AWS S3 bucket. Individual log files are generated as csv and gzipped. In addition, I need to add a field to each record containing customer id, as the logs as they come from Cisco do not have this (customers logs are separated by folder in the same s3 bucket, with the folder name being the customer ID). That being said, I'm trying to figure out the most efficient way to ingest this data and make the field modification.

As of right now, I've got a scheduled task to pull down the files from AWS and gunzip them, and then remove the gz files leaving only CSV. From there, my thought was to use Filebeat to move those logs to Elastic. If I were to append a customer ID to the name of each CSV file, is there a way that I could create and populate a customer ID field in elastic based on the filename?

[tudor (Tudor Golubenco)]

If I were to append a customer ID to the name of each CSV file, is there a way that I could create and populate a customer ID field in elastic based on the filename?

Filebeat stores the source filename in a field (log.file.name). You can then use the dissect processor (either in Beats or Ingest Node) to extract the ID from the filename.

Then, Ingest Node has a CSV processor that you can use if you want to. You will also want to map the fields to ECS, so that they are usable in the SIEM app. I recommend looking at this blog post for the general approach for using Ingest Node and the ecs-mapper tool.