I'm working on a PoC with ELK to replace our existing SIEM product. I've used ELK in the past to pull in log data from filebeat with great success. One thing I always struggled with was schema/mapping between different sources. I stumbled across Elastic Common Schema and this looks to be exactly what I need.
I understand it's in beta. Is there a list of suggested ways to map fields from Winlogbeat, Filebeat, etc into the ECS format? I found this, but it's very vague. I've resorted to a spreadsheet to attempt to map this out manually, but I'm hoping there's something out there already to help get me going. I'm also a little concerned that some of the fields in ECS are "reserved".
Any help with ECS would be appreciate. I have registered for the ECS webinar next week, but I'm not that patient. If this isn't the right area to ask, let me know. Thanks.
Yes, it looks like ECS should help you accomplish your goal. Note that the "Use cases" section in the ECS repo is quite out of date, so you should ignore it for now. We'll be making a big push to get better ECS documentation in time for the release of 7.0.
There's a few ways you can start migrating to ECS.
The upcoming release of the Elastic Stack 7.0 will come with a full set of Beats that are migrated to ECS. If you plan on upgrading to 7 in short order, you may want to wait for that, and get most of this migration work done for you automatically. If you're using the Beats templates and example dashboards, they will also be migrated for you, mainly using Elasticsearch field aliases. In other words, if you had visualizations that depended on nginx.access.remote_ip, Filebeat 7's index template will have the field nginx.access.remote_ip be an alias to source.ip, ensuring all of your visualizations just keep working. After the stack upgrade, you can then adjust all of your visualizations as time permits, to directly refer to the ECS fields.
If you'd rather not wait for 7.0, you can start migrating to ECS right away. Being a Logstash user, I assume you're comfortable managing your own index templates. This is all that's required in order to start using ECS with Elastic Stack 6.x. I say this because you can't use the 6.x Filebeat index templates, as it has a source field of type keyword, whereas ECS has source as an object, with multiple fields instead.
But if you create your own index template to match ECS (you can start from the template.json in the ECS repo), then you're good to start creating indices with it, and start benefiting from these common names.
One last note: you'll undoubtedly have fields that don't map to ECS at some point. Contrary to other schemas, with ECS you're free to define more fields in your template, that aren't specified in ECS. You should try to nest them in such a way that won't conflict with future versions of ECS. If you nest your fields under your organization name (e.g. acme.x_forwarded_for) or nest them under a product name (nginx.x_forwarded_for) you should be good.
Thanks for the info Mat. Is there an expected release date for version 7? I'm a do to replace our current system prior to July. I'm trying to get something set up soon as I have co-workers that will need to play with this as well. Plus I'm really excited to get rid of our current solution.
I have loaded the ECS template. Would it make more sense to map the fields from Beats in Logstash rather than using a processor on the Beats side? I feel like that would be easier to maintain (we'll have 100's of Beats clients).
I'm not too familiar with aliasing the fields in ES. I'll have to look into that since it seems like it would give me some flexibility if I choose to move fields around.
Great timing, the first 7.0 beta release is fresh out of the oven. Head over here to grab it This should let you and your team start experimenting with all this. As for an official release version, unfortunately I can't give you an exact date. But let's say approximately between now and your July go live you'd get the first official 7.0 release.
I forgot to mention: Kibana 6.7 has a migration assistant that's ECS-aware and can help you through the process of migrating an existing use case, preparing for the 7.0 upgrade.
We will also have a blog post specifically about migrating options. It should be published in the next few weeks (following the ECS intro blog post and webinar).
For your question about where you should be mapping your fields. The Beats modules actually come with Elasticsearch ingest node pipelines, the conversion doesn't happen in the agent itself. You can check out an example ingest pipeline here. If you prefer Logstash go for it, though
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
If this isn't the right area to ask, let me know. Thanks.