# Migrating logstash filters to ECS **URL:** \ **Category:** Logstash\ **Tags:** ecs-elastic-common-schema\ **Created:** [August 5, 2019, 7:04pm UTC](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866 "2019-08-05T19:04:16Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mich-b](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@Mich-b](https://discuss.elastic.co/u/Mich-b)\ **Post date:** [August 5, 2019, 7:04pm UTC](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866/1 "2019-08-05T19:04:16Z")
I want to migrate my custom Logstash pipeline to the Elastic Common Scheme, since I want to get started with the Elastic SIEM UI. In [https://www.elastic.co/blog/migrating-to-elastic-common-schema-in-beats-environments](https://www.elastic.co/blog/migrating-to-elastic-common-schema-in-beats-environments) it is mentioned that migrating custom pipelines will be covered in a future post, however I hope I can already get some input here on this forum. I have a PFSense running which sends its firewall logs to ELK via syslog =\> logstash =\> elasticsearch. I also have Snort enabled on that PFSense which has barnyard sending the logs to ELK via syslog =\> logstash =\> elasticsearch. For the syslog entries to get parsed nicely, I'm using grok patterns, both for the firewall logs as well as the Snort logs. - the firewall log filtering + grok patterns are based on [https://github.com/a3ilson/pfelk](https://github.com/a3ilson/pfelk) - the snort log filtering + grok patterns are custom created The fields are therefore custom labelled in the grok patterns, like this: `%{INT:ids_priority}\]\:? \<%{PFSENSE_IFACE}\> \{PROTO\:%{INT}\} %{IP:src_ip} \-\> %{IP:dst_ip}$",` Now, my question is if it would be good practice to simply migrate to ECS in my logstash filter configuration, for example by changing the `src_ip` to `source.ip` ([https://www.elastic.co/guide/en/ecs/current/ecs-source.html](https://www.elastic.co/guide/en/ecs/current/ecs-source.html)) in the above example? I could then do a search replace in my dashboards for the src\_ip field. Or should I do the mapping higher up the chain (e.g. in elasticsearch)? ---
**Author:** ![webmat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/webmat/32/46191_2.png) [@webmat](https://discuss.elastic.co/u/webmat)\ **Post date:** [August 13, 2019, 1:30pm UTC](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866/2 "2019-08-13T13:30:27Z")
Yes, changing the output field name directly in the grok or in a plugin's attribute for the destination of the plugin output (e.g. GeoIP) would be a great start. Now that you have the field names right, you'll want to look carefully at the field datatypes as well. You can of course start with the documentation for this. But here are two additional pointers: You can look at, and experiment with sample Elasticsearch templates that contain only the ECS fields. You can check them out directly in the git repo here [https://github.com/elastic/ecs/tree/master/generated/elasticsearch](https://github.com/elastic/ecs/tree/master/generated/elasticsearch). You'll likely want to use the appropriate git version tag, not the file in master, which is a development branch. Secondly, ECS formalizes the pattern that most text fields in monitoring use cases are used with the `keyword` datatype for aggregations and exact match searches / filtering, and `text` is rarely used. So in general, text fields are directly the `keyword` datatype, and there's no multi-field named `myfield.keyword`. In other words, where Elasticsearch and the Logstash ES template default to having all text fields set up like this `myfield` == `text` and `myfield.keyword` == `keyword`, ECS will mostly only have `myfield` == `keyword`. If you need full text search on some of these fields, it's perfectly valid to add a multi-field in your template, and end up with the reverse convention: `myfield` == `keyword` (the ECS field) and `myfield.text` == `text` (your custom additional field). ---
**Author:** ![webmat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/webmat/32/46191_2.png) [@webmat](https://discuss.elastic.co/u/webmat)\ **Post date:** [August 13, 2019, 2:46pm UTC](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866/3 "2019-08-13T14:46:19Z")
The two exceptions on the string field datatype are `message` and `error.message`, which are `text` only, at this time. Once again, if you have log sources where you need aggregations on these fields, you can add a custom multi-field using the default ES convention, which would be to add `message.keyword`. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 10, 2019, 2:46pm UTC](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866/4 "2019-09-10T14:46:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.