Custom fields in ECS

[phruby]

Hello,
for our use case is sometimes important to include fields not present in Elastic Common Schema. There are two main cases:

1. Fields that conceptually belong an existing ECS fieldset, but are not included to ECS 1.1; an example is "protocol_number", an integer that a firewall assigns to a network protocol. Would it be OK to map this field to network.protocol_id, although this field does not exist in the ECS network fieldset? Another example is "service_id" a string identifying a service that a firewall finds on the destination port of the connection. Would it be OK to map this field to destination.service, although this field does not exist in the ECS destination fieldset?

2. Fields that conceptually do not belong to any existing ECS fieldset, such as "Operation Number", an integer identifying an operation performed by a firewall administrator. Would you recommend to (i) keep this field as is, just renamed using ECS guidelines, i.e. operation_number, (ii) or use a custom prefix, such as audit.operation_number?

Thanks,
Pavel

[webmat (Mathieu Martin)]

Hi @phruby,

I'm thinking the protocol number you're looking for is network.iana_number?

1. There's always a risk in nesting new fields inside ECS field sets.
   1.1. If you add a field and ECS subsequently adds a field of the same name, then you'll have to adjust your pipeline.
   1.2. Or perhaps you'll independently add a field, and ECS will add exactly the same field with the same meaning & datatype. If that's the case, this is of course a non-issue It's actually how many of the new ECS fields currently come to life. Someone adds them in their pipeline, or Beats adds it in a module, it's suggested in the ECS repo via an issue or PR, and it officially makes its way into ECS.

2. If you want to add custom fields in custom field sets, I would suggest avoiding very common names, that risk getting into ECS. A good example of a field set that may be added eventually is audit, given Elastic's efforts in the security space nowadays

Here are a few ways to think about adding custom fields in your events, that minimize the risk of conflicts:

• Under labels is always the first place you should think to add data, if it's only one or two keyword fields.
• ECS avoids proper names (e.g. brands, companies, technologies) as much as possible
  • To be bulletproof, you can nest your custom fields under your company name, then nest per use case. E.g. acme.firewall.*,acme.web.*, etc.
  • You can also directly nest under a technology name, but there's a risk of conflict at query time, with third party solutions, if you query across indices. E.g. apache.access.* is a custom field used by Beats
• You can also completely avoid the risk of conflict by making use of capitalization. ECS has a guideline to always use lowercase key names, so you could use Audit.* and you'll never conflict with ECS.

[phruby]

Thanks for very useful and comprehensive guidelines - they definitely allowed me to move forward.
Yes, network.iana_number indeed matches the "proto" attribute (protocol number) of the Check Point firewall.
-Pavel