Mutate/remove fields with dot caracters

Elastic Stack Logstash
1

Hi all!
We use the input cef codec because we have some arcsight connectors that we want to send their events to our ELK platform. CEF is cool but we would like to remove a lot ot fields that are not necessary for the dashboard we want to make. Of course mutate/remove field is used here but it seems that a lot of cef field have the "." character in them and for what I have read on some forum, it does not seems possible to use the mutate/remove field function with fields that have special characters.
Can you tell me a good way to manage this?
ex:

filter {
if [cef_vendor] =~ "Fortinet" {
mutate {
remove_field => [ "cef_version", "cef_sigid", "cef_ext.destinationZoneURI", "cef_ext.sourceZoneURI", "cef_ext.sourceZoneURI.keyword", "cef_ext.sourceZoneURI.keyword" ]
}
}
}

Here, the fields "cef_version" and "cef_sigid" are removed by the filters but not cef_ext.*

Thanks!

2

Do you actually have dots in the field names or do you have a hierarchy of fields and subfields? What do your events really look like?

3

Sorry, let me put mine 2 cents as the issue I'm currently researching could be the same. I have a json filter which parses json sent by filebeat, so after json filter I have data like

eventData.userIdentity.currency EUR
eventData.userIdentity.ip 13.46.17.118
eventData.userIdentity.language en

The problem is that 1) geoip filter do not work with field eventData.userIdentity.ip 2) mutate { copy => { "eventData.userIdentity.ip", "userIp" } } do not work , even 3) mutate { copy => { "beat.name", "beatNameCopy" } } do not work, but 4) mutate { copy => { "host", "hostCopy" } } works ,

so I'm assuming this is a global issue with fields with dots.

4

Hi,
2 things (and it's good news! :slight_smile:

  • One of my colleague found the solution (in fact, consider those fields as nested arrays ). You can manage them using the syntax [part1][part2]..[partx].
    ex:
    remove_field => [ "cef_version", "cef_sigid", "[cef_ext].[destinationZoneURI]", "[cef_ext].[sourceZoneURI]", "[cef_ext][sourceZoneURI.keyword]", "[cef_ext][sourceZoneURI.keyword]" ]
  • The other good news is that all my events come from an ArcSight connectors and with the latest logstash version (or cef plugin), now the field are manged using the same ArcSight syntax. Ex: sourceZoneURI instead of [cef_ext].[sourceZoneURI] which make the process more easier.

Thanks! Hope this can help.

5

Makes sense, thanks for the hint!

6

so I'm assuming this is a global issue with fields with dots.

The syntax for nested fields is described here: Accessing event data and fields | Logstash Reference [8.11] | Elastic

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.