Field contains CSV want to extract to array of strings

Hello all,

I am working on migrating from Splunk to ELK for my Palo Alto Networks firewall logs using the pre-built processor in Filebeat. The pipeline is Syslog > Filebeat > Elastic. The PANW logs contain two separate fields for URL Category: one with just a single value (which is extracted in the stock Filebeat input.yml as a string) and another one which contains a quoted string which is a list of comma separated categories (e.g. "music, streaming-services,low-risk"). I would like to extract this list into a field with multiple values (similar to how the Tags field can have multiple values). Is it possible to do this in Filebeat?


Hi ian 351c

This is done using the ingest processor in Elasticsearch that receives the data from filebeat.

If you migrating from splunk I would recommend to get in touch with us using the contact form. You will receive a lot of help that way:

That works perfectly, thanks! Now the trick is to get my visualizations to not show super common values for this field. For instance, from my example above: "music, streaming-services,low-risk". I now get all three values in my field, but I want to exclude "low-risk" from my visualizations, since that's about 90% of my events, but not very useful information. I tried using a filter, but of course, that excludes the entire event rather than just removing "low-risk" from the visualization (a Lens in this case). How do I go about excluding a single value for a field from a visualization without discarding the entire event?


Edit: It looks like I can use a "classic" aggregation based visualization to do this, as it includes the ability to exclude a value from the visualization rather than the underlying query. I'd rather do this with a lens, but it doesn't appear to have that ability...

I forwarded your feedback to the lens team. They will take care to implement it.
Thanks for raising it.

Thanks Felix. I appreciate that!