Field contains CSV want to extract to array of strings

Elastic Stack Beats
1

Hello all,

I am working on migrating from Splunk to ELK for my Palo Alto Networks firewall logs using the pre-built processor in Filebeat. The pipeline is Syslog > Filebeat > Elastic. The PANW logs contain two separate fields for URL Category: one with just a single value (which is extracted in the stock Filebeat input.yml as a string) and another one which contains a quoted string which is a list of comma separated categories (e.g. "music, streaming-services,low-risk"). I would like to extract this list into a field with multiple values (similar to how the Tags field can have multiple values). Is it possible to do this in Filebeat?

Thanks!

2

Hi ian 351c

This is done using the ingest processor in Elasticsearch that receives the data from filebeat.

If you migrating from splunk I would recommend to get in touch with us using the contact form. You will receive a lot of help that way:

3

That works perfectly, thanks! Now the trick is to get my visualizations to not show super common values for this field. For instance, from my example above: "music, streaming-services,low-risk". I now get all three values in my field, but I want to exclude "low-risk" from my visualizations, since that's about 90% of my events, but not very useful information. I tried using a filter, but of course, that excludes the entire event rather than just removing "low-risk" from the visualization (a Lens in this case). How do I go about excluding a single value for a field from a visualization without discarding the entire event?

Thanks!

Edit: It looks like I can use a "classic" aggregation based visualization to do this, as it includes the ability to exclude a value from the visualization rather than the underlying query. I'd rather do this with a lens, but it doesn't appear to have that ability...

4

I forwarded your feedback to the lens team. They will take care to implement it.
Thanks for raising it.

5

Thanks Felix. I appreciate that!

Ian

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.