Am I reading it right that the raw JSON coming in is formatted with => instead of :? I've tried both the gsub regex as shown above as well as just =>. Is there a different syntax I should be using with gsub? Or maybe I'm reading the log entry wrong and the raw message is also missing the opening and close script blocks {}?
I believe it would work if you merged those two, provided "[vpn][tunnel_type][tunnel_type]" was listed last, but I do not think logstash provides a guarantee of that. Doing it in two filters should guarantee it.
I scrapped everything that was in the pipeline for this event and started from scratch. I don't know why JSON filter was ever used as the data comes in structured as key/value pairs. It also looks like mutate's rename filter was being invoked to rename the nested duplicate tunnel_type up a level...but that wasn't working for some reason. What I got to work was using ruby to pull the information up a level and then delete the original nested field:
I probably need to put some logic in to ensure that a given field is present before executing one of these ruby filters. However, in the absence of a value, the key/value pair is still present as tunnel_type=.
Regardless, you helped put me on the right path, so much appreciated for the help @Badger
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.