Hello all. I'm new to Logstash/ELK and a pretty weak coder. I'm trying to tidy up a filter which uses Grok and running into an error. Basically I've got a given log file and a filter that works but after adding a couple of addidtional Grok statements (to mark two specific events for use with Elapsed by adding tags to them) it ends up marking about 98% of events as "_grokparsefailed". I know why...because 98% of events don't contain info such that they should be marked/get tags added.
Anyway...I thought the way to clean this up would be to move all of my Grok statements into a single block featuring "else" conditionals. Essentially the flow looks like "Mark for event 1? -> else -> Mark for even 2? -> else -> Final Grok statement -> out to rest of filter". It looks to me like it should work. But when I run "logstash configtest" I keep getting an error which indicates it's expecting something "Expected one of #, => at line X, column Y (byte Z) after filter {". Which brings me back to that whole I'm not a very capable coder bit.
If anyone could take a fast look at the paste below and point out what I'm doing wrong I'd really appreciate it. FWIW I have searched and read the docs...I've been stuck on this for about 4 hours. TIA!
Thank you for the quick reply Magnus, I think I see what you're suggesting and I'm about to go try it. May I ask though, why would I want a _grokparsefailure?
One last snag, if I get through this it's done. The first two Grok statements are now working properly, adding the tags I want added (for use with Elapsed). The problem is that they're also being sent through the final Grok statement and ending up with duplicate information in certain tags. For example, I have a tag named "slotID", in the desired state that has a value but only has it once. The events that do not match the first two Grok statements (and thus fall to the third for processing) are in that state. But all of the events that match one of the first two Grok statements have multiple entries in slotID (and other tags as well).
I'm trying to keep those events from going through the third Grok statment with this:
if "firstImage" or "launchReq" not in [tags] {
grok {
break_on_match => true
match => { "message" => ["^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>[^\;]*$)",
"^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>%{GREEDYDATA});\s(?<spaJson>%{GREEDYDATA})"
]
}
}
}
But based on my results that doens't seem to be working. Any ideas?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.