I'm having a log file, which actually has INFOs' and ERRORs. So I tried to match only the needful INFOs by using the grok filter. So this is how my log lines look like. Few of them from the file.
And this is how my grok look like in my logstash conf:
The pattern seems to be working fine. I could provide the pattern if required.
I've got two questions. One is I wanted only the grok matched lines to be sent to the index, and prevent Logstash from indexing the non-matched ones and the second is to prevent Logstash from showing the message in every single ES record.
I tried using the overwrite as such under the match but still no luck:
overwrite => [ "message" ]
All in all what I need to see in my indice are the messages (app_message, response_message from the above match), which should match the above two conditions. Where as now, all the lines are getting indexed.
Is it possible do something like this? Or does Logstash index all of them by default?
Where am I going wrong? Any help could be appreciated.
Does the message that slipped through have a _grokparsefailure tag? If no, the grok filter was successful. Is it app_message or response_message that's populated with data? That'll tell us which expression that matched.
I couldn't remove the message since I was using the message for filtering purposes.
So I had to throw in a if condition, for all the messages which didn't match and drop them.
if "<<<<< API Request" in [message] {
drop { }
}
The above method somehow satisfied my need but then has been a pain now, since I'm having too much of ifs at the moment trying to match and drop the unwanted lines.
Am I going wrong somewhere? Or how can I make it more efficient?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.