How to remove lines with specifics information from text files

Hello,

I'm using filebeat to redirect dns log to logstash, then elasticsearch.

Here my configuration :

filter {
if "windows_dnslog" in [tags]
{
    grok {
     match => { "message" => "%{DATESTAMP:dns_time} %{GREEDYDATA:dns_thread}%{SPACE}%{WORD:dns_context}  %{GREEDYDATA:dns_id} %{GREEDYDATA:dns_protocol} %{GREEDYDATA:dns_sr_indicator} %{IP:dns_remote_ip}%{SPACE}%{WORD:dns_xid} ?(%{WORD:dns_response}|%{SPACE:dns_response})%{SPACE}%{DATA:dns_opcode} \[%{GREEDYDATA:dns_responsecode}\] %{WORD:dns_questiontype}%{SPACE}%{GREEDYDATA:dns_question_name}" }
   }

}
}


filter {
if "windows_dnslog" in [tags]
{
mutate {
gsub => [
  # Remove leading (n)
  "dns_question_name", "^\(\d+\)", "",
  # Remove trailing (n)
  "dns_question_name", "\(\d+\)$", "",
  # Replace inner (n)
  "dns_question_name", "\(\d+\)", "."
]
 }
}
}

It works fine, but I have a lot of information I don't want, like www.google.com, www.facebook.com, so I want to remove those lines.
I already exclude some domain with yml configuration file on my servers but I want to centralize.

What's is the best way with logstash ? multiline, drop or another filter ?

And is it possible to place all domains I don't want in a text file and the filter use this text file to remove the lines ?

Thank you for your answers !

kindly share your log file sample, without that we cannot help you out of this.

You could do this using a translate filter with the fallback option. You would drop all events unless the translation is the fallback value.

Hi,
here an exemple.

On my windows server, I have a log file with a lot of line like this :

22/03/2019 13:55:30 0E28 PACKET  0000007F67E71C00 UDP Rcv 172.20.18.26    92d0   Q [0001   D   NOERROR] A      (3)www(8)facebook(3)com(0)
22/03/2019 13:55:30 0E28 PACKET  0000007F67E71C00 UDP Rcv 172.20.18.26    92d0   Q [0001   D   NOERROR] A      (3)www(8)facebook(3)com(0)
22/03/2019 13:56:32 0E28 PACKET  0000007F628781B0 UDP Rcv 172.20.18.38    3b1a   Q [0001   D   NOERROR] A      (3)www(4)bing(3)com(0)

all the lines are sent to logstash.

The filters I used is in my first message.
I use grok to parse the line and add some fields.

I want to drop the lines containg" (3)www(4)bing(3)com(0)" so I have done this configuration :

filter {
if "windows_dnslog" in [tags]
{
    grok {
     match => { "message" => "%{DATESTAMP:dns_time} %{GREEDYDATA:dns_thread}%{SPACE}%   {WORD:dns_context}  %{GREEDYDATA:dns_id} %{GREEDYDATA:dns_protocol} %{GREEDYDATA:dns_sr_indicator} %{IP:dns_remote_ip}%{SPACE}%{WORD:dns_xid} ?(%{WORD:dns_response}|%{SPACE:dns_response})%{SPACE}%{DATA:dns_opcode} \[%{GREEDYDATA:dns_responsecode}\] %{WORD:dns_questiontype}%{SPACE}%{GREEDYDATA:dns_question_name}" }
   }
    if [dns_question_name] == "(3)www(4)bing(3)com(0)" {
            drop {}
      }

   }
  }

I dont' want to repeat X drop filter for X domains I don't want, so it's possible to put all domains in a text file and logstash uses a kind of loop to know the domain to be drop ?

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.