# How to remove lines with specifics information from text files

**URL:** <https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479>\
**Category:** Logstash\
**Created:** [March 22, 2019, 10:58am UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479 "2019-03-22T10:58:30Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [March 22, 2019, 10:58am UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479/1 "2019-03-22T10:58:31Z")

</div>

Hello,

I'm using filebeat to redirect dns log to logstash, then elasticsearch.

Here my configuration :

```
filter {
if "windows_dnslog" in [tags]
{
    grok {
     match => { "message" => "%{DATESTAMP:dns_time} %{GREEDYDATA:dns_thread}%{SPACE}%{WORD:dns_context} %{GREEDYDATA:dns_id} %{GREEDYDATA:dns_protocol} %{GREEDYDATA:dns_sr_indicator} %{IP:dns_remote_ip}%{SPACE}%{WORD:dns_xid} ?(%{WORD:dns_response}|%{SPACE:dns_response})%{SPACE}%{DATA:dns_opcode} \[%{GREEDYDATA:dns_responsecode}\] %{WORD:dns_questiontype}%{SPACE}%{GREEDYDATA:dns_question_name}" }
   }

}
}

filter {
if "windows_dnslog" in [tags]
{
mutate {
gsub => [
  # Remove leading (n)
  "dns_question_name", "^\(\d+\)", "",
  # Remove trailing (n)
  "dns_question_name", "\(\d+\)$", "",
  # Replace inner (n)
  "dns_question_name", "\(\d+\)", "."
]
 }
}
}

```

It works fine, but I have a lot of information I don't want, like [www.google.com](http://www.google.com), [www.facebook.com](http://www.facebook.com), so I want to remove those lines.  
I already exclude some domain with yml configuration file on my servers but I want to centralize.

What's is the best way with logstash ? multiline, drop or another filter ?

And is it possible to place all domains I don't want in a text file and the filter use this text file to remove the lines ?

Thank you for your answers !

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 22, 2019, 12:39pm UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479/2 "2019-03-22T12:39:59Z")

</div>

kindly share your log file sample, without that we cannot help you out of this.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 22, 2019, 12:56pm UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479/3 "2019-03-22T12:56:56Z")

</div>

> [@Ludovic9](#):
>
> And is it possible to place all domains I don't want in a text file and the filter use this text file to remove the lines ?

You could do this using a translate filter with the fallback option. You would drop all events unless the translation is the fallback value.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [March 22, 2019, 1:05pm UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479/4 "2019-03-22T13:05:00Z")

</div>

Hi,  
here an exemple.

On my windows server, I have a log file with a lot of line like this :

```
22/03/2019 13:55:30 0E28 PACKET 0000007F67E71C00 UDP Rcv 172.20.18.26 92d0 Q [0001 D NOERROR] A (3)www(8)facebook(3)com(0)
22/03/2019 13:55:30 0E28 PACKET 0000007F67E71C00 UDP Rcv 172.20.18.26 92d0 Q [0001 D NOERROR] A (3)www(8)facebook(3)com(0)
22/03/2019 13:56:32 0E28 PACKET 0000007F628781B0 UDP Rcv 172.20.18.38 3b1a Q [0001 D NOERROR] A (3)www(4)bing(3)com(0)

```

all the lines are sent to logstash.

The filters I used is in my first message.  
I use grok to parse the line and add some fields.

I want to drop the lines containg" (3)www(4)bing(3)com(0)" so I have done this configuration :

```
filter {
if "windows_dnslog" in [tags]
{
    grok {
     match => { "message" => "%{DATESTAMP:dns_time} %{GREEDYDATA:dns_thread}%{SPACE}% {WORD:dns_context} %{GREEDYDATA:dns_id} %{GREEDYDATA:dns_protocol} %{GREEDYDATA:dns_sr_indicator} %{IP:dns_remote_ip}%{SPACE}%{WORD:dns_xid} ?(%{WORD:dns_response}|%{SPACE:dns_response})%{SPACE}%{DATA:dns_opcode} \[%{GREEDYDATA:dns_responsecode}\] %{WORD:dns_questiontype}%{SPACE}%{GREEDYDATA:dns_question_name}" }
   }
    if [dns_question_name] == "(3)www(4)bing(3)com(0)" {
            drop {}
      }

   }
  }

```

I dont' want to repeat X drop filter for X domains I don't want, so it's possible to put all domains in a text file and logstash uses a kind of loop to know the domain to be drop ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2019, 1:05pm UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479/5 "2019-04-19T13:05:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.