# Logstash include filter

**URL:** <https://discuss.elastic.co/t/logstash-include-filter/308853>\
**Category:** Logstash\
**Created:** [July 5, 2022, 4:39am UTC](https://discuss.elastic.co/t/logstash-include-filter/308853 "2022-07-05T04:39:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tharunkumar](https://avatars.discourse-cdn.com/v4/letter/t/f05b48/32.png) [@tharunkumar](https://discuss.elastic.co/u/tharunkumar)\
**Post date:** [July 5, 2022, 4:39am UTC](https://discuss.elastic.co/t/logstash-include-filter/308853/1 "2022-07-05T04:39:10Z")

</div>

Hi  
i need to send the logs of particular lines to logstash .i have tried by using include filter is not working .can u provide me the solution  
Eg:  
2022-06-30 00:00:07 10.32.13.12 POST /maruvayaparpaidhee-cug/MF/Mfajaxapi.aspx - 443 - 66.249.79.241 Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/102.0.5005.115+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) [https://www.edelbusiness.in/maruvayaparpaidhee-cug/MF/SectorMore.aspx?shcd=35281](https://www.edelbusiness.in/maruvayaparpaidhee-cug/MF/SectorMore.aspx?shcd=35281) 200 0 0 9893 797 468  
2022-06-30 00:00:17 10.32.13.12 GET /maruvayaparpaidhee-cug/tools/charting.aspx co\_code=28145&Exchange=NSE 80 - 66.249.79.250 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 149857 430 2140  
2022-06-30 00:00:17 10.32.13.12 GET /robots.txt - 80 - 129.146.158.17 AddThis.com+([http://support.addthis.com/](http://support.addthis.com/)) - 404 0 2 1766 165 234

in this above log lines i need to push nly /robots.txt line?.can anyone provide me the solution

---

<div class="post-metadata">

**Author:** ![sudhagar\_ramesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhagar_ramesh/32/105673_2.png) [@sudhagar\_ramesh](https://discuss.elastic.co/u/sudhagar_ramesh)\
**Post date:** [July 5, 2022, 6:09am UTC](https://discuss.elastic.co/t/logstash-include-filter/308853/2 "2022-07-05T06:09:41Z")

</div>

Hello @tharunkumar

You can try the below code which would send the lines which has only /robots.txt

```auto
input{
<your input config>
}
filter
{
grok
{
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{IP:clientip} %{WORD:verb} %{URIPATH:response} %{GREEDYDATA:otherdetails}"}
}
}

output
{
if [response] == "/robots.txt"
{
<your-output-logstash-conf>
}

```

Keep Posted how it goes!!! Thanks!!!

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [July 5, 2022, 7:02am UTC](https://discuss.elastic.co/t/logstash-include-filter/308853/3 "2022-07-05T07:02:34Z")

</div>

@sudhagar_ramesh has made if you want to search on web root.

Another option is to search for robots.txt in url part and drop messages which doesn't contain robots.txt word

```auto
input {

  generator {
        lines => [
          "2022-06-30 00:00:07 10.32.13.12 POST /maruvayaparpaidhee-cug/MF/Mfajaxapi.aspx - 443 - 66.249.79.241 Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/102.0.5005.115+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) https://www.edelbusiness.in/maruvayaparpaidhee-cug/MF/SectorMore.aspx?shcd=35281 200 0 0 9893 797 468",
          "2022-06-30 00:00:17 10.32.13.12 GET /maruvayaparpaidhee-cug/tools/charting.aspx?Robots.txt&co_code=28145&Exchange=NSE - 80 - 66.249.79.250 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 149857 430 2140",
          "2022-06-30 00:00:17 10.32.13.12 GET /robots.txt - 80 - 129.146.158.17 AddThis.com+(http://support.addthis.com/) - 404 0 2 1766 165 234"
        ]
        count => 1
  }

} # input

filter {

    grok {
	  match => { "message" => "%{DATA:date} %{TIME:timestamp} %{IPORHOST:serverip} %{WORD:verb} %{URIPATHPARAM:url} %{DATA:port:int} %{POSINT:port:int} - %{IPORHOST:clientip} %{DATA:ua} %{NOTSPACE:referer} %{NUMBER:response:int} %{NUMBER:subresponse:int} %{NUMBER:scstatus:int} %{POSINT:bytessent:int} %{POSINT:bytesrsvd:int} %{NUMBER:timetaken:int}" 
	  }
	}

	#if [message] =~ /\b(?i)robots.txt(?-i)\b/ {
	if [url] =~ /\b(?i)robots.txt(?-i)\b/ {
	 mutate{
	  add_field => {"filter" => "robots"} 
	 }
	}
	else {
	drop { }	 
	}

} #filter

output {
  
    stdout {
        codec => rubydebug{}
		id => "debug" 
    }
} # output

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2022, 7:03am UTC](https://discuss.elastic.co/t/logstash-include-filter/308853/4 "2022-08-02T07:03:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.