Logstash include filter

[tharunkumar (tharun)]

Hi
i need to send the logs of particular lines to logstash .i have tried by using include filter is not working .can u provide me the solution
Eg:
2022-06-30 00:00:07 10.32.13.12 POST /maruvayaparpaidhee-cug/MF/Mfajaxapi.aspx - 443 - 66.249.79.241 Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/102.0.5005.115+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) https://www.edelbusiness.in/maruvayaparpaidhee-cug/MF/SectorMore.aspx?shcd=35281 200 0 0 9893 797 468
2022-06-30 00:00:17 10.32.13.12 GET /maruvayaparpaidhee-cug/tools/charting.aspx co_code=28145&Exchange=NSE 80 - 66.249.79.250 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 149857 430 2140
2022-06-30 00:00:17 10.32.13.12 GET /robots.txt - 80 - 129.146.158.17 AddThis.com+(http://support.addthis.com/) - 404 0 2 1766 165 234

in this above log lines i need to push nly /robots.txt line?.can anyone provide me the solution

[sudhagar_ramesh]

Hello @tharunkumar

You can try the below code which would send the lines which has only /robots.txt

input{
<your input config>
}
filter
{
grok
{
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{IP:clientip} %{WORD:verb} %{URIPATH:response} %{GREEDYDATA:otherdetails}"}
}
}

output
{
if [response] == "/robots.txt"
{
<your-output-logstash-conf>
}

Keep Posted how it goes!!! Thanks!!!

[Rios]

@sudhagar_ramesh has made if you want to search on web root.

Another option is to search for robots.txt in url part and drop messages which doesn't contain robots.txt word

input {

  generator {
        lines => [
          "2022-06-30 00:00:07 10.32.13.12 POST /maruvayaparpaidhee-cug/MF/Mfajaxapi.aspx - 443 - 66.249.79.241 Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/102.0.5005.115+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) https://www.edelbusiness.in/maruvayaparpaidhee-cug/MF/SectorMore.aspx?shcd=35281 200 0 0 9893 797 468",
          "2022-06-30 00:00:17 10.32.13.12 GET /maruvayaparpaidhee-cug/tools/charting.aspx?Robots.txt&co_code=28145&Exchange=NSE - 80 - 66.249.79.250 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 149857 430 2140",
          "2022-06-30 00:00:17 10.32.13.12 GET /robots.txt - 80 - 129.146.158.17 AddThis.com+(http://support.addthis.com/) - 404 0 2 1766 165 234"
        ]
        count => 1
  }

} # input

filter {

    grok {
	  match => { "message" => "%{DATA:date} %{TIME:timestamp} %{IPORHOST:serverip} %{WORD:verb} %{URIPATHPARAM:url} %{DATA:port:int} %{POSINT:port:int} - %{IPORHOST:clientip} %{DATA:ua} %{NOTSPACE:referer} %{NUMBER:response:int} %{NUMBER:subresponse:int} %{NUMBER:scstatus:int} %{POSINT:bytessent:int} %{POSINT:bytesrsvd:int} %{NUMBER:timetaken:int}" 
	  }
	}

	#if [message] =~ /\b(?i)robots.txt(?-i)\b/ {
	if [url] =~ /\b(?i)robots.txt(?-i)\b/ {
	 mutate{
	  add_field => {"filter" => "robots"} 
	 }
	}
	else {
	drop { }	 
	}

} #filter

output {
  
    stdout {
        codec => rubydebug{}
		id => "debug" 
    }
} # output