Unable to read filebeat output in Logstash

Elastic Stack Logstash
1

Dear Team,

I tried to read the filebeat output in logstash but am getting error.

my Logstash conf file

input {
beats {
port => 5044
}
}

filter{

if[message] =~ /^\[.*/ {
	grok {
		match => ["message" ,'%{SYSLOG5424SD:cli_ipv6}, %{URIHOST:webserver_ip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:appserver} %{NOTSPACE:id5}']
	}
}
	
else{
	grok {
		match => ["message", '%{URIHOST:cli_ipv4} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:appserver} %{NOTSPACE:id5}']
	}
}	
	
mutate {
	convert => {
		"responsecode" => "integer"
		"bitstransfer" => "integer"
		}
	}

}

output {

if "_grokparsefailure" in [tags] {

	file { 
	"path" => "/app/Exports/grok_failures%{+yyyy-MM-dd}.txt" 
	}
} 

else {
	elasticsearch{
		hosts => "x.x.x.x"
		index => "myindex"
	}
}

}

Error file: gorkparsefailure

{"@timestamp":"2019-01-11T06:35:27.722Z","offset":4589642,"input":{"type":"log"},"host":{"name":"myhost"},"source":"/my/log/path/access_log.2019-01-02-12_00_00","beat":{"hostname":"myhost","version":"6.5.4","name":"myhost"},"@version":"1","prospector":{"type":"log"},"tags":["beats_input_codec_plain_applied","_grokparsefailure"],"fields":{"log_type":"accesslog"},"message":"10.10.10.10:- - - - [02/Jan/2019:12:04:46 +0530] "POST /my/api/loguploader HTTP/1.1" 201 - - 296446 - app:9080 -"}

{"@timestamp":"2019-01-11T06:35:27.722Z","offset":4590317,"input":{"type":"log"},"host":{"name":"myhost"},"source":"/my/log/path/access_log.2019-01-02-12_00_00","beat":{"hostname":"myhost","version":"6.5.4","name":"myhost"},"@version":"1","prospector":{"type":"log"},"tags":["beats_input_codec_plain_applied","_grokparsefailure"],"fields":{"log_type":"accesslog"},"message":"20.20.20.20:- - - - [02/Jan/2019:12:04:46 +0530] "GET /my/api/clientLogProfile/mydomain HTTP/1.1" 200 28 - 1640 - app2:9080 -"}

{"@timestamp":"2019-01-11T06:35:27.719Z","offset":4552031,"input":{"type":"log"},"host":{"name":"myhost"},"source":"/my/log/path/access_log.2019-01-06-19_30_00","beat":{"hostname":"myhost","version":"6.5.4","name":"myhost"},"@version":"1","prospector":{"type":"log"},"tags":["beats_input_codec_plain_applied","_grokparsefailure"],"fields":{"log_type":"accesslog"},"message":"20.20.20.20:- - - - [06/Jan/2019:19:43:52 +0530] "POST /my/api/adapters/ActivationEncoded/rract04 HTTP/1.1" 200 2898 - 76929 - app:9080 -"}

{"@timestamp":"2019-01-11T06:35:27.723Z","offset":4591436,"input":{"type":"log"},"source":"/my/log/path/access_log.2019-01-02-12_00_00","host":{"name":"myhost"},"beat":{"hostname":"myhost","version":"6.5.4","name":"myhost"},"@version":"1","prospector":{"type":"log"},"tags":["beats_input_codec_plain_applied","_grokparsefailure"],"fields":{"log_type":"accesslog"},"message":"[2405:204:5387:6a0c:692d:7cf5:267f:fd0b], 10.10.10.10:- - - - [02/Jan/2019:12:04:46 +0530] "POST /my/api/loguploader HTTP/1.1" 201 - - 2031 - app290:9080 -"}

2

conf file is working properly with same logs in local.
input {

file{
path => "/my/log/path/access_log.2018-11-21"
start_position => "beginning"
}
}

Could anyone help me. where i did the mistake?

3

Dear All,

Sorry for my mistake. It's solved.

Issue was

2405:204:5387:6a0c:692d:7cf5:267f:fd0b], 10.10.10.10:- - - - [02/Jan/2019:12:04:46 +0530] "POST /my/api/loguploader HTTP/1.1" 201 - - 2031 - app290:9080 -

Earlier:
%{SYSLOG5424SD:cli_ipv6}, %{URIHOST:cli_ipv4} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} .......

Fixed:
%{SYSLOG5424SD:cli_ipv6}, %{URIHOST:cli_ipv4}:- %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} .......

Sorry for my mistake.

Thanks,
Sripal

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.