Hi there, I am trying to create some filter on logstash based on fields name example
filebeat.yml
type: log
/var/log/httpd/*log
type: log
/var/log/*log
/var/log/messages*
so I do have to different fields here
and I'm trying to execute this filter
10-filter.conf
filter {
if [fields][app_id] == "apache_httpd" {
grok {
match => { "message" => "%{IP:client}" }
add_field => [ "read_es", "%{@timestamp}" ]
}
}
else if [fields][app_id] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} \[%{DATA}\] %{DATA} %{WORD:type} \"userText\":\"%{DATA:userText}\",\"prduction\":%{NUMBER:prduction:int},\"version\":%{NUMBER:version:int},\"bool\":%{DATA:bool},\{\"geoPoint\":\{\"location\":\"%{DATA:location}\",\"ip\":\"%{IP:ip}\",\"latitude\":%{BASE10NUM:latitude},\"longitude\":%{BASE10NUM:longitude},(\"optionalField\":%{BASE10NUM:optionalField},)?\}\}" }
add_field => [ "read_es", "%{@timestamp}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
mutate
{
remove_field => [ "message" ]
remove_field => [ "syslog_timestamp" ]
convert => ["bool","boolean"]
rename => { "ip" => "[geoPoint][ip]" }
rename => { "latitude" => "[geoPoint][latitude]" }
rename => { "longitude" => "[geoPoint][longitude]" }
rename => { "location" => "[geoPoint][location]" }
}
}
}
the log format of /var/log/httpd/*.log is this192.168.9.1 - - [28/Feb/2020:10:37:44 +0000] "GET / HTTP/1.1" 403 4006 "-" "curl/7.64.1"
and according to the filter that I've implemented, I should get only the client IP as the message, but for some reason, I'm still getting the full output, maybe I'm missing something or I misunderstood how a filter should be used.
Any help will be much appreciated.
Hi Carmine,
can you post here couple of outputs (apache_http and syslog) of a simple pipeline without any filter applied to the events? Just take as input what you're already taking, and spit it in stdout.
Thanks
Hi @Fabio-sama , thanks for the reply, do you mean something like this?
output {
and this is the output that I get with journalctl -u logstash
apache_httpd
Feb 28 12:02:01 logstash01 logstash[6150]: "@timestamp " => 2020-02-28T12:02:27.812Z,@version " => "1",@timestamp " => 2020-02-28T12:02:27.812Z,
Syslog
Feb 28 12:01:01 logstash01 logstash[6150]: "@timestamp " => 2020-02-28T12:01:27.814Z,@version " => "1",
Let me know if its enough, thanks!
Badger
February 28, 2020, 5:25pm
4
carmine.fabrizio:
[fields][app_id]
You set fields_under_root true, so your events have [app_id], and not [fields][app_id].
system
March 30, 2020, 1:05pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.