Logstash sending /var/log/messages to Elastic

[Robert_Fyffe]

Hi Everyone,

I am using the following 6.x stack:

Filebeat -> Logstash -> Elastic -> Kibana

I am trying to configure filebeat/logstash to send a specific logfile only to Elastic. Right now it appears to be sending everything.

When I look in Kibana I see most messages from /var/log/messages.

Filebeat config:

filebeat.inputs:

• type: log
  enabled: true
  paths:
  #- /var/log/*.log
  • /var/log/nginx/app.access.log
    #- c:\programdata\elasticsearch\logs*

output.logstash:
hosts: ["localhost:5044"]
index: logstash

Logstash Config:

input {
beats {
port => "5044"
}
}

filter {
grok {
match => {"message" => '%{IP:client} - %{USERNAME:username} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{DATA: http_x_forwarded_for} %{DATA:gzip_ratio} rt=%{NUMBER:request_time} uct=%{NUMBER:upstream_connect_time} uht=%{NUMBER:upstream_header_time} urt=%{NUMBER:upstream_response_time} %{NUMBER:user_id}'}
}
grok {
match => {"request" => "/dataset/%{NUMBER:dataset_id}"}
}
kv {
source => "request"
field_split => "&?"
transform_key => "lowercase"
}
if "/search" in [request] and [q] {
mutate {add_field => {"search_action" => "search" }}
}
if [request] =~ "/dataset/%{NUMBER:dataset_id}" {
mutate {add_field => {"dataset_id" => dataset_id}}
}
mutate {
convert => {
"user_id" => "integer"
"dataset_id" => "integer"
}
}
}

output {
elasticsearch {
codec => "json"
hosts => ["127.0.0.1:9200"]
}
stdout { codec => rubydebug }
}

With this configuration I should only be getting /var/log/nginx/app.access.log data sent to elastic?

Within Kibana I see all activity from multiple log files:

/var/log/messages
/var/log/secure

Any ideas why this happening as I can't really find any solutions to this?

Thanks

[AquaX (Andreas Helmer)]

Have you restarted beats since making that configuration change?