Hey
I am using newest ELK stack (V7) on Ubuntu 16.04 to process my log files from several nginx-servers.
All the access-logs are working pretty fine. Whats not working is the error-log. Only 1% of all lines are visibile in Kibana-UI. And none of them is being parsed at all.
This is my configuration:
Filebeat sends data to logstash, as configured in /etc/filebeat/filebeat.yml):
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
Besides I configured path's to log files in /etc/filebeat/modules/nginx.yml:
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-module-nginx.html
- module: nginx
access:
enabled: true
var.paths:
- /var/nginx/foobar1/logs/access.log*
- /var/nginx/foobar2/logs/access.log*
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: true
error:
enabled: true
var.paths:
- /var/nginx/foobar1/logs/error.log*
- /var/nginx/foobar2/logs/error.log*
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: true
My /etc/logstash/filebeat.conf looks like this:
input {
beats {
port => 5044
client_inactivity_timeout => "600"
} }
filter {
if [event][module] == "nginx" {
if [fileset][name] == "access" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => [ "%{NGINX_ACCESS1}" ] }
remove_tag => ["_grokparsefailure"]
add_tag => ["nginx_access"]
}
mutate {
add_field => { "read_timestamp" => "%{@timestamp}" }
}
date {
match => [ "[nginx][access][datetime]", "dd/MMM/YYYY:H:m:s Z" ]
remove_field => "[nginx][access][datetime]"
}
useragent {
source => "[nginx][access][agent]"
target => "[nginx][access][user_agent]"
remove_field => "[nginx][access][agent]"
}
} else if [fileset][name] == "error" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => [ "%{NGINX_ERROR1}", "%{NGINX_ERROR2}" ] }
remove_tag => ["_grokparsefailure"]
add_tag => ["nginx_error"]
}
mutate {
add_field => { "read_timestamp" => "%{@timestamp}" }
}
date {
match => [ "[nginx][error][datetime]", "YYYY/MM/dd H:m:s" ]
remove_field => "[nginx][error][datetime]"
}
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "filebeat"
}
}
And just to make it complete, this is the pattern file:
NGINX_ACCESS1 %{HTTPDATE:[nginx][access][datetime]}\|%{IPORHOST:[nginx][access][client]}\|%{DATA:[nginx][access][user]}\|\"%{WORD:[nginx][access][method]} (%{URI:[nginx][access][resource]})?(?:%{URIPATH:[nginx][access][resource]})?(?:%{URIPARAM:[nginx][access][params]})?(?:%{GREEDYDATA:[nginx][access][payload]})?(\ )?HTTP/%{NUMBER:[nginx][access][httpversion]}\" %{NUMBER:[nginx][access][response]} (?:%{NUMBER:[nginx][access][bytes]}|-) \"(-|(?:%{URI:[nginx][access][referrer_domain]})|(?:%{IPORHOST:[nginx][access][referrer_domain]})(?:%{URIPATH:[nginx][access][referrer_ressource]})?(?:%{URIPARAM:[nginx][access][referrer_params]})?)\" \"%{GREEDYDATA:[nginx][access][agent]}\" (?:%{WORD:[nginx][access][cache]}|-)
NGINX_ERROR1 %{DATESTAMP:[nginx][error][datetime]} \[(error|crit)\] %{NUMBER:[nginx][error][id1]}#%{NUMBER:[nginx][error][id2]}\: \*%{NUMBER:[nginx][error][id3]} %{GREEDYDATA:[nginx][error][message]}(, client: %{IPORHOST:[nginx][error][client]})?(, server: %{IPORHOST:[nginx][error][server]})?(, request: "%{WORD:[nginx][error][method]})? (%{URIPATH:[nginx][error][path]})?(%{URIPARAM:[nginx][error][params]})?%{GREEDYDATA} HTTP/%{NUMBER:[nginx][error][httpversion]}\"(, upstream: \"%{NOTSPACE:[nginx][error][socket]}\")?(,host: \"{NOTSPACE:[nginx][error][host]\")?
NGINX_ERROR2 %{DATESTAMP:[nginx][error][datetime]} \[(error|crit)\] %{NUMBER:[nginx][error][id1]}#%{NUMBER:[nginx][error][id2]}\: \*%{NUMBER:[nginx][error][id3]} %{GREEDYDATA:[nginx][error][message]}
As I already said, all access-logs are processed pretty fine, I have around 300.000 for the last 30 days.
But when it comes to the error-logs, I only see a couple of lines there.
First thing I noticed, is that the multiline-stuff is not working correctly, even when I try to set it up in modules.d/nginx.yml. Whatever, I would be fine with that. But second thing is, and this is the main issue for me here:
I onyl see around 2, 3 lines from each log-file and have no clue why. Any help is highly appreciated, cause I now already spend days searching for the problem.
Thanks a lot
cheers
//EDIT
After hours of additional research I decided to dump the complete installation and did a fresh re-installation of the ELK stack in V7.