Multiple output in logstash config, pls help!


#1

I managed to put multiple inputs/filters/outputs in one logstash conf, with no error or warning while running elasticsearch and logstash, but there is only one index in elasticsearch, pls help:

my con is as below:

input {
beats {
port => "5044"
type => "applog"
}
udp {
port => "514"
type => "syslog"
}
}
filter {
if [type] == "applog" {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
date {
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
remove_field => [ "logdate" ]
}
}
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
if [type] == "applog" {
elasticsearch {
hosts => [ "192.168.11.2:9200" ]
index => "applog.logstash-%{+YYYY.MM.dd}"
}
}
if [type] == "syslog" {
elasticsearch {
hosts => [ "192.168.11.2:9200" ]
index => "syslog.logstash-%{+YYYY.MM.dd}"
}
}
}

as curl to elasticsearch, I got this:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana fQuowSJwRMO8uQ17yzmqlw 1 1 2 0 10.1kb 10.1kb
yellow open syslog.logstash-2016.11.19 PEsD8uhNSVSLj1U3hxkaJQ 5 1 1501 0 878.3kb 878.3kb

what's wrong and what should I do?


(Magnus Bäck) #2

Is Logstash receiving anything via the beats input? What do the logs on the sending side look like (Filebeat or whatever it is you're using)?


#3

the sending side is : filebeat for sending nginx(access log), and winlogbeat for sending windows eventlog.


(Magnus Bäck) #4

Yes, and what do the logs of those applications say?


(Magnus Bäck) #6

No, I'm asking for the log of Filebeat itself. Is there any sign in its log that it's either successful or not successful in sending data to Logstash?


#7

Wow!, just check the filebeat log, it showed:

2016-11-21T20:59:18+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T20:59:48+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:00:18+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:00:48+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:01:18+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:01:48+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:02:18+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:02:48+08:00 INFO No non-zero metrics in the last 30s
2016-11-21T21:03:18+08:00 INFO No non-zero metrics in the last 30s


#8

Finally got this solved by using filebeat to elasticsearch directly, after restarting filebeat, everything is fine.


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.