Hi!
I'm having an issue with logstash, it's sending logs to the configured output indices of ALL the pipeline configuration files (I have 2 with 1 output index per file) even though it shouldn't.
I'm using the ELK stack version 5.6. I have 1 logstash VM and 2 clustered elasticsearch VMs, it's a very basic setup.
The flow for my logs is as follows:
Cisco Syslog files => /ELK/logstash/Cisco_Syslogs/ => Logstash => elasticsearch:logstash-syslogs-%{+YYYY.MM.dd}
IIS log files => /ELK/logstash/IIS/ => Logstash => elasticsearch:logstash-iis_logs-%{+YYYY.MM.dd}
I copy log files on the logstash VM in two different directories, one for Cisco syslogs (/ELK/logstash/Cisco_Syslogs/) and one for IIS logs (/ELK/logstash/IIS/).
Logstash then processes these logs and sends them to the elasticsearch cluster, each file type goes to a dedicated index.
I have 2 pipeline configuration files in /etc/logstash/conf.d, one is used for processing IIS logs, the other is for processing the Cisco Syslogs.
The relevant pipeline configurations are:
/etc/logstash/conf.d/Cisco_Syslogs.conf
input {
file {
path => '/ELK/logstash/Cisco_Syslogs/*'
start_position => 'beginning'
}
}filter {
...
}output {
elasticsearch {
hosts => ['10.199.151.181:9200','10.199.151.50:9200']
index => 'logstash-syslogs-%{+YYYY.MM.dd}'
}
}
/etc/logstash/conf.d/IIS.conf
input {
file {
path => '/ELK/logstash/IIS/*'
start_position => 'beginning'
}
}filter {
...
}output {
elasticsearch {
hosts => ['10.199.151.181:9200','10.199.151.50:9200']
index => 'logstash-iis_logs-%{+YYYY.MM.dd}'
}
}
For each and every log line in either one of the source folders, logstash will systematically create 1 document in each destination indices, resulting in a big mixup.
When I just have 1 pipeline conf file in the conf.d folder (either one of them), everything works perfectly.
Why is logstash being this way and how can I prevent it from doing so?
For instance, I just extracted one line from a random cisco syslog file and put it under the source directory for the Cisco_Syslogs pipeline:
[root@logstash-1 ~]# cat /ELK/logstash/Cisco_Syslogs/test
<190>1 2017-11-23T05:02:53-05:00 SOURCEDEVICENAME - - - [meta sequenceId="821"] %SEC-6-IPACCESSLOGP: list TRAC2.0-ingress denied udp 123.123.123.123(37585) -> 234.234.234.234(123), 1 packet
[root@logstash-1 ~]#
When logstash parses this file, it will output it to the 2 indices:
[root@logstash-1 ~]# curl -XGET "http://10.199.151.181:9200/logstash-*/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"match_all": {}
}
}'
{
"took":1,
"timed_out":false,
"_shards":{
"total":10,
"successful":10,
"skipped":0,
"failed":0
},
"hits":{
"total":2,
"max_score":1.0,
"hits":[
{
"_index":"logstash-syslogs-2017.11.23",
"_type":"logs",
"_id":"AWAZWwwom91xtBkxIlEH",
"_score":1.0,
"_source":{
"path":"/ELK/logstash/Cisco_Syslogs/test",
"@timestamp":"2017-11-23T10:02:53.000Z",
"syslog_mnemonic":"SEC-6-IPACCESSLOGP",
"syslog_source_device":"TMI-ONNRBA-NORBAT1-RTR01-EDG",
"syslog_timestamp":"2017-11-23T05:02:53-05:00",
"@version":"1",
"host":"logstash-1",
"message":"<190>1 2017-11-23T05:02:53-05:00 TMI-ONNRBA-NORBAT1-RTR01-EDG - - - [meta sequenceId="821"] %SEC-6-IPACCESSLOGP: list TRAC2.0-ingress denied udp 172.18.49.24(37585) -> 149.56.121.19(123), 1 packet ",
"tags":[
"cisco_syslog",
"_grokparsefailure"
]
}
},
{
"_index":"logstash-iis_logs-2017.11.23",
"_type":"logs",
"_id":"AWAZWwylm91xtBkxIlEK",
"_score":1.0,
"_source":{
"path":"/ELK/logstash/Cisco_Syslogs/test",
"@timestamp":"2017-11-23T10:02:53.000Z",
"syslog_mnemonic":"SEC-6-IPACCESSLOGP",
"syslog_source_device":"TMI-ONNRBA-NORBAT1-RTR01-EDG",
"syslog_timestamp":"2017-11-23T05:02:53-05:00",
"@version":"1",
"host":"logstash-1",
"message":"<190>1 2017-11-23T05:02:53-05:00 TMI-ONNRBA-NORBAT1-RTR01-EDG - - - [meta sequenceId="821"] %SEC-6-IPACCESSLOGP: list TRAC2.0-ingress denied udp 172.18.49.24(37585) -> 149.56.121.19(123), 1 packet ",
"tags":[
"cisco_syslog",
"_grokparsefailure"
]
}
}
]
}
}
[root@logstash-1 ~]#
As you can see, that log line was put in both logstash-syslogs-2017.11.23 & logstash-iis_logs-2017.11.23.
It should only go to logstash-syslogs-2017.11.23.
The _grokparsefailure isn't caused by my grok pattern, when I just have 1 pipeline file they don't occur.