# Logstash sending logs to wrong indices
**URL:** https://discuss.elastic.co/t/logstash-sending-logs-to-wrong-indices/110024
**Category:** Logstash
**Created:** [December 2, 2017, 11:09pm UTC](https://discuss.elastic.co/t/logstash-sending-logs-to-wrong-indices/110024 "2017-12-02T23:09:36Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@GuillaumeN](https://discuss.elastic.co/u/GuillaumeN)
#### Post date: [December 2, 2017, 11:09pm UTC](https://discuss.elastic.co/t/logstash-sending-logs-to-wrong-indices/110024/1 "2017-12-02T23:09:36Z")
Hi!
I'm having an issue with logstash, it's sending logs to the configured output indices of ALL the pipeline configuration files (I have 2 with 1 output index per file) even though it shouldn't.
I'm using the ELK stack version 5.6. I have 1 logstash VM and 2 clustered elasticsearch VMs, it's a very basic setup.
The flow for my logs is as follows:
Cisco Syslog files =\> /ELK/logstash/Cisco\_Syslogs/ =\> Logstash =\> elasticsearch:logstash-syslogs-%{+YYYY.MM.dd}
IIS log files =\> /ELK/logstash/IIS/ =\> Logstash =\> elasticsearch:logstash-iis\_logs-%{+YYYY.MM.dd}
I copy log files on the logstash VM in two different directories, one for Cisco syslogs (/ELK/logstash/Cisco\_Syslogs/) and one for IIS logs (/ELK/logstash/IIS/).
Logstash then processes these logs and sends them to the elasticsearch cluster, each file type goes to a dedicated index.
I have 2 pipeline configuration files in /etc/logstash/conf.d, one is used for processing IIS logs, the other is for processing the Cisco Syslogs.
The relevant pipeline configurations are:
/etc/logstash/conf.d/Cisco\_Syslogs.conf
> input {
> file {
> path =\> '/ELK/logstash/Cisco\_Syslogs/\*'
> start\_position =\> 'beginning'
> }
> }
>
> filter {
> ...
> }
>
> output {
> elasticsearch {
> hosts =\> ['10.199.151.181:9200','10.199.151.50:9200']
> index =\> 'logstash-syslogs-%{+YYYY.MM.dd}'
> }
> }
/etc/logstash/conf.d/IIS.conf
> input {
> file {
> path =\> '/ELK/logstash/IIS/\*'
> start\_position =\> 'beginning'
> }
> }
>
> filter {
> ...
> }
>
> output {
> elasticsearch {
> hosts =\> ['10.199.151.181:9200','10.199.151.50:9200']
> index =\> 'logstash-iis\_logs-%{+YYYY.MM.dd}'
> }
> }
For each and every log line in either one of the source folders, logstash will systematically create 1 document in each destination indices, resulting in a big mixup.
When I just have 1 pipeline conf file in the conf.d folder (either one of them), everything works perfectly.
Why is logstash being this way and how can I prevent it from doing so?
For instance, I just extracted one line from a random cisco syslog file and put it under the source directory for the Cisco\_Syslogs pipeline:
> [root@logstash-1 ~]# cat /ELK/logstash/Cisco\_Syslogs/test
> \<190\>1 2017-11-23T05:02:53-05:00 SOURCEDEVICENAME - - - [meta sequenceId="821"] %SEC-6-IPACCESSLOGP: list TRAC2.0-ingress denied udp 123.123.123.123(37585) -\> 234.234.234.234(123), 1 packet
> [root@logstash-1 ~]#
When logstash parses this file, it will output it to the 2 indices:
> [root@logstash-1 ~]# curl -XGET "[http://10.199.151.181:9200/logstash-\*/\_search](http://10.199.151.181:9200/logstash-*/_search)" -H 'Content-Type: application/json' -d'
> {
> "query": {
> "match\_all": {}
> }
> }'
> {
> "took":1,
> "timed\_out":false,
> "\_shards":{
> "total":10,
> "successful":10,
> "skipped":0,
> "failed":0
> },
> "hits":{
> "total":2,
> "max\_score":1.0,
> "hits":[
> {
> "\_index":"logstash-syslogs-2017.11.23",
> "\_type":"logs",
> "\_id":"AWAZWwwom91xtBkxIlEH",
> "\_score":1.0,
> "\_source":{
> "path":"/ELK/logstash/Cisco\_Syslogs/test",
> "@timestamp":"2017-11-23T10:02:53.000Z",
> "syslog\_mnemonic":"SEC-6-IPACCESSLOGP",
> "syslog\_source\_device":"TMI-ONNRBA-NORBAT1-RTR01-EDG",
> "syslog\_timestamp":"2017-11-23T05:02:53-05:00",
> "@version":"1",
> "host":"logstash-1",
> "message":"\<190\>1 2017-11-23T05:02:53-05:00 TMI-ONNRBA-NORBAT1-RTR01-EDG - - - [meta sequenceId="821"] %SEC-6-IPACCESSLOGP: list TRAC2.0-ingress denied udp 172.18.49.24(37585) -\> 149.56.121.19(123), 1 packet ",
> "tags":[
> "cisco\_syslog",
> "\_grokparsefailure"
> ]
> }
> },
> {
> "\_index":"logstash-iis\_logs-2017.11.23",
> "\_type":"logs",
> "\_id":"AWAZWwylm91xtBkxIlEK",
> "\_score":1.0,
> "\_source":{
> "path":"/ELK/logstash/Cisco\_Syslogs/test",
> "@timestamp":"2017-11-23T10:02:53.000Z",
> "syslog\_mnemonic":"SEC-6-IPACCESSLOGP",
> "syslog\_source\_device":"TMI-ONNRBA-NORBAT1-RTR01-EDG",
> "syslog\_timestamp":"2017-11-23T05:02:53-05:00",
> "@version":"1",
> "host":"logstash-1",
> "message":"\<190\>1 2017-11-23T05:02:53-05:00 TMI-ONNRBA-NORBAT1-RTR01-EDG - - - [meta sequenceId="821"] %SEC-6-IPACCESSLOGP: list TRAC2.0-ingress denied udp 172.18.49.24(37585) -\> 149.56.121.19(123), 1 packet ",
> "tags":[
> "cisco\_syslog",
> "\_grokparsefailure"
> ]
> }
> }
> ]
> }
> }
> [root@logstash-1 ~]#
As you can see, that log line was put in both logstash-syslogs-2017.11.23 & logstash-iis\_logs-2017.11.23.
It should only go to logstash-syslogs-2017.11.23.
The \_grokparsefailure isn't caused by my grok pattern, when I just have 1 pipeline file they don't occur.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 3, 2017, 10:05am UTC](https://discuss.elastic.co/t/logstash-sending-logs-to-wrong-indices/110024/2 "2017-12-03T10:05:45Z")
When you put multiple config files in the same directory, Logstash will concatenate these into a single pipeline. If you do not use [conditional to separate out the different flows](https://www.elastic.co/guide/en/logstash/6.0/event-dependent-configuration.html#event-dependent-configuration), all inputs will go to all outputs.
---
### Author:  [@GuillaumeN](https://discuss.elastic.co/u/GuillaumeN)
#### Post date: [December 3, 2017, 7:10pm UTC](https://discuss.elastic.co/t/logstash-sending-logs-to-wrong-indices/110024/3 "2017-12-03T19:10:02Z")
Thank you for your answer Magnus.
I totally overlooked this fact in the documentation.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 31, 2017, 7:10pm UTC](https://discuss.elastic.co/t/logstash-sending-logs-to-wrong-indices/110024/4 "2017-12-31T19:10:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.