# Logstash indices mixup in Kibana

**URL:** <https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454>\
**Category:** Logstash\
**Created:** [April 6, 2017, 11:40am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454 "2017-04-06T11:40:19Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 6, 2017, 11:40am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/1 "2017-04-06T11:40:19Z")

</div>

Am using filebeat to get logs from remote app server and Logstash jdbc plugin to get logs from DB server but at Kibana when the log is flowing the indices are mixing up i,e logs from app server is showing in DB index and vice-versa.  
Any suggestions on how to avoid this.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 6, 2017, 11:48am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/2 "2017-04-06T11:48:22Z")

</div>

What's in your Logstash configuration file(s)?

---

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 6, 2017, 11:55am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/3 "2017-04-06T11:55:19Z")

</div>

input {  
beats {  
port =\> 5044  
}  
}

filter {  
if "access\_logs" in [tags] {  
grok {  
match =\> {  
"message" =\> [  
"%{IPORHOST:x\_forwarded\_for} %{IPORHOST:load\_balancer} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb}%{SPACE}  
/%{WORD:application}}%{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)  
(?:%{WORD:ServerHost}:%{WORD:ServerPort})",  
"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} /%{WORD:application}%{NOTSPACE:request}(?:  
HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{WORD:ServerHost}:%{WORD:ServerPort})",  
"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} /%{WORD:application}%{NOTSPACE:request}(?:  
HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)"  
]  
}  
}  
}  
if "BPM" in [tags] {  
grok {  
match =\> {  
"message" =\> [  
"%{SYSLOG5424SD}%{SPACE}%{BASE16NUM:ThreadID}%{SPACE}%{WORD:ShortName}%{SPACE}%{WORD:EventType}%{SPACE}%{WORD:MessageIdentifier}:%{SPACE}%{GREEDYDATA:event}",  
"%{SYSLOG5424SD}%{SPACE}%{WORD:ThreadID}%{SPACE}%{WORD:Logger}%{SPACE}%{WORD:MessageType}%{SPACE}%{GREEDYDATA:event}"  
]  
}  
}  
}  
if "syslog" in [type][tags] {  
grok {  
match =\> {  
"message" =\> [  
"%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?:  
%{GREEDYDATA:syslog\_message}"  
]  
}  
}  
}  
}

if[tags] == "access\_log"  
{  
output {  
elasticsearch {  
hosts =\> ["10.190.188.174:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}  
}  
else if [tags] == "BPM"  
{  
output {  
elasticsearch {  
hosts =\> ["10.190.188.174:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}  
}  
else [tags] == "syslog"  
{  
output {  
elasticsearch {  
hosts =\> ["10.190.188.174:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 6, 2017, 12:01pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/4 "2017-04-06T12:01:16Z")

</div>

You're complaining about events being mixed up but you're sending all events to "logstash-%{+YYYY.MM.dd}" so I'm confused over what the problem is. All your elasticsearch outputs are identical.

> ```
> if "syslog" in [type][tags] {
> 
> ```

This doesn't make sense. The `type` field doesn't have any subfields.

> ```
> if[tags] == "access_log"
> 
> ```

The `tags` field is an array and not a string so I wouldn't expect this to ever be true. I suggest you use `if "access_log" in [tags]` instead.

> ```
> else [tags] == "syslog"
> 
> ```

Use plain `else` or `else if "syslog" in [tags]`.

---

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 6, 2017, 1:55pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/5 "2017-04-06T13:55:21Z")

</div>

Am a newbie to ELK and i didn't understand the output part do i need to change **else [tags] == "syslog"** to **else if "syslog" in [tags]**??

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 6, 2017, 1:56pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/6 "2017-04-06T13:56:53Z")

</div>

Yes.

---

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 6, 2017, 2:09pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/7 "2017-04-06T14:09:12Z")

</div>

so i have made the changes as said  
now when i do configtest for logstash am getting below error  
**Expected one of #, input, filter, output at line 155, column 1 (byte 5330) after {:level=\>:error}**

input {  
beats {  
port =\> 5044  
}  
}

filter {  
if "access\_logs" in [tags] {  
grok {  
match =\> {  
"message" =\> [  
"%{IPORHOST:x\_forwarded\_for} %{IPORHOST:load\_balancer} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb}%{SPACE}  
/%{WORD:application}}%{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)  
(?:%{WORD:ServerHost}:%{WORD:ServerPort})",  
"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} /%{WORD:application}%{NOTSPACE:request}(?:  
HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{WORD:ServerHost}:%{WORD:ServerPort})",  
"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} /%{WORD:application}%{NOTSPACE:request}(?:  
HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)"  
]  
}  
}  
}  
if "BPM" in [tags] {  
grok {  
match =\> {  
"message" =\> [  
"%{SYSLOG5424SD}%{SPACE}%{BASE16NUM:ThreadID}%{SPACE}%{WORD:ShortName}%{SPACE}%{WORD:EventType}%{SPACE}%{WORD:MessageIdentifier}:%{SPACE}%{GREEDYDATA:event}",  
"%{SYSLOG5424SD}%{SPACE}%{WORD:ThreadID}%{SPACE}%{WORD:Logger}%{SPACE}%{WORD:MessageType}%{SPACE}%{GREEDYDATA:event}"  
]  
}  
}  
}  
if "syslog" in [tags] {  
grok {  
match =\> {  
"message" =\> [  
"%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?:  
%{GREEDYDATA:syslog\_message}"  
]  
}  
}  
}  
}

if[tags] == "access\_log"  
{  
output {  
elasticsearch {  
hosts =\> ["10.190.188.174:9200"]  
index =\> "access-%{+YYYY.MM.dd}"  
}  
}  
}  
else [tags] == "BPM"  
{  
output {  
elasticsearch {  
hosts =\> ["10.190.188.174:9200"]  
index =\> "bpm-%{+YYYY.MM.dd}"  
}  
}  
}  
else if "syslog" in [tags]  
{  
output {  
elasticsearch {  
hosts =\> ["10.190.188.174:9200"]  
index =\> "sys-%{+YYYY.MM.dd}"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 6, 2017, 2:47pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/8 "2017-04-06T14:47:24Z")

</div>

> ```
> else [tags] == "BPM"
> 
> ```

Why did you drop the "if" from here? The only things allowed after `else` is `{` or `if`.

It would be much much easier to debug your configuration if it was indented properly and posted as preformatted text. Right now one has to count braces and that's both boring and prone to mistakes.

---

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 12, 2017, 10:23am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/9 "2017-04-12T10:23:39Z")

</div>

So i have re-written all the grok filter and formatted with indentation still am facing the  
**Error: Expected one of #, {, } at line 13, column 85 (byte 185) after filter {**

All the grok filters are fine i have debugged it from [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)

```
 input {
      beats {
    port => 5044
    		}
    	}

filter {
		if "access_logs" in [tags] 
		{
		grok {
			match => {
					    "message" => "%{IPORHOST:x_forwarded_for} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion})" %{NUMBER:response}"
					}
			}
		}
	    if "BPM" in [tags] 
		{
		grok {
			match => {
						"message" => "%{SYSLOG5424SD:BPM_timestamp} %{BASE16NUM:ThreadID} %{WORD:EventType} %{WORD:ShortName} %{WORD:MessageIdentifier}:%{SPACE}%{GREEDYDATA:event}"
					}
			}
		}
	    if "syslog" in [tags]
		{
		grok {
			match => {
						"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program} (?:\[%{POSINT:syslog_pid}\])?%{GREEDYDATA:syslog_message}"
					}
			}
		}
	}

if [tags] == "access_log"
{
	output {
	elasticsearch { 
					hosts => ["10.190.188.174:9200"]
					index => "access-%{+YYYY.MM.dd}"
				}
			}
	}
else if [tags] == "BPM"
{
	output {
	elasticsearch { 
					hosts => ["10.190.188.174:9200"]
					index => "bpm-%{+YYYY.MM.dd}"
				}
		}
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 11:15am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/10 "2017-04-12T11:15:05Z")

</div>

If think your output section needs to look like this:

```nohighlight
output {
  if ... {
    elasticsearch {
      ...
    }
  } else if ... {
    ..
  }
}

```

---

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 12, 2017, 11:25am UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/11 "2017-04-12T11:25:00Z")

</div>

awesome finally this works, thanks for your valuable quick replies 🙂

---

<div class="post-metadata">

**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\
**Post date:** [April 12, 2017, 1:25pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/12 "2017-04-12T13:25:27Z")

</div>

Config test is ok but now the indices are not creating in elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2017, 1:35pm UTC](https://discuss.elastic.co/t/logstash-indices-mixup-in-kibana/81454/13 "2017-05-10T13:35:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.