# Index Naming
**URL:** https://discuss.elastic.co/t/index-naming/126584
**Category:** Elasticsearch
**Created:** [April 3, 2018, 1:51pm UTC](https://discuss.elastic.co/t/index-naming/126584 "2018-04-03T13:51:05Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@mmethe](https://discuss.elastic.co/u/mmethe)
#### Post date: [April 3, 2018, 1:51pm UTC](https://discuss.elastic.co/t/index-naming/126584/1 "2018-04-03T13:51:05Z")
How do I get my index named properly?
I have an index appearing as:
%{[@metadata][beat]}-2018.04.03
While my beats appear properly as:
winlogbeat-2017.12.19
winlogbeat-2017.12.20
My conf:
input {
beats {
port =\> 5044
}
}
filter {
if [type] == "syslog" {
grok {
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }
add\_field =\> ["received\_at", "%{@timestamp}"]
add\_field =\> ["received\_from", "%{host}"]
}
syslog\_pri { }
date {
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][type]}"
}
}
input {
jdbc {
jdbc\_driver\_library =\> "/usr/local/sbin/mssql-jdbc-6.2.2.jre8.jar"
jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc\_connection\_string =\> "jdbc:sqlserver://brt1-trackit01\BRT1TRACKIT01;databaseName=TRACKIT8\_DATA2"
jdbc\_user =\> "reports\_jsp"
jdbc\_password =\> "letmeinNOW!"
type =\> "sql\_tk\_tasks"
statement =\> "select WO\_NUM,OPENDATE,CLSDDATE,TYPE,RESPONS from tasks where OPENDATE BETWEEN DATEADD(day,-30,GETDATE()) AND GETDATE()"
}
}
output {
if [type] == "sql\_tk\_tasks" {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "tasks2-%{+YYYY.MM.dd}"
document\_type =\> "tasks2"
}
}
}
Thank You!
---
### Author:  [@Evesy](https://discuss.elastic.co/u/Evesy)
#### Post date: [April 3, 2018, 2:25pm UTC](https://discuss.elastic.co/t/index-naming/126584/2 "2018-04-03T14:25:25Z")
Hi Mike,
It looks as though this config is configured as one single pipeline, as opposed to multiple pipelines which became available in Logstash 6 ([Multiple Pipelines | Logstash Reference [6.7] | Elastic](https://www.elastic.co/guide/en/logstash/6.x/multiple-pipelines.html))
If this is the case, since your first output has no conditionals on it, your JDBC logs will be outputting to both the 'sql tasks' output, but also your 'beat' output. Since those JDBC logs presumably don't have any beats metadata, it takes the name literally and creates an index with that name (i.e. `%{[@metadata][beat]}-2018.04.03`)
> [@mmethe](#):
>
> output {
>
> elasticsearch {
>
> hosts =\> ["localhost:9200"]
>
> sniffing =\> true
>
> manage\_template =\> false
>
> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
>
> document\_type =\> "%{[@metadata][type]}"
>
> }
>
> }
)
I think you can wrap your beats output in something like `if ("" in [@metadata][beat])` so that only beat related documents hit this output.
The other option is to split your config into multiple pipelines, as per the link I posted above
Cheers,
Mike
---
### Author:  [@mmethe](https://discuss.elastic.co/u/mmethe)
#### Post date: [April 4, 2018, 1:56pm UTC](https://discuss.elastic.co/t/index-naming/126584/3 "2018-04-04T13:56:01Z")
Thank You.
I built a test ELK stack yesterday afternoon to take advantage of multiple piplines. I have one beats server shipping events. This index gets created using the first pipline and is named correctly. The second pipeline never creates an index (JDBC input). When I run the config using --dubug, I can see the database rows being returned as they should be, just never indexed. the JDBC plugin is installed.
**piplines.ylm**
- pipeline.id: beats
path.config: "/etc/logstash/conf.d/beats.conf"
- pipeline.id: tasks
path.config: "/etc/logstash/conf.d/tasks/tasks.conf"
**tasks.conf**
input {
jdbc {
jdbc\_driver\_library =\> "/usr/local/sbin/mssql-jdbc-6.2.2.jre8.jar"
jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc\_connection\_string =\> "jdbc:sqlserver://brt1-trackit01\BRT1TRACKIT01;databaseName=TRACKIT8\_DATA2"
jdbc\_user =\> "reports\_jsp"
jdbc\_password =\> "letmeinNOW!"
type =\> "sql\_tk\_tasks"
statement =\> "select WO\_NUM,OPENDATE,CLSDDATE,TYPE,RESPONS from tasks where OPENDATE BETWEEN DATEADD(day,-60,GETDATE()) AND GETDATE()"
}
}
output {
if [type] == "sql\_tk\_tasks" {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "tasks2-%{+YYYY.MM.dd}"
document\_type =\> "tasks2"
}
}
}
---
### Author:  [@Evesy](https://discuss.elastic.co/u/Evesy)
#### Post date: [April 4, 2018, 2:05pm UTC](https://discuss.elastic.co/t/index-naming/126584/4 "2018-04-04T14:05:33Z")
I'd probably suggest upping the log level via the API for Logstash on your JDBC input, double check data is being read correctly: [https://www.elastic.co/guide/en/logstash/current/logging.html](https://www.elastic.co/guide/en/logstash/current/logging.html).
Your output block looks OK to me
---
### Author:  [@mmethe](https://discuss.elastic.co/u/mmethe)
#### Post date: [April 4, 2018, 2:24pm UTC](https://discuss.elastic.co/t/index-naming/126584/5 "2018-04-04T14:24:34Z")
Thanks. Upped to debug. I can see the pipeline starting in logstash-plain.log. Still dont see the index though.
[2018-04-04T10:22:12,362][INFO][logstash.pipeline] Pipeline started succesfully {:pipeline\_id=\>"tasks", :thread=\>"#\"}
[2018-04-04T10:22:12,365][INFO][logstash.agent] Pipelines running {:count=\>2, :pipelines=\>["beats", "tasks"]}
[2018-04-04T10:22:12,873][INFO][logstash.inputs.jdbc] (0.038025s) select WO\_NUM,OPENDATE,CLSDDATE,TYPE,RESPONS from tasks where OPENDATE BETWEEN DATEADD(day,-60,GETDATE()) AND GETDATE()
[2018-04-04T10:22:14,260][INFO][logstash.pipeline] Pipeline has terminated {:pipeline\_id=\>"tasks", :thread=\>"#\"}
[2018-04-04T10:22:16,512][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[[http://localhost:9200/](http://localhost:9200/)], :added=\>[[http://127.0.0.1:9200/](http://127.0.0.1:9200/)]}}
[2018-04-04T10:22:16,513][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://127.0.0.1:9200/](http://127.0.0.1:9200/), :path=\>"/"}
[2018-04-04T10:22:16,527][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://127.0.0.1:9200/](http://127.0.0.1:9200/)"}
---
### Author:  [@mmethe](https://discuss.elastic.co/u/mmethe)
#### Post date: [April 4, 2018, 5:30pm UTC](https://discuss.elastic.co/t/index-naming/126584/6 "2018-04-04T17:30:06Z")
Further testing: I peeled a beat off into a pipeline separate from other beats. I did see the new index in kibana. This verifies that multi-pipeline is working. MYSTERY: Why is the JDBC input not creating an output index? How can watch for errors on the index creation?
**beats.conf** :
input {
beats {
port =\> 5044
}
}
filter {
if [type] == "syslog" {
grok {
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }
add\_field =\> ["received\_at", "%{@timestamp}"]
add\_field =\> ["received\_from", "%{host}"]
}
syslog\_pri { }
date {
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][type]}"
}
}
**beats2.conf** :
input {
beats {
port =\> 5045
}
}
filter {
if [type] == "syslog" {
grok {
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }
add\_field =\> ["received\_at", "%{@timestamp}"]
add\_field =\> ["received\_from", "%{host}"]
}
syslog\_pri { }
date {
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "beats2-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][type]}"
}
}
**tasks.conf** (no index being created):
input {
jdbc {
jdbc\_driver\_library =\> "/usr/local/sbin/mssql-jdbc-6.2.2.jre8.jar"
jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc\_connection\_string =\> "jdbc:sqlserver://brt1-trackit01\BRT1TRACKIT01;databaseName=TRACKIT8\_DATA2"
jdbc\_user =\> "reports\_jsp"
jdbc\_password =\> "letmeinNOW!"
type =\> "sql\_tk\_tasks"
statement =\> "select WO\_NUM,OPENDATE,CLSDDATE,TYPE,RESPONS from tasks where OPENDATE BETWEEN DATEADD(day,-60,GETDATE()) AND GETDATE()"
}
}
output {
if [type] == "sql\_tk\_tasks" {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "tasks2-%{+YYYY.MM.dd}"
document\_type =\> "tasks2"
}
}
}
**pipelines.yml:**
- pipeline.id: beats
path.config: "/etc/logstash/conf.d/beats.conf"
- pipeline.id: beats2
path.config: "/etc/logstash/conf.d/beats2/beats2.conf"
- pipeline.id: tasks
path.config: "/etc/logstash/conf.d/tasks/tasks.conf"
---
### Author:  [@Evesy](https://discuss.elastic.co/u/Evesy)
#### Post date: [April 4, 2018, 7:14pm UTC](https://discuss.elastic.co/t/index-naming/126584/7 "2018-04-04T19:14:27Z")
If you're using multiple pipelines then you don't need to worry about wrapping the output in `if [type] == "sql_tk_tasks"`, since you want all documents in that pipeline to use that output.
I'd try removing that first, see if that gets you anywhere. If you also have Logstash monitoring enabled you could view the pipeline in Kibana, and double check the JDBC input is receiving events, and if the output is receiving any. [https://www.elastic.co/guide/en/logstash/current/logstash-pipeline-viewer.html](https://www.elastic.co/guide/en/logstash/current/logstash-pipeline-viewer.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 2, 2018, 7:14pm UTC](https://discuss.elastic.co/t/index-naming/126584/8 "2018-05-02T19:14:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.