# How to create multiple Indices of elasticsearch using logstash configuration
**URL:** https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537
**Category:** Logstash
**Created:** [March 7, 2020, 4:50am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537 "2020-03-07T04:50:38Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 7, 2020, 4:50am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/1 "2020-03-07T04:50:38Z")
Hello everyone,
I am trying to create different indices of elasticsearch using logstash but i am not able to do this.can anyone help me out with this.How we can create and send multiple different index to elasticsearch.
Thanks in advance.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 7, 2020, 12:43pm UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/2 "2020-03-07T12:43:09Z")
What have you tried so far? What are your inputs?
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 5:20am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/3 "2020-03-09T05:20:49Z")
Hi Christian Dahlqvist,
Thanks for replying,
Now I can able to create multiple indices.
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 5:33am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/4 "2020-03-09T05:33:13Z")
Hi Christian,
I created two different droplets in Digitalocen and install nginx and filebeat on those two droplets.I configured my filebeat.yml file as below:
VM1 filebeat.yml:\_
filebeat.prospectors:
- type: log
paths:
- /var/log/nginx/_.log
filebeat.config.modules:
path: ${path.config}/modules.d/_.yml
reload.enabled: true
setup.template.settings:
index.number\_of\_shards: 3
output.logstash:
# The Logstash hosts
hosts: ["134.209.153.109:5045"]
VM2 filebeat.yml:\_
filebeat.prospectors:
- type: log
paths:
- /var/log/nginx/_.log
filebeat.config.modules:
path: ${path.config}/modules.d/_.yml
reload.enabled: true
setup.template.settings:
index.number\_of\_shards: 3
output.logstash:
# The Logstash hosts
hosts: ["134.209.153.109:5044"]
My Logstash vm1.conf file:
input {
beats {
port =\> 5045
}
}
filter {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match =\> { "message" =\> ["%{IPORHOST:[nginx][access][remote\_ip]} - %{DATA:[nginx][access][user\_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http\_version]}" %{NUMBER:[nginx][access][response\_code]} %{NUMBER:[nginx][access][body\_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
remove\_field =\> "message"
}
output {
elasticsearch {
hosts =\> ["134.209.153.109:9200"]
manage\_template =\> false
index =\> "vm1\_nginx-1"
}
}
My Logstash vm2.conf file:
input {
beats {
port =\> 5044
}
}
filter {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match =\> { "message" =\> ["%{IPORHOST:[nginx][access][remote\_ip]} - %{DATA:[nginx][access][user\_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http\_version]}" %{NUMBER:[nginx][access][response\_code]} %{NUMBER:[nginx][access][body\_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
remove\_field =\> "message"
}
output {
elasticsearch {
hosts =\> ["134.209.153.109:9200"]
manage\_template =\> false
index =\> "vm2\_nginx-1"
}
}
I configured my file and whhen i run logstash i am getting the error like below
[FATAL] 2020-03-09 05:19:25.972 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[ERROR] 2020-03-09 05:19:25.988 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Can you please help me out how to solve the problem.Can you help me out what is the meaning " If you wish to run multiple instances, you must change the "path.data" setting"
Thanks in advance
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 9, 2020, 6:30am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/5 "2020-03-09T06:30:08Z")
There is no need to have multiple Logstash confogurations to do what you are you are requesting. You can even have just a single beats input.
Assuming you have a field (or combination of fields) in the logs that determine which index they are supposed to go to you [can use conditionals](https://www.elastic.co/guide/en/logstash/7.6/event-dependent-configuration.html) based on this to control the flow like this (you need to change field references and values):
```auto
if [host][name] == "vm1" {
elasticsearch {
...
index => "vm1_nginx-1"
...
}
} else {
elasticsearch {
...
index => "vm2_nginx-1"
...
}
}
```
An even easier way might be to [use the elasticsearch output plugin directly to write to different indices based on data in the event](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices).
I see that you are separating data into different indices based on the source. This is generally not necessary as a single index can handle vary large amounts of data. What is driving this requirement?
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 6:43am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/6 "2020-03-09T06:43:47Z")
Hi ,
Thanks for reply,
My task is to create different index for my project.My project using technologies like php,Laravel and Nginx,Mysql. So i need to get logs for each one and have to create separate index for it like for Laravel separate index and for Mysql separate index and so on.Because of this i am creating multiple conf files.Can you suggest me the best way this could be done.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 9, 2020, 7:44am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/7 "2020-03-09T07:44:56Z")
Did you look at the resources I linked to?
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 7:50am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/8 "2020-03-09T07:50:42Z")
Yes i saw those resources and now i am creating single logstash.conf and having multiple elasticsearch outputs .Let me check and I will keep you post.
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 8:00am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/9 "2020-03-09T08:00:30Z")
Hi have a doubt should i fill any data in [host][name] fields ??
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 9, 2020, 8:03am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/10 "2020-03-09T08:03:32Z")
Look at the events you have got in Elasticsearch so you can see the fields that are populated. If f nothing has reached Elasticsearch instead add a stipulation output filter with a rubydebug codec so you can see what is being processed.
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 8:05am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/11 "2020-03-09T08:05:08Z")
Ok I that is fine.I want to know should i write any data in
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 8:07am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/12 "2020-03-09T08:07:02Z")
sorry for above comment by mistake it has sent\>\>
if [host][name] == "vm1" {
elasticsearch {
above if condition [host] and [name] fields are there should i have to write my host ip or anything??
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 9, 2020, 8:13am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/13 "2020-03-09T08:13:34Z")
Did you read both links I provided? If so, what in the examples given there is not clear?
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 10:15am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/14 "2020-03-09T10:15:29Z")
Hi I went through those topics and i understood that thanks for that
---
### Author:  [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)
#### Post date: [March 9, 2020, 12:16pm UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/15 "2020-03-09T12:16:57Z")
Can you tell me why i am getting error like below
INFO ] 2020-03-09 10:13:50.206 [[main]\5044, id=\>"80e52ac0ad4b661509cb2f6834a82e7666f97274a7d23997b34b29cb7e49cce4", enable\_metric=\>true, codec=\>\"plain\_55210164-c165-4c19-bdc6-25897016d396", enable\_metric=\>true, charset=\>"UTF-8"\>, host=\>"0.0.0.0", ssl=\>false, add\_hostname=\>false, ssl\_verify\_mode=\>"none", ssl\_peer\_metadata=\>false, include\_codec\_tag=\>true, ssl\_handshake\_timeout=\>10000, tls\_min\_version=\>1, tls\_max\_version=\>1.2, cipher\_suites=\>["TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA256"], client\_inactivity\_timeout=\>60, executor\_threads=\>2\>
Error: Address already in use
Exception: Java::JavaNet::BindException
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 6, 2020, 12:17pm UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537/16 "2020-04-06T12:17:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.