# Create multiple Index for different steams of data

**URL:** <https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203>\
**Category:** Logstash\
**Created:** [January 15, 2021, 3:53am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203 "2021-01-15T03:53:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![leinad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leinad/32/82281_2.png) [@leinad](https://discuss.elastic.co/u/leinad)\
**Post date:** [January 15, 2021, 3:53am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/1 "2021-01-15T03:53:48Z")

</div>

Hi ,  
I am very new on setting up log stash, but have been using Kibana on user level of createing dashboard and analysis using basic templates

My input logs are all JSON types, this is the logstash config which works presently

input  
{  
beats  
{  
port =\> "5044"  
codec =\> "json"  
include\_codec\_tag =\> false  
}  
}

## output { elasticsearch { hosts =\> "localhost:9200" index =\> "fourindex" } }

the filebeat setting is here

- type: log

* * *

Question (1) How to add another stream of data placed in different paths example : C:\ProgramData\filebeat\logs\NewData\*.json and what is the corresponding change in the logstash config so that its created as a separate index in the kibana

---

<div class="post-metadata">

**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\
**Post date:** [January 15, 2021, 9:11am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/2 "2021-01-15T09:11:00Z")

</div>

Hi @leinad,

Input:  
You can use file input plugin for reading the logs from this path.  
_C:\ProgramData\filebeat\logs\NewData_.json\*

Output:  
Use _else, if conditions_  
Example:

else [path] == "NewData" {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> ["host:9200"]  
index =\> "new-index"  
}  
}

---

<div class="post-metadata">

**Author:** ![leinad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leinad/32/82281_2.png) [@leinad](https://discuss.elastic.co/u/leinad)\
**Post date:** [January 19, 2021, 12:02pm UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/3 "2021-01-19T12:02:56Z")

</div>

Thank @tahseen_fatima I am new so I trying to figure it out step by step,

---

<div class="post-metadata">

**Author:** ![leinad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leinad/32/82281_2.png) [@leinad](https://discuss.elastic.co/u/leinad)\
**Post date:** [January 25, 2021, 1:57am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/4 "2021-01-25T01:57:23Z")

</div>

hi @tahseen_fatima this is the code I used and got the error below

---------------SETTING IN FILEBEAT---------------------

- type: log

--- # Change to true to enable this input configuration.  
enabled: true

-# Paths that should be crawled and fetched. Glob based paths.  
paths:

# - /var/log/\*.log

#- C:\ProgramData\filebeat\logs\atg\Rev4\*.json

- D:\Logs\Rev4\*.json
- D:\Logs\Rev5\*.json  
close\_removed: true

--------------------logstash config CODE USED----------------------------  
input  
{  
beats  
{  
port =\> "5044"  
codec =\> "json"  
include\_codec\_tag =\> false  
}  
}

output  
{  
if [path] == "D:\Logs\Rev4"{  
stdout { codec =\> rubydebug }  
elasticsearch  
{  
hosts =\> "localhost:9200"  
index =\> "Rev4"  
}  
else if [path] == "D:\Logs\Rev5"{  
stdout { codec =\> rubydebug }   
{  
hosts =\> "localhost:9200"  
index =\> "Rev5"  
}  
}

## -------ERROR AT LOGSTASHOUTPUT--------------------- [2021-01-25T09:53:02,106][DEBUG][logstash.agent] Executing action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main} [2021-01-25T09:53:02,473][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "and", "or", "xor", "nand", "{" at line 17, column 13 (byte 196) after output \n{\n\tif [path] == "D:\Logs\Rev4\"{\nstdout { codec =\> rubydebug }\n\telasticsearch \n\t{ \n\t\thosts =\> "", :backtrace=\>["D:/Kibana/logstash-7.10.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "D:/Kibana/logstash-7.10.1/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "D:/Kibana/logstash-7.10.1/logstash-core/lib/logstash/pipeline\_action/create.rb:52:in `execute'", "D:/Kibana/logstash-7.10.1/logstash-core/lib/logstash/agent.rb:365:in `block in converge\_state'"]} [2021-01-25T09:53:02,517][DEBUG][logstash.instrument.periodicpoller.os] Stopping

---

<div class="post-metadata">

**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\
**Post date:** [January 25, 2021, 2:42am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/5 "2021-01-25T02:42:18Z")

</div>

Hi @leinad

**I can see there is an error at line 17, column 13.**

From your error message I can see the brackets are not closed properly in the output.

output  
{  
if [path] == "D:\Logs\Rev4"{  
stdout { codec =\> rubydebug }  
elasticsearch  
{  
hosts =\> "localhost:9200"  
index =\> "Rev4"  
}  
} **# you forgot to close this elasticsearch bracket here**  
else if [path] == "D:\Logs\Rev5"{  
stdout { codec =\> rubydebug }  
{  
hosts =\> "localhost:9200"  
index =\> "Rev5"  
}  
}  
} **# same mistake here**

Regards,  
Tahseen

---

<div class="post-metadata">

**Author:** ![leinad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leinad/32/82281_2.png) [@leinad](https://discuss.elastic.co/u/leinad)\
**Post date:** [January 25, 2021, 8:07am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/6 "2021-01-25T08:07:02Z")

</div>

Hi @tahseen_fatima Thanks for the corrections, i am able to run the config successfully, but i did no see the index created in the elastic search, not sure where to start the debug, i am able to create index without using any path separation pls advise

---

<div class="post-metadata">

**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\
**Post date:** [January 25, 2021, 8:34am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/7 "2021-01-25T08:34:38Z")

</div>

Hi @leinad

Insted of path separation take a unique key word from your logs. For eg you can also take host or any filed name or field value but it should be unique.

**if [host] == "10.5.----"**

Regards,  
Tahseen

---

<div class="post-metadata">

**Author:** ![leinad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leinad/32/82281_2.png) [@leinad](https://discuss.elastic.co/u/leinad)\
**Post date:** [February 2, 2021, 11:46pm UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/8 "2021-02-02T23:46:44Z")

</div>

Hi @tahseen_fatima Thanks for your assistance I am stuck with some mappings, I am trying to create a table from array values,  
Basically I have two arrays  
array1[str1,str2,str,3]  
array [val1,val2,val3]  
how to display it in two columns and each row of the table is  
str1,val1  
str2,val2

---

<div class="post-metadata">

**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\
**Post date:** [February 3, 2021, 3:29am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/9 "2021-02-03T03:29:20Z")

</div>

Hi @leinad

I believe your logstash issue has been resolved, so could you open a separate post for this so people have an easier time finding the topic later on.

Regards,  
Tahseen

---

<div class="post-metadata">

**Author:** ![leinad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leinad/32/82281_2.png) [@leinad](https://discuss.elastic.co/u/leinad)\
**Post date:** [February 3, 2021, 3:45am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/10 "2021-02-03T03:45:05Z")

</div>

Hi @tahseen_fatima Yes, I did opened a new post as well thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2021, 3:45am UTC](https://discuss.elastic.co/t/create-multiple-index-for-different-steams-of-data/261203/11 "2021-03-03T03:45:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.