# Is it possible to create multiple index with multiple folder name **URL:** \ **Category:** Logstash\ **Created:** [December 17, 2022, 3:35pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457 "2022-12-17T15:35:23Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 3:35pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/1 "2022-12-17T15:35:23Z")
Am monitoring logs files in different folders. I have 3 folders, folder1,folder2,folder3. Is it possible to Create different index name with name of folders using logstash.? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 17, 2022, 4:49pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/2 "2022-12-17T16:49:27Z")
Yes, it is possible. Use gsub to extract folder name, then add in [@metadata][dir] The best is to provide with dir name samples. ```auto output { elasticsearch { hosts => ["http://localhost:9200"] index => "indexname_%{[@metadata][dir]}" } } ``` Other possibly is to set with IFs which is more strict and more usable in some cases. ```auto output { if ([@metadata][dir]=="dir1"){ elasticsearch { hosts => ["http://localhost:9200"] index => "indexname_%{[@metadata][dir]}" } } else if ([@metadata][dir]=="dir2"){ elasticsearch { hosts => ["http://localhost:9200"] index => "indexname_%{[@metadata][dir]}" } } else if ([@metadata][dir]=="dir3"){ elasticsearch { hosts => ["http://localhost:9200"] index => "indexname_%{[@metadata][dir]}" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 17, 2022, 4:53pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/3 "2022-12-17T16:53:13Z")
You can use grok to extract a directory name from a path. Examples are [here](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/9) and [here](https://discuss.elastic.co/t/extracting-particular-folder-from-the-path-and-adding-that-to-a-field/142277/4). Depending on your versions and ECS compatibility the path of the file will likely be in [path] or, more recently [log][file][path] ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 4:57pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/4 "2022-12-17T16:57:04Z")
Hi buddy, thanks for your reply. My directories structure /apps/app1/logs/SystemOut.log apps/app2/logs/SystemOut.log apps/app3/logs/SystemOut.log I tired with following pipeline. But its not working input { file { path =\> "/folder1/logs/server1/SystemOut.log" start\_position =\> "beginning" } } filter { if [path] == "/folder1/logs/server1/SystemOut.log" { grok { match =\> { "message" =\> [ "[%{DATA:timestamp1}]%{SPACE}%{WORD:value1}%{SPACE}%{WORD:value2}%{SPACE}%{WORD:value3}%{SPACE}%{TIMESTAMP\_ISO8601:timestamp2}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{WORD:value4}:%{NUMBER:value5}%{SPACE}-%{SPACE}%{GREEDYDATA:value6}" ] } } } } output { if [path] == "/folder1/logs/server1/SystemOut.log" { opensearch { hosts =\> ["[https://url:443](https://url:443)"] index =\> "ee" user =\> "in" password =\> "dm3" } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 17, 2022, 4:57pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/5 "2022-12-17T16:57:04Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 4:58pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/6 "2022-12-17T16:58:36Z")
Hi buddy, thanks for your reply. My directories structure /apps/app1/logs/SystemOut.log apps/app2/logs/SystemOut.log apps/app3/logs/SystemOut.log I tired with following pipeline. But its not working input { file { path =\> "/folder1/logs/server1/SystemOut.log" start\_position =\> "beginning" } } filter { if [path] == "/folder1/logs/server1/SystemOut.log" { grok { match =\> { "message" =\> [ "[%{DATA:timestamp1}]%{SPACE}%{WORD:value1}%{SPACE}%{WORD:value2}%{SPACE}%{WORD:value3}%{SPACE}%{TIMESTAMP\_ISO8601:timestamp2}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{WORD:value4}:%{NUMBER:value5}%{SPACE}-%{SPACE}%{GREEDYDATA:value6}" ] } } } } output { if [path] == "/folder1/logs/server1/SystemOut.log" { opensearch { hosts =\> ["[https://url:443](https://url:443)"] index =\> "dev-logs" user =\> "in" password =\> "dm3" } } ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 5:08pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/7 "2022-12-17T17:08:40Z")
Orelse Is it possible to add tag in new field with folder name??? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 17, 2022, 5:41pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/8 "2022-12-17T17:41:34Z")
This has been working on my simplified version with files as output. Try something like this: ```auto input { file { path => "/apps/app*/logs/SystemOut.log" start_position => beginning sincedb_path => "/dev/null" # change to keep records or do not use NUL in prod mode => "tail" } } filter { grok { match => { "[log][file][path]" => "%{GREEDYDATA:dir}/%{DATA:[@metadata][dest]}/%{DATA}/%{GREEDYDATA}" } } } output { stdout {codec => rubydebug { metadata => true} } # test to see results, latter remove or comment elasticsearch { hosts => ["http://localhost:9200"] index => "indexname_%{[@metadata][dest]}" } } ``` ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 5:47pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/9 "2022-12-17T17:47:33Z")
Thanks rios, let me try this and update here. One more question in this. Is it possible to add tag or extract a new field with folder name.? If you give any example for this i will try that one also. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 17, 2022, 5:53pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/10 "2022-12-17T17:53:14Z")
Yes, but I don't see point of tag because every tag will be recorded in index, except if you want 1 single index with extra tag or field to know the source, for example app1 directory. Save the space in big data 🙂 use @metadata ```auto mutate { add_tag => ["%{[@metadata][dest]}" ] } ``` ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 6:02pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/11 "2022-12-17T18:02:19Z")
Might be i have added wrong question, right now i have one field name "path" ( /apps/folder1/logs). My question is, can we add that "folder1" name in new filed??? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 17, 2022, 6:12pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/12 "2022-12-17T18:12:03Z")
```auto mutate { add_field => { "%{[@metadata][dest]}" => "%{[@metadata][dest]}" # v1 "source_%{[@metadata][dest]}" => "%{[@metadata][dest]}" # v2 } } ``` or do not use temp [@metadata][dest], use just source: %{DATA:source} ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 17, 2022, 6:13pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/13 "2022-12-17T18:13:53Z")
Let me try rios, thanks mate. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 17, 2022, 6:16pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/14 "2022-12-17T18:16:04Z")
Also you can use [this](https://discuss.elastic.co/t/split-filepath-to-a-new-field/319657/2) to avoid grok, rarely rare cases. ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 18, 2022, 5:00am UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/15 "2022-12-18T05:00:21Z")
may i try like this ?? input { file { path =\> "/folder1/logs/server1/SystemOut.log" start\_position =\> "beginning" sincedb\_path =\> "nul" } } filter { if [path] == "/folder1/logs/server1/SystemOut.log" { grok { mutate { copy =\> { "[log][file][path]" =\> "[@metadata][path]"}} mutate { split =\> { "[@metadata][path]" =\> "/" } } mutate { add\_field =\> { "folder1" =\> "%{[@metadata][path][4]}"}} } } } output { opensearch { hosts =\> ["[https://url:443](https://url:443)"] index =\> "dev-logs" user =\> "" password =\> "" } } ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 18, 2022, 9:05am UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/16 "2022-12-18T09:05:16Z")
> [@guhi\_rockky](#): > > if [path] == "/folder1/logs/server1/SystemOut.log" { This will not work because it is [log][file][path], not just path `if [log][file][path] == "/folder1/logs/server1/SystemOut.log"` But...since you have change the position in path, should use this: ```auto input { file { path => "/folder1/logs/server1/SystemOut.log" start_position => "beginning" sincedb_path => "nul" } } filter { grok { match => { "[log][file][path]" => "%{GREEDYDATA:dir}/%{DATA:[@metadata][dest]}/%{GREEDYDATA}" } } mutate { add_field => { "folder1" => "%{[@metadata][dest]}"}} } output { opensearch { hosts => ["https://url:443"] index => "dev-logs" } } ``` ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 18, 2022, 10:24am UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/17 "2022-12-18T10:24:36Z")
Thanks buddy, its working fine. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 18, 2022, 11:43am UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/18 "2022-12-18T11:43:01Z")
Good. 👍 1. "NUL" on Windows or "/dev/null" on Linux it to avoid tracking. Set a file, to keep record which lines has been read otherwise you might have duplicated documents. Set something like this: _sincedb\_path =\> "/path/file.db"_ 2. Remove dir from grok, it's useless, I left from old code. _grok { match =\> { "[log][file][path]" =\> "%{GREEDYDATA}/%{DATA:[@metadata][dest]}/%{GREEDYDATA}" } }_ 3. Debugger not need, remove the line _stdout {codec =\> rubydebug_... ---
**Author:** ![guhi\_rockky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guhi_rockky/32/114849_2.png) [@guhi\_rockky](https://discuss.elastic.co/u/guhi_rockky)\ **Post date:** [December 18, 2022, 2:43pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/19 "2022-12-18T14:43:05Z")
Sure rios, let me do like this ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 15, 2023, 2:43pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/20 "2023-01-15T14:43:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.