# Want to create multiple index for multiple input **URL:** \ **Category:** Logstash\ **Created:** [August 2, 2017, 1:30pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538 "2017-08-02T13:30:14Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 2, 2017, 1:30pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/1 "2017-08-02T13:30:14Z")
Hi I have multiple log file created based on user. i want to take input all user log file and create index for that. For Ex Input section nput { ``` file { add_field => ["host", "my-dev-host"] path => "D:\JHipster_Demo\adminlogFile.%d{yyyy-MM-dd}.log" codec => "plain" } file { add_field => ["host", "my-dev-host"] path => "D:\JHipster_Demo\unknownlogFile.%d{yyyy-MM-dd}.log" codec => "plain" } ``` Filter section filter { grok { match =\> ["path", "D:/JHipster\_Demo/(?[^]+)/" ] } date { match =\> ["timestamp" , "YYYY/MM/DD:HH:mm:ss Z"] } } output section utput { ``` elasticsearch{ index => "global2-%{project}-%{+YYYY.MM.dd}" hosts => ["localhost:9200"] } ``` stdout { codec =\> rubydebug } } Getting below error [2017-08-02T18:51:57,754][ERROR][logstash.agent] Pipeline aborted due to error {:exception=\>#\, :backtrace=\>["org/jruby/RubyRegexp.java:1434:in `initialize'", "D:/logstash-5.5.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/li b/grok-pure.rb:127:in`compile'", "D:/logstash-5.5.0/vendor/bundle/jruby/1.9/gem s/logstash-filter-grok-3.4.2/lib/logstash/filters/grok.rb:286:in `register'", "o rg/jruby/RubyArray.java:1613:in`each'", "D:/logstash-5.5.0/vendor/bundle/jruby/ 1.9/gems/logstash-filter-grok-3.4.2/lib/logstash/filters/grok.rb:280:in `registe r'", "org/jruby/RubyHash.java:1342:in`each'", "D:/logstash-5.5.0/vendor/bundle/ jruby/1.9/gems/logstash-filter-grok-3.4.2/lib/logstash/filters/grok.rb:275:in `r egister'", "D:/logstash-5.5.0/logstash-core/lib/logstash/pipeline.rb:281:in`reg ister\_plugin'", "D:/logstash-5.5.0/logstash-core/lib/logstash/pipeline.rb:292:in `register_plugins'", "org/jruby/RubyArray.java:1613:in`each'", "D:/logstash-5. 5.0/logstash-core/lib/logstash/pipeline.rb:292:in `register_plugins'", "D:/logst ash-5.5.0/logstash-core/lib/logstash/pipeline.rb:302:in`start\_workers'", "D:/lo gstash-5.5.0/logstash-core/lib/logstash/pipeline.rb:226:in `run'", "D:/logstash- 5.5.0/logstash-core/lib/logstash/agent.rb:398:in`start\_pipeline'"]} [2017-08-02T18:51:57,988][INFO][logstash.agent] Successfully started ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 2, 2017, 1:37pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/2 "2017-08-02T13:37:36Z")
What is `[^]+` supposed to match? `[^x]` means "match anything except x" but you're not saying what x is. Are your files really named D:\JHipster\_Demo\adminlogFile.%d{yyyy-MM-dd}.log? ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 7:19am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/3 "2017-08-03T07:19:26Z")
Hi Thanks For Reply. For the post "How to handle multiple inputs with Logstash to different indices" You said create different topic so created this. My intention to take all log as input from one folder. suppose there are log file based on an user. I want to take input of that log file and create multiple indexe based logfile name. Please suggest me how i will take all input log file any for loop anything? And how i will create index any for loop here? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 8:04am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/4 "2017-08-03T08:04:12Z")
The Logstash configuration language doesn't have loops and I don't understand why you'd need any loops. > My intention to take all log as input from one folder. suppose there are log file based on an user. > I want to take input of that log file and create multiple indexe based logfile name. Yes, that much is clear. The problem is that your grok expression doesn't work. Please answer the questions I asked previously. ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 8:47am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/5 "2017-08-03T08:47:21Z")
I have one folder where all the log file will be created based on user like if there are 10 user 10 log file will be created now if i want to pass as input i cant write input section 10 time. if next time user increases i cant go every time add new input in config file of logstash. So i want something like that the input will take all log file from some folder and create respective indexes. Are you clear now what I want to say. Thanks for your reply. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 9:35am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/6 "2017-08-03T09:35:45Z")
Your current approach is correct but your grok expression is bad. You don't need a loop. Now, last chance to answer my questions: - What is `[^]+` supposed to match? - Are your files really named D:\JHipster\_Demo\adminlogFile.%d{yyyy-MM-dd}.log? ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 9:56am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/7 "2017-08-03T09:56:47Z")
Thanks for your reply. What is [^]+ supposed to match? That i copied from your example Are your files really named D:\JHipster\_Demo\adminlogFile.%d{yyyy-MM-dd}.log? Yes this is filename. what expression i will use so that it will match. nput { ``` file { add_field => ["host", "my-dev-host"] path => "D:\JHipster_Demo\adminlogFile.%d{yyyy-MM-dd}.log" codec => "plain" } ``` "adminlogFile.%d{yyyy-MM-dd}.log" Instead of this what i should write so that it will pick my all log file? grok { match =\> ["path", "D:/JHipster\_Demo/(?[^]+)/" ] } (?[^]+)/" instead of this what i need to use? ``` index => "global2-%{project}-%{+YYYY.MM.dd}" ``` %{project}-%{+YYYY.MM.dd}" instead of this what i need to use? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 11:08am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/8 "2017-08-03T11:08:53Z")
> That i copied from your example What example? > “adminlogFile.%d{yyyy-MM-dd}.log” Instead of this what i should write so that it will pick my all log file? Use a wildcard like \*.log. > (?[^]+)/" instead of this what i need to use? What do you want to match? The whole filename? Just the date? Something else? ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 11:25am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/9 "2017-08-03T11:25:58Z")
Thanks for replying. 1.Now my input is Input { ``` file { add_field => ["host", "my-dev-host"] path => "D:\JHipster_Demo\*.log" /// As it will take all log file as input? codec => "plain" } ``` 1. Now i want to do filter with grok grok { match =\> ["path", "D:/JHipster\_Demo/(?[^]+)/" ] } Here Instead "?[^]+)/" of this what i should use so that it will apply filter on log file. 1. How to create index for all log file based on log file name? First point is clear using \* it will take all log file as input please correct me if am wrong Second and third point how i will do please help me? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 11:37am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/10 "2017-08-03T11:37:55Z")
Please answer my questions. ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 12:27pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/11 "2017-08-03T12:27:10Z")
Which question you talking about? \*one that just an wild character i used in last example where u said create new thread. Please specify your question again? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 12:56pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/12 "2017-08-03T12:56:06Z")
> \*one that just an wild character i used in last example where u said create new thread. I participate in _dozens_ of threads each day and I can't keep track of them all. > Which question you talking about? What do you want to match? The whole filename? Just the date? Something else? ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 1:17pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/13 "2017-08-03T13:17:00Z")
Thanks For your reply. Please see this link from where i added wild character. > [@How to handle multiple inputs with Logstash to different indices](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541): > > Directory Structure: ....Results ....Project1 +....RUN1 +....Run2 ....Project2 +....RUN1 +....Run2 "Results" directory contains Project1 & Project2 sub directories. Also there might be more "Project....n" sub dir gets created depending upon test run for several projects. Each Project DIR contains more that one RUN directories.... I want to process each "Projects" directories as and when they are created and out put them to different indices at elasticsearch. e.g. For Project1 index to… I just want to match with file name .for eg admin.log user1.log for above log file i need to have respective index I hope you understand. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 1:22pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/14 "2017-08-03T13:22:42Z")
To match the name of path component that follows "D:/JHipster\_Demo" you can use this grok expression: ``` D:/JHipster_Demo/(?[^/]+) ``` Then, if the `path` fields contains "D:/JHipster\_Demo/adminlogFile.%d{yyyy-MM-dd}.log" you'll end up getting "adminlogFile.%d{yyyy-MM-dd}.log" in the `filename` field. ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 1:31pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/15 "2017-08-03T13:31:47Z")
Thanks for reply oke so now my input is file { add\_field =\> ["host", "my-dev-host"] path =\> "D:\JHipster\_Demo\*.log" codec =\> "plain" } Filter is grok { match =\> ["path", "D:/JHipster\_Demo/(?[^/]+)" ] //Filter path is same as your am not able type same thing due to editor } output is index =\> "global2-%{filename}-%{+YYYY.MM.dd}" is this correct? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 1:52pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/16 "2017-08-03T13:52:26Z")
> is this correct? Maybe. Try it out. I strongly suggest that you comment out the elasticsearch output first and use a `stdout { codec => rubydebug }` output for verifying that events look as expected. In this case you'd verify that the `filename` field is created and has the expected contents. ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 3, 2017, 3:45pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/17 "2017-08-03T15:45:07Z")
I am not getting any error . But no log details are coming on console.. I think input is not working. file { add\_field =\> ["host", "my-dev-host"] path =\> "D:\JHipster\_Demo\*.log" codec =\> "plain" } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2017, 5:26pm UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/18 "2017-08-03T17:26:47Z")
Make sure you use `D:\JHipster_Demo\*.log` (or `D:/JHipster_Demo/*.log`) as the filename pattern. If it still doesn't work, point to an exact file and make sure you're able to get data from it. If you want to parse these files from the beginning you need to adjust the file input's `start_position` parameter _and_ clear any existing sincedb state. This has been covered here a hundred times before so I will not elaborate. Over and out. ---
**Author:** ![vivekpandey564](https://avatars.discourse-cdn.com/v4/letter/v/858c86/32.png) [@vivekpandey564](https://discuss.elastic.co/u/vivekpandey564)\ **Post date:** [August 4, 2017, 5:06am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/19 "2017-08-04T05:06:01Z")
Yup i am giving exact file name it is working, but if am giving "\*.log" it doent. below point i have not asked to you. If you want to parse these files from the beginning you need to adjust the file input’s start\_position parameter and clear any existing sincedb state. This has been covered here a hundred times before so I will not elaborate. My concern only "\*.log" is not working. As you suggested. Even grokparse is not giving correct name.getting output on console. ``` "path" => "D:\\JHipster_Demo\\logFile.2017-08-04.log", "@timestamp" => 2017-08-04T05:29:21.985Z, "@version" => "1", "host" => [ [0] "FS-WS195-D45", [1] "my-dev-host" ], "message" => " 2017-08-04 10:59:21,907 INFO [metrics-logger-reporter-1-t ``` hread-1] metrics: type=TIMER, name=com.hexaware.rad.web.rest.UserResource.update User, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95= 0.0, p98=0.0, p99=0.0, p999=0.0, mean\_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate\_un it=events/second, duration\_unit=milliseconds\r", "tags" =\> [ [0] "\_grokparsefailure" ] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 1, 2017, 5:06am UTC](https://discuss.elastic.co/t/want-to-create-multiple-index-for-multiple-input/95538/20 "2017-09-01T05:06:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.