# How to handle multiple inputs with Logstash to different indices **URL:** \ **Category:** Logstash\ **Created:** [June 29, 2015, 11:21am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541 "2015-06-29T11:21:02Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [June 29, 2015, 11:21am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/1 "2015-06-29T11:21:02Z")
_Directory Structure:_ _....Results_ _....Project1_ _+....RUN1_ _+....Run2_ _....Project2_ _+....RUN1_ _+....Run2_ "Results" directory contains Project1 & Project2 sub directories. Also there might be more "Project....n" sub dir gets created depending upon test run for several projects. Each Project DIR contains more that one RUN directories.... **I want to process each "Projects" directories as and when they are created and out put them to different indices at elasticsearch.** **e.g. For Project1 index to be set as "Logstash-Project1-%{+YYYY.MM.dd}", similarly for Project2 as "Logstash-Project2-%{+YYYY.MM.dd}" and so on.** How do I handle input/ output.... input { file { path =\> "/Results/Project\*/RUN\*/\*.csv" start\_position =\> "beginning" } } output { elasticsearch { action =\> "index" host =\> "localhost" index =\> "logstash-%{+YYYY.MM.dd}" ``` } ``` } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 29, 2015, 12:54pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/2 "2015-06-29T12:54:27Z")
Use the [grok filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) to extract the project name from the input file path (stored in the `path` field), then reference that field when setting the index pattern of the [elasticsearch output](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html). ``` elasticsearch { ... index => "logstash-%{project}-%{+YYYY.MM.dd}" } ``` ---
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [June 29, 2015, 2:09pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/3 "2015-06-29T14:09:55Z")
I dont know how to use grok filter to extract the project name from the input file path. What pattern i need to match for extracting Project from path. Any help? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 29, 2015, 2:21pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/4 "2015-06-29T14:21:05Z")
If you want to extract the first directory component below a directory named Results, this is untested but should work: ``` grok { match => ["path", "/Results/(?[^/]+)/"] } ``` ---
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [June 29, 2015, 3:13pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/5 "2015-06-29T15:13:52Z")
thank you so much. you saved my day... ---
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [June 29, 2015, 3:55pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/6 "2015-06-29T15:55:53Z")
Sorry to disturb you again. But got to know that file path is something like path =\> "/opt/Results/\*/Run/webobj.csv" Grok filter which i want to use for extracting the project name is from /\*/ of path. Can you please suggest appropriate match for getting the \* value immediate after "Results/\*" directory. Is there any way from where I can generate grok pattern to match. Your help is much appreciated. Thanks again. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 30, 2015, 3:34am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/7 "2015-06-30T03:34:33Z")
The input file path has nothing to do with the `path` field. Its value is taken from the name of the actual file from which a particular log message actually came. ---
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [June 30, 2015, 5:45am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/8 "2015-06-30T05:45:34Z")
Great Help I could do it... grok { match =\> ["path", "/opt/Log/Results/(?[^/]+)/" ] } Thanks ---
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [July 2, 2015, 8:43am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/9 "2015-07-02T08:43:05Z")
In case of linux it worked so I just thought of to run on windows using grok { match =\> ["path", "C:\Test\Result(?[^/]+)"] } This didn't work. any idea why? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 2, 2015, 9:37am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/10 "2015-07-02T09:37:28Z")
Backslashes are metacharacters in regexps so to get match literal backslashes you need to use "\\". ---
**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Post date:** [July 2, 2015, 10:06am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/11 "2015-07-02T10:06:42Z")
grok { match =\> ["path", "C:\Test\Result\(?[^\]+)\"] } When i run logstash with above config (As you suggested) I get Error: Expected one of #, {, ,,] at line .... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 2, 2015, 2:08pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/12 "2015-07-02T14:08:41Z")
It looks like you're escaping the closing double quote. This is what you need: ``` grok { match => ["path", "C:\\Test\\Result\\(?[^\]+)\\"] } ``` ---
**Author:** ![vasumathy](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@vasumathy](https://discuss.elastic.co/u/vasumathy)\ **Post date:** [June 28, 2016, 12:34am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/13 "2016-06-28T00:34:26Z")
I am using logstash-2.3.2. "type" option is not working input { jdbc { jdbc\_driver\_library =\> .. jdbc\_driver\_class =\>.. jdbc\_connection\_string =\>.. jdbc\_user =\> .. jdbc\_password =\> .. statement =\> .. type =\> "deploy" } } output { if [type]=="deploy"{ elasticsearch { hosts =\> ["localhost:9200"] index =\> "metricslog" } } } ---
**Author:** ![RandyChen](https://avatars.discourse-cdn.com/v4/letter/r/48db29/32.png) [@RandyChen](https://discuss.elastic.co/u/RandyChen)\ **Post date:** [August 11, 2016, 8:24am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/14 "2016-08-11T08:24:50Z")
I find the grok match can not enable the "path" and "message" work well together. like this : match =\> { "message" =\> "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" "path" =\> "/logfile/(?\w+?)/" } the "message" will run fail ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 11, 2016, 8:26am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/15 "2016-08-11T08:26:30Z")
I suspect you'll want to split that grok filter in two to make sure both expressions are always evaluated. ---
**Author:** ![Apache\_HOU](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Apache\_HOU](https://discuss.elastic.co/u/Apache_HOU)\ **Post date:** [February 27, 2017, 1:01pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/16 "2017-02-27T13:01:26Z")
@magnusbaeck Hi Magnus, First of all, thanks for your previous posts about index. I've tried your suggestion but I don't kown what i did wrong as it does' work for me... (Please inform me if you prefer opening a new topic) I've 2 log files who stored in the directories /tmp/toto/first/ and /tmp/toto/second/ . I want to have the different index name distinguished by the project name (first and second in this case). Here are my configurations : Filebeat : ``` ... paths: - /tmp/toto/*/*.log ... ``` Logstash ``` input { beats { port => "5043" } } filter { grok { match => { "path" => "/tmp/toto/(?[^/]+)/" } match => { "message" => "%{COMBINEDAPACHELOG}"} } date { match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] } } output { elasticsearch { hosts => ["127.0.0.1:9200"] index => ["log-%{project}-%{+YYYY.MM.dd}"] } } ``` After starting all services, it seems elasticsearch doesn't understand `project` variable : ``` health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open log-%{project}-2016.06.20 QNPYvvFqRzGFEC9_32da7g 5 1 450 0 807.6kb 807.6kb yellow open log-%{project}-2017.02.22 RzDgOdWKQXqmItnm0zNdGw 5 1 232 0 138.5kb 138.5kb ``` Do you have any ideas ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 27, 2017, 1:08pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/17 "2017-02-27T13:08:36Z")
It looks like the event doesn't have a `project` field. Check what the events in the log-%{project}-2017.02.22 index look like. ---
**Author:** ![Apache\_HOU](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Apache\_HOU](https://discuss.elastic.co/u/Apache_HOU)\ **Post date:** [February 27, 2017, 3:21pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/18 "2017-02-27T15:21:54Z")
@magnusbaeck Here is an example of output. I think you're right, i don't have the \> project fiield. Do you know how to correct it ? ``` { "_index": "log-%{project}-2017.02.22", "_type": "log", "_id": "AVqAEzYEO-tgZd20LulR", "_score": null, "_source": { "request": "/", "agent": "\"Links (1.03; Linux 2.6.32-642.6.2.el6.x86_64 x86_64; dump)\"", "offset": 172883, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/tmp/toto/first/access_20170222.log", "message": "10.124.49.22 - - [22/Feb/2017:23:00:02 +0100] \"GET / HTTP/1.1\" 404 2916 \"-\" \"Links (1.03; Linux 2.6.32-642.6.2.el6.x86_64 x86_64; dump)\" ", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": "\"-\"", "@timestamp": "2017-02-22T22:00:02.000Z", "response": "404", "bytes": 2916, "clientip": "10.124.49.22", "@version": "1", "beat": { "hostname": "new", "name": "new", "version": "5.2.1" }, "host": "new", "httpversion": "1.1", "timestamp": "22/Feb/2017:23:00:02 +0100" }, "fields": { "@timestamp": [ 1487800802000 ] }, "sort": [ 1487800802000 ] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 27, 2017, 3:31pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/19 "2017-02-27T15:31:35Z")
We've digressed from the original topic. Please start a new thread for your question. (Hint: Where's the `path` field you're attempting to parse with grok?) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:28am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541/20 "2017-07-06T04:28:15Z")