# Dynamic Indexing based on user input

**URL:** <https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238>\
**Category:** Logstash\
**Created:** [October 5, 2016, 7:37am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238 "2016-10-05T07:37:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\
**Post date:** [October 5, 2016, 7:37am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/1 "2016-10-05T07:37:51Z")

</div>

I am developing an application using node.js (an custom interface) to visualize logs.  
There are two issues that I am facing:

1. How to dynamically change the index based on user input???

Through the interface, user would be able to decide/change the index name under which the logs will be stored.

1. How to know whether logstash has completely processed the logs till the end???

Once the logs are processed, I want the logs to be deleted. I don't want the logstash to continuously monitor the logs.

Please note that logstash application would already/always be running in the background.

Any help would be appreciated... 🙂

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 5, 2016, 7:47am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/2 "2016-10-05T07:47:16Z")

</div>

> 1. How to dynamically change the index based on user input???

You could generate a Logstash configuration file and feed it to Logstash. You could also have a static configuration file and use the possibility to reference environment variables inside it.

> 1. How to know whether logstash has completely processed the logs till the end???

You'll have to monitor the sincedb file. You could also use the stdin input but then the process won't be restartable, i.e. if Logstash is interrupted you'll have the process the input file all over again. That might not be a problem with your use case.

---

<div class="post-metadata">

**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\
**Post date:** [October 5, 2016, 11:13am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/3 "2016-10-05T11:13:08Z")

</div>

> [@magnusbaeck](#):
>
> You'll have to monitor the sincedb file.

How shall i do it ???? (I am new to logstash 🙂 )  
I may need to add multiple files at once to logstash to read the logs.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 5, 2016, 11:14am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/4 "2016-10-05T11:14:50Z")

</div>

See below.

> <https://stackoverflow.com/questions/33143437/logstash-parsing-progress-bar>

---

<div class="post-metadata">

**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\
**Post date:** [October 5, 2016, 11:46am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/5 "2016-10-05T11:46:13Z")

</div>

Thanks a lot.

Since in windows, it is not possible to get inode number of a file, so it won't be possible to track multiple files simultaneously.

**Possible workaround** : I would let the logstash process the file one by one. When a particular file is processed (this could be tracked using the value of 'offset' i.e. 4th column in .sincedb file) , I would delete that particular file and clear the contents of .sincedb file.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 5, 2016, 11:58am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/6 "2016-10-05T11:58:12Z")

</div>

You could use different file inputs with different `sincedb_path` values.

---

<div class="post-metadata">

**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\
**Post date:** [October 5, 2016, 12:17pm UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/7 "2016-10-05T12:17:48Z")

</div>

I can't use different file inputs because in config file, log file name conforms to a specific name pattern (this implies, any number of files can be uploaded/processed at a time). So, I can't have different file inputs in logstash config file.

Example:  
file {  
type =\> "type\_log"  
path =\> "C:/logfiles/logmessages.log\*"  
sincedb\_path =\> "C:/logfiles/.sincedb123"   
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:35am UTC](https://discuss.elastic.co/t/dynamic-indexing-based-on-user-input/62238/8 "2017-07-06T04:35:40Z")

</div>


