Dynamic Indexing based on user input

Elastic Stack Logstash
1

I am developing an application using node.js (an custom interface) to visualize logs.
There are two issues that I am facing:

  1. How to dynamically change the index based on user input???

Through the interface, user would be able to decide/change the index name under which the logs will be stored.

  1. How to know whether logstash has completely processed the logs till the end???

Once the logs are processed, I want the logs to be deleted. I don't want the logstash to continuously monitor the logs.

Please note that logstash application would already/always be running in the background.

Any help would be appreciated... :slight_smile:

2
  1. How to dynamically change the index based on user input???

You could generate a Logstash configuration file and feed it to Logstash. You could also have a static configuration file and use the possibility to reference environment variables inside it.

  1. How to know whether logstash has completely processed the logs till the end???

You'll have to monitor the sincedb file. You could also use the stdin input but then the process won't be restartable, i.e. if Logstash is interrupted you'll have the process the input file all over again. That might not be a problem with your use case.

3

How shall i do it ???? (I am new to logstash :slight_smile: )
I may need to add multiple files at once to logstash to read the logs.

4

See below.

5

Thanks a lot.

Since in windows, it is not possible to get inode number of a file, so it won't be possible to track multiple files simultaneously.

Possible workaround: I would let the logstash process the file one by one. When a particular file is processed (this could be tracked using the value of 'offset' i.e. 4th column in .sincedb file) , I would delete that particular file and clear the contents of .sincedb file.

6

You could use different file inputs with different sincedb_path values.

7

I can't use different file inputs because in config file, log file name conforms to a specific name pattern (this implies, any number of files can be uploaded/processed at a time). So, I can't have different file inputs in logstash config file.

Example:
file {
type => "type_log"
path => "C:/logfiles/logmessages.log*"
sincedb_path => "C:/logfiles/.sincedb123"
}