I am currently working on an ELK stack where we have a large number of documents (~50k in 1min) coming into our cluster. All the documents are coming from a bunch of log sources with a large number of fields. Currently all the documents go into a single index pattern because of which queries on the data take a long time to respond.
Now, we have identified particular fields(eg: username, IP) in all log sources which are prominently used for querying the data.
- I wanted to know how can I properly index the documents (using the above information )so that the response time is improved. (I would like to improve on this part more for our case.)
- Also I was thinking of creating different indices for the various log sources which should help in improving the speed.