Elastic Search for Concurrent queries from user mobile applications

[mnallath (Murugesan)]

Hi,
We are planning to use Elastic search in our cloud application project. We are planning to use this for the below purposes

1. Logging - for debugging the cloud application
2. Reports generation based on log – For management reports and do little bit analytics
3. History/Alarms - For our cloud application users

We want your expertise suggestion for the item# 3 when use Elastic search for this purpose.
For this item, number of concurrent queries will be high (say in the order of thousands) with complex queries like multiple terms search, pagination, sorting which need to do minimum 30 indexes . As this query exposed to end user mobile application, query time also shall be faster.

In general, Is ELK designed to handle many queries at the same time ? or it is mainly for offline processing with minimal user ? Do you we need to allocate more cpu and memory to achieve this ? or query is not that intensive ?

PS :

1. We already have Cassandra database in our application. So we want to know Which one is best architecture method for the item#3 . Should keep it in Cassandra or Elastic search database ?

Thank you for your time

[dadoonet (David Pilato)]

Is ELK designed to handle many queries at the same time ?

Yes. If you hit at some point a limit, you can increase the number of replicas. All primaries and replicas are serving searches.

Then have a look at some elasticsearch users: Github (I believe many users are searching at the same time), Wikipedia, Uber...

query is not that intensive ?

It depends. Really. It depends on what you are doing and how. So you need to test your scenario.

[mnallath (Murugesan)]

Thank you. one more query related to this. In my case

1. Sharding key will be user id and
2. Store data for "last" 30 days ( it is not monthly , last 30 days )
3. have to provide search with any time range - say from feb 27, 3.00 pm to feb 1,2.00 pm
4. number of records per day per user is usually ~150
   5.Number of user will be around 50000

In this case, how we should create our index ? if it is daily index then we may need to search in 30 indexes (maximum) , will it not create any performance impact ? if we have many Elastic search node then we can achieve parallel query but what about if we have just 3 elastic node/machine ?

[nik9000 (Nik Everett)]

That number of documents per day is super small. I think you will be fine if you make a new index per day so you can drop the index after 30 days. Stick all the data from all users into that index. I think using user_id as the routing key would be over engineering things so I wouldn't do it without proof it helps.

It all sounds fine. If you need to scale read load add more nodes and more replicas. That is super common and easy to do.