Memory requirements for mobile app

[mackrispi]

Hello,

I would really like to ask you for a little help …

As a startup project,paid from my own pocket, I would like to integrate fulltext search capabilities into my mobile app.

I use Mysql 5.7.19 on CentOS inside Amazon cloud. I already use mysql fulltext index but was said that myslq can be slow if there are a few million rows …

If you take into consideration that:

• you only search on one column that is "the title"…
• that title column is max Varchar(100) characters/bytes

And if I’m very optimistic, I would expect someday to have from 1milion-100milion rows but not in the next 2 years …. not even 1million in 1 year...

Please correct me if my calculation is wrong :

than that would mean that for 1 million rows where every row takes max 100bytes …. this would be like 100MB of storage/memory

and for 100 million like 10Gb ….

Of course for only this column …. plus what ES takes for other stuff …

But in the beginning where there is almost nothing in the db, and I would expect maybe like 100-1000 rows per week I probably don’t need nor can’t afford
a machine with 32-64Gb of memory as EC2 instance inside Amazon.

I was thinking like 8-16Gb max for the beginning …..

Let me just say that I've looked at alternatives like SOLR,SPHINX but I’m no where near system administrator to be able to maintain them so ES inside Amazon looks like a better or should I say safer way for me especially now in the beginning …. and PHP is my only communication option …

At first SPHINX seams like an ideal solution for me cause I already have mysql as datastore so just need an index but then I saw and read that they have some internal organisation problems and came across posts like this:
Warning!!! Re-indexing can take many hours for a large database. Full text search will not be available during this period but other services will not be affected. This process is CPU intensive.

And I can not afford to be unavailable for so long….

How would you guys set up ES?

Any hint is highly appreciated ...

Thank you for all the comments
Best regards,
Kris

[dadoonet (David Pilato)]

And if I’m very optimistic, I would expect someday to have from 1milion-100milion rows but not in the next 2 years …. not even 1million in 1 year...

Cool thing is that elasticsearch can scale out as your activity grows.

han that would mean that for 1 million rows where every row takes max 100bytes …. this would be like 100MB of storage/memory

You can't really say that with elasticsearch without testing it. It depends on so many factors...

I was thinking like 8-16Gb max for the beginning …..

Even less could be largely enough to start. As I said, if you business grows, then just start a new instance (bigger may be), let the data replicate, and kill the old one.

Let me just say that I've looked at alternatives like SOLR,SPHINX but I’m no where near system administrator to be able to maintain them so ES inside Amazon looks like a better or should I say safer way for me especially now in the beginning …. and PHP is my only communication option …

You need to compare things but do you know that you can have a cloud.elastic.co instance for 45$/month? 1gb memory for 24gb of data. May be it could fit with your budget?

Cool thing is that everything is included, including security, upgrades in one click, automatic backup of your data, Kibana...

So for your needs, my advice are:

• if you can afford it, go to cloud.elastic.co
• if not, start a 2gb or 4gb machine and install and manage all that by yourself.

My 2 cents.

Disclaimer: I'm working at elastic.

[mackrispi]

Hello David

Thank you so much for your reply.....

You need to compare things but do you know that you can have a cloud.elastic.co instance for 45$/month? 1gb memory for 24gb of data. May be it could fit with your budget?

Cool thing is that everything is included, including security, upgrades in one click, automatic backup of your data, Kibana...

Does this mean inside Amazon ? Cause my system is made out of 5 EC2 instances connected with each other , some are web servers, some Mysql servers and some replicas and one Global/central server ....
So ES would me just one of them tu support fulltext search ....

Thank you.
You helped me a lot!

Best regards,
Kris

[dadoonet (David Pilato)]

It can run on AWS or Google Compute Platform.