Has anyone made a "serverless" elasticsearch?

Elastic Stack Elasticsearch
1

hello,

I'm a big fan of ES and have been using it for years now. Just wondering if anyone has made an attempt to make a serverless approach to ES?

Let me expand on what I mean:
I personally use Elasticsearch for a project, the problem is I'm paying to have servers run 24 hours a day for basically very infrequent searches (maybe a few an hour).

Is there is a version/plugin/library/service for Elasticsearch which simply caches the configuration for Elasticsearch, quickly mounts the storage disk drive, and boots up the ES server everytime a query is made?

I store the data on SSD elastic block storage on AWS anyways, but i could really save costs if I only paid for servers when a user is actually searching or if I could pay PER request like AWS Lambda. I pay far more for servers to even run the elasticsearch server than I do for the SSD storage with data ES has already indexed.

I'm open to even making an open source project about this, but open to everyone's thoughts on this?

Thanks in advance would love to hear feedback,

2

I don't have answer but do you know that cloud.elastic.co has cluster starting from less than 15$ per month?

3

Thanks for sharing David :smiley: let me take a look.

4

I don't know of one, but it's an interesting idea. The Elasticsearch server process has nontrivial startup costs which you would be paying per-search. It's certainly possible to do this: indeed much of the test suite starts up some nodes, waits for them to be ready, runs some commands, and then shuts them down again. You can run Elasticsearch in Docker containers so maybe you can get these to run on-demand most simply?

I do, however, wonder about the payoff. You wouldn't have to be doing very many searches per hour before it became cheaper to keep Elasticsearch running the whole time.

You would then need to load all the shards for the search, which is also nontrivial. Normally Elasticsearch keeps a lot of information in cache, or otherwise pre-computed in memory, so you would have to warm all this up every time. Yet there is a feature that does exactly this: frozen indices allows you to load shards on demand as needed for searches, and discard them later, to avoid consuming too much heap all at once. Perhaps you could freeze your indices and move to cheaper instances with less RAM?

Maybe consider moving to "Cold HDD" EBS volumes, which are priced comparably with S3 per GB.

To be honest I think it's going to be tough to beat Elastic Cloud's pricing, but if you fancy the challenge then all power to you :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

6

@hacker_21 you have been heard!

1 Like