I am working on a Business Intelligence projects, it has a large number of records, 4.6 million companies and 80 million directors. Currently I am replacing the full text queries with Elastic Search ones.
Once this is done, I need to cache the web pages, the data only changes in frequently e.g once every 12 months or so, so the page itself is best being cached rather than the database.
What is the best cache solution? Currently the server uses Varnish but this only caches the non dynamic page elements so actually doesn't help much in this scenario.
I've looked at fastcgi which seems suitable but I am not familar with this area only that caching is needed.
What is the best web cache solution in this situation?
It does not sound like a very large data from an Elasticsearch point if view so I am sure it would be able to handle a significant request volume. What kind of traffic are you expecting that makes it necessary to cache results?
Our main competitor sees around 200,000 page views per day.
Currently the script is "dog slow" but that is mostly a combination of the current absence of lucene and the number of queries on a page.
Load time is very important from an SEO point of view, our site suffers from the Google Crawl budget which means we have a limited amount of resource allocated to crawling our pages so we need to be able to deliver as many as possible as fast as possible.
It therefore makes sense that a cached html page will be far faster for a server to deliver than even elastic search can create etc
Caching dynamic content like the results of search results is tricky, so I would set it up without caching and see how it performs. I suspect you may not need to add caching if you tune your searching.
Yes that is the approach I am taking, but anticipate that there will be benefits by even minor speed increases.
Here is an example of the type of query that I am talking about
The results are based on a series of criteria such as location, industry sector and similarity of company name.
They are ideal to cache as they consume resource to create each time, but there is no reason to repeat the query when the whole page itself could be cached.
My server has 24 cores and 64Gig of RAM and is completely overloaded at the moment, hence the move to improve performance using Elastic Search.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
