Hi -- we're about to replace a lot of our search and index queries with
elasticsearch, but were also thinking about having it serve get requests by
id on items.
To do this, we'd need to keep the search index totally up-to-date with the
database, which may involve a lot of writes.
We estimate these (research ongoing) to be < 5000 per day, but as we mainly
operate in the UK, those will cluster around UK daylight hours.
Would elasticsearch comfortably handle that amount of index writing?
And at what point do people think we'd need to start worrying about pulling
the updates to one side to perform as a bulk operation?
Thanks for any & all advice,
Doug.
5000 is not a lot for any configuration. In one of my projects,
elasticsearch continuously indexes (using a bulk indexer for performance
reasons, but not really necessary) around 75K documents a day on an EC2
microinstance (613MBs total of memory/300MBs allocated toward the JVM). For
another project, I was incorrectly not using a bulk indexer when batch
updating, resulting in 50K+ continuous inserts. Once again, no issues.
--
Ivan
On Fri, Aug 12, 2011 at 7:01 AM, doug livesey biot023@gmail.com wrote:
Hi -- we're about to replace a lot of our search and index queries with
elasticsearch, but were also thinking about having it serve get requests by
id on items.
To do this, we'd need to keep the search index totally up-to-date with the
database, which may involve a lot of writes.
We estimate these (research ongoing) to be < 5000 per day, but as we mainly
operate in the UK, those will cluster around UK daylight hours.
Would elasticsearch comfortably handle that amount of index writing?
And at what point do people think we'd need to start worrying about pulling
the updates to one side to perform as a bulk operation?
Thanks for any & all advice,
Doug.
That's excellent news to hear, thankyou! 
On 12 August 2011 14:59, Ivan Brusic ivan@brusic.com wrote:
5000 is not a lot for any configuration. In one of my projects,
elasticsearch continuously indexes (using a bulk indexer for performance
reasons, but not really necessary) around 75K documents a day on an EC2
microinstance (613MBs total of memory/300MBs allocated toward the JVM). For
another project, I was incorrectly not using a bulk indexer when batch
updating, resulting in 50K+ continuous inserts. Once again, no issues.--
IvanOn Fri, Aug 12, 2011 at 7:01 AM, doug livesey biot023@gmail.com wrote:
Hi -- we're about to replace a lot of our search and index queries with
elasticsearch, but were also thinking about having it serve get requests by
id on items.
To do this, we'd need to keep the search index totally up-to-date with the
database, which may involve a lot of writes.
We estimate these (research ongoing) to be < 5000 per day, but as we
mainly operate in the UK, those will cluster around UK daylight hours.
Would elasticsearch comfortably handle that amount of index writing?
And at what point do people think we'd need to start worrying about
pulling the updates to one side to perform as a bulk operation?
Thanks for any & all advice,
Doug.