We have been integrating an ElasticSearch log writer into Bro network
monitor (http://www.bro-ids.org) and we have a few users that are
monitoring extremely high volume networks and want to insert their logs
into ElasticSearch but their logging rate will hover around 40k-50k
documents per second for relatively long periods of time. We are already
doing index rotation which has been nice for expiring old data and with
searching constrained time periods but I suspect there is more we
could/should be doing.
Are there any tuning guides available for techniques we could be using to
insert documents at high rates?