Hi,
I am looking at the possibility of using ES without a database, constructing my data from my python application and sending it straight to ES in real time. It says me a lot of complexity, however my concern is that I might be generating data very quickly and sending requests relentlessly, even when ES might not be ready to accept it.
My question is, in this case does it makes sense to use a queue system as a buffer between the two, so my application sends everything to a queue, and then queue tries to add it to ES, retrying if it doesn't have success.
I am not sure if this is the most logical or efficient method. If anyone has any information or ideas on what queue systems would be suited, or if I even need one, I'd be very interested to hear.
James