We are facing a strange and critical issue while testing elastic-search in high speed transaction monitoring.
We are using Elastic-search for dumping high speed data coming from JMX MXbean. Our JMX listeners are getting objects from MXbean and using Elastic search High Level Java REST API, we are sending these objects (as JSON format) for indexing. Note : We are using Bulk insert with 500 document at once.
With single JMX object, its working good, But when we start load tests, first few iterations for 2 min (with about 150K documents) goes fine, and after sometime we will see Listeners are receiving more data but Document inserted in elastic-search are quite less. For example, I sent 150K Documents, but at the end of Load test, the document count in elastic search shows only 148K. There is no error i see in logs.
Any one faced this issue? How can we resolve this?
I saw, good design will be to dump into DB first and then from there using Log-stash we can get or index those documents. Is this only option left with me?