Are there any known compatibility issues between rest client v5.5.0 trying to index data into elasticsearch 5.6.2? We have been seeing, intermittently, indexing taking more than 10s.
We have our app that is using elastic rest client v5.5.0 and the cluster has recently been upgraded to v5.6.2.
Merges, GC, refreshes, CPU increases.
It's hard to say because this could be anything, but look for anomalies around the time the longer requests take.
no... our use case is more like a realtime update... so no we are not using _bulk.
But our rate of writes is very low... and this is happening in a test env where we are writing data sequentially... think about an integration test running... indexing a record, reading it and then deleting it... the record size is about 1-2 KB.
We are using ES rest client v5.5.0. But the cluster is at v5.6.2 - This is a pre-production test cluster.
When do indexing, immediately after that we invoke the refresh API as well. I know it is not the most efficient way to make data available sooner for searching but since our write rates are low, we thought we could live with this inefficiency. If there is a better way to handle this scenario then please let us know.
Based on what you have suggested and my usecase, I cannot use BulkProcesser API. In addition, if, by default, fsync is happening on each write, do I need to invoke refresh API to make the document searchable?
EDIT:
Wanted to add that the behavior I describe above in #2 is actually running live in our production cluster. We were evaluating v5.6.2 so that we can upgrade production to that newer version. For we have a bunch of integration tests that run sequentially. And during that test run, we ran into this issue. We are going to try and create new cluster with v5.5.0 to see if this issue persists or not. We had not seen this issue previously.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.