Hi,
On our ES 6.x (various versions tested with 6.4.1 being the last one) installed on Ubuntu 16 hosts we are experiencing temporary problems with Indexing queries performance.
Single query can take up to 10 or more seconds to return results to client (we use curl for testing).
But if we see into elasticsearch logs and even find the specific query there it shows normal time (usually couple of milliseconds).
We didn't notice such problems with GET queries. Only Index.
Does anybody have any idea what can cause the difference between the time in ES log and the time it takes the whole process to return query result to client?
We were able to eliminate network problems between client and server machine as the situation takes place even when we query ES from localhost.
just to clarify: can you explain what the difference between a GET query and a an index query is? Do you refer to Index API as an operation to index a new document into your data?
Yes, by 'index' I mean indexing a new document. In fact - for test purposes to monitor the behavior - we are trying to index exact the same data in a loop.
Please take the time to properly write up your issue. I still have a hard time to understand what exactly is not working. Is it the indexing that takes a lot of time or is it the querying of data while indexing is happening?
Where did you extract the data regarding the time it took to execute the action from the logs? Which logs are you referring to?
And to clarify when you write you index the exact same data in a loop: Does this mean you are basically updating the exact same data all the time or are you only appending new data?
Problem is on the client side.
We do simple thing there - just send index API query to ES host, wait for response and record the time it all took. Then the same request is send again and again in a loop. And for some of those requests the recorded time is seconds, while for most of them it's milliseconds.
As for the logs - we have Slow log configured (as dectribed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html ) with tested index set: index.indexing.slowlog.threshold.index.warn: 0nanos - so every request is logged.
So writing about the times, I'm refferring to slowlog logfiles - no lines there about requests taking more than a second.
As to clarify what is the test request doing - yes, it's "basically updating the exact same data all the time".
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.