We're currently working with re-indexing one of our indexes on one of our
test servers, and we appear to be having issues. The operation should be
fairly obvious, and there's not a lot of data -- around 320.000 total
documents in the index, at around 1.15GB of data. We scan from the old
index, and bulk index 1000 documents at a time (we've also tried 500
documents at a time to no avail). Since our production indexes are more
around the 30GB size, we're a bit discouraged by hitting this wall.
Once in a while, we get a random "resource not available" error, but they
usually resolves themselves and go away -- however, after indexing around
260.000 documents, the Python script starts behaving weirdly, throwing a
lot of resource not available errors. A full gist is
here: https://gist.github.com/HenrikOssipoff/ac63060d4c18182719dd (note
that this is Python 3.4, so chained exceptions and everything -- the top
exception is not actually the important one).
Now, thing is. During this time, Elasticsearch itself seems fine, and the
system is generally available for querying. The JVM heap is fine, and the
Elasticsearch log itself has absolutely no warnings or errors (log level
INFO enabled).
Our next course of action will be try to doing this by just cURLs outside
of Python, or even enabling debug logging in Elasticsearch, but we wanted
to see if anyone has some ideas for what could be the course.
It seems logical, that since it only happens every one in a while, and gets
worse the more documents we index, it has something to do with the amount
of _bulk indexes we do, but we can't seem to find anything in logs or stats
that support this.
Any help appreciated, or any pointers in the right direction.
Thank you!
Regards,
Henrik Ossipoff Hansen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2afa6d99-b9d4-42a2-9f4e-b717a3a41251%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.